Bitcoin, Explained 67: Insights From the 4th Largest Lightning Network Node

Aaron van Wirdum Live from Utrecht, this is Bitcoin.

Sjors Provoost Explained.

Aaron van Wirdum Hey, Sjors.

Sjors Provoost Hello.

Aaron van Wirdum Hey, Sam.

Sam Wouters Hello.

Aaron van Wirdum We have another guest, Sjors.

Sjors Provoost Hello, Sam. Welcome.

Sam Wouters Thank you.

Aaron van Wirdum Wouters, who are you?

Sam Wouters I'm a Bitcoin research analyst at River Financial.

Aaron van Wirdum All right. What is River Financial?

Sam Wouters It's a US-based Bitcoin-focused company. Offers brokerage. Offers like zero fee DCA. So no fees on recurring orders. And offers hosted mining services. And increasingly, we're starting to do more with the Lightning Network. We first integrated it in 2019. And now we're offering a new product around it as well. Around Lightning.

Aaron van Wirdum All right. What are you doing here? Why are you in Sjors's living room with us now?

Sam Wouters Well, it's an honor to be here, I'd say. Yeah, recently released a report around our learnings from the Lightning Network, the insights that we gathered from running some of the highest capacity nodes in a network. And basically you reached out to me and you were like, well, it would be interesting to have a podcast about this. We talked briefly at Bitcoin Amsterdam about it, awesome conference. And yeah, we'd love to share more.

Aaron van Wirdum All right. Sjors, what do you think? Sam's here now. Do you want to do it?

Sjors Provoost Sounds good.

Aaron van Wirdum Might as well discuss the report.

Sjors Provoost Yep.

Aaron van Wirdum First of all, though, we did an episode about the Lightning LND bug. So that was the last episode two weeks ago. And I think there is an update about that.

Sjors Provoost Yes.

Aaron van Wirdum So to start off, we kind of speculated in the episode itself what the exact cause was. And we mentioned that we weren't 100% sure.

Sjors Provoost But we were 100% correct.

Aaron van Wirdum We were not correct?

Sjors Provoost No, we nailed it.

Aaron van Wirdum Did you, Sjors? Did you nail it?

Sjors Provoost Well, roughly. Anyway, we speculated on a few different possibilities. But yeah, it turns out that especially we were talking about what happens when you're not actually using btcd as a full node. You're just using Bitcoin Core as a full node, yet your Lightning node still stops and you're using LND as a Lightning node. And the reason that happens is because LND uses parts of btcd because btcd is not just a node, it's also a library of components that you can use. And so if you're using one of those components, even though you're not using btcd as your node, you still rely on how those components check certain rules about blocks that they should probably not check or at least should check correctly if they do. So those components broke.

Aaron van Wirdum Right. So it's basically because LND was using code from btcd.

Sjors Provoost Yeah.

Aaron van Wirdum Okay. Got it. That's clear enough. But there's been another bug this week.

Sjors Provoost Yes.

Aaron van Wirdum LND nodes crashed again. Is that right?

Sjors Provoost Yes. As the song by Britney Spears goes, oops, I did it again. The same person crafted another special transaction, put it in a Bitcoin block and voila, all the LND nodes crashed. Now, in this case, there are a few differences. I mean, in general, it was the same kind of bug. It was like a Bitcoin transaction that was perfectly valid, though a bit weird, and in such a way that Bitcoin Core nodes would be perfectly happy with it. But, [unclear], the btcd-based nodes or libraries that use btcd code would not be happy with it and would stop, basically.

Aaron van Wirdum Right well so last time the reason was that there was a limit on the witness size of SegWit transactions and then the Taproot transactions don't have that limit but for the blocks that that come in People should listen to the last episode, but I'm getting to the point where my question is, so was it exactly the same? Wasn't the bug fixed?

Sjors Provoost Yeah, that specific bug was fixed, but it was another difference in interpretation out there, which had to do with another limit that was slightly different in LND.

Aaron van Wirdum I see. So it was a very similar bug, but not the same bug.

Sjors Provoost The same kind of bug, but it was a different one.

Aaron van Wirdum Okay, but the last time LND nodes crashed, it was an accident, I think, right?

Sjors Provoost It could have been an accident. We don't know. The second time was definitely not an accident.

Aaron van Wirdum This time was not an accident. So this time this developer, and I think his name was Burak, if I'm remembering that correctly. And I actually I'm sure I remember that correctly because you told me five minutes ago my memory isn't that bad he did it presumably by accident last time so this time he did it intentionally he was looking for bugs and he exploited that.

Sjors Provoost Well the transaction has a message in the OP_RETURN that says something along the lines of haha you should be using C-Lightning so it was you know very clearly an attempt to to cause a problem there which is not very nice but it can happen.

Aaron van Wirdum Right. So have you guys have had any problem with this? Did this affect you at all?

Sam Wouters Well, we run multiple nodes and I believe the developers were immediately on top of it. And I think like upgraded the nodes, made sure that there were no issues. And I believe there was no downtime on the API that we have either. So altogether it was, it went pretty smooth. So fortunately not too many issues, but could have been worse for sure.

Sjors Provoost Yeah that sounds lucky because it would have it did stop a lot of nodes yes.

Aaron van Wirdum So hang on though but so this bug was also fixed by now within a couple of hours it sounds like?

Sam Wouters I believe it was pretty quick.

Aaron van Wirdum Yeah and then it was basically fixed before it affected you?

Sam Wouters Believe so yeah.

Aaron van Wirdum Okay so but that does obviously mean you need to pay attention?

Sam Wouters Yes yeah for sure.

Aaron van Wirdum Yeah Okay, I think that's also the type of stuff we're going to discuss in this episode.

Sjors Provoost Yeah, and I guess the failure could depend on exactly the details of your setup, which I don't think you know in that much detail, but... I think it would stop processing new blocks, but if the channels are still there, those channels might keep working. So that's maybe why it didn't go down, something like that. There's a couple of other details about this exploit that might be worth mentioning before we move on. One of them is that this transaction was a so-called non -standard transaction. And that means that it is a weird transaction that is valid, but nodes will not relay it by default. So if you see this transaction in your mempool, you will not send it to your neighbors. Which begs the question of how did this transaction ever find a miner? And this mechanism is mostly a safety mechanism to prevent you from accidentally throwing away your money.

Aaron van Wirdum Not only how does it find a miner, but even why would a miner itself include it then, right?

Sjors Provoost Yeah, because miner software would, again, generally not include those things because it has to do with some safety mechanisms for soft forks that if you don't include non-standard transactions, that's a double negative, but if you don't include non-standard transactions... then you will probably not get screwed at the next soft fork if the rules change. That's the way. We've talked about that in some other episodes. We'll skip that now. But basically, so the way he got it in there was just emailing this pool and just asking them to please include it. And it included a nice $750 transaction fee.

Aaron van Wirdum Interesting.

Sjors Provoost Now, it was a very, very large transaction. I think it was half a megabyte. So this was 26 Satoshi per byte, which is a high fee, but not an absurd fee rate. But the total fee was pretty high. So yeah, that's how it got included. And once it's in a block, nodes will simply accept it.

Aaron van Wirdum All right. I don't want to dwell on the topic, but I guess now my... curiosity has been sparked like what what's the incentive for Burak to even do this I I mean I know there is this sort of controversy about LND maybe being too dominant on the network is that why?

Sjors Provoost I don't know there could be my guess would be without having spent too much time it's like there's two ways you can deal with security issues when you when you see a product that you think is not secure you can contact the developers of that product and disclose carefully what the bugs are and be very, very patient. So it could take months for them to address it. Or you can go out and simply break their product in public and create sort of a warning signal to the general public. Now, that last thing could be warranted under some really extreme circumstances, but I think this was too early for that. But it may have been something like that, like telling the world, like, hey, be careful. Lightning is not as mature as you think it is. Things can still break. I don't think it's the right way to do it, so responsible disclosure would require you to do these things. Contact developers directly be very very patient about stuff and then even if you did want to do a public demonstration of the vulnerability I would say do it on testnet so that you know you can show that you have this thing that can crash nodes but it only crashes test nodes and not real nodes.

Aaron van Wirdum Okay I'm gonna move over move on to the report very soon but first of all because I don't think this is in the report but now that I mentioned it Sam do you have any insights about which types of nodes are used on the Lightning network? At all? It is mostly LND, for example, right?

Sam Wouters I think largely, yeah. It's a very popular one, indeed.

Aaron van Wirdum Yeah, but you don't have any...

Sam Wouters No, I don't have any statistics.

Aaron van Wirdum There wouldn't even be any statistics.

Sam Wouters Exactly, yeah.

Aaron van Wirdum I guess, I would assume.

Sjors Provoost Well, my guess is you could figure it out, but I've looked at mempool.space. They have a new Lightning Explorer, and they don't show the node type. So I guess you'd have to pay attention to some subtle behaviors to guess whether somebody has a C-Lightning node or an LND node or an Eclair node.

Aaron van Wirdum Yeah.

Sjors Provoost Okay but also I do know that most of the mobile wallets are based on LND yes and that's probably the majority of nodes but not maybe not the majority of visible nodes.

Aaron van Wirdum Anyway anyways okay moving on Sam, your report it's called insights from the fourth largest Lightning network node yes and I think you briefly mentioned it already but let's start again by why did you write this report why does this report exist yeah what is it and why does it exist.

Sam Wouters Yeah so the report's focus is to share data and essentially as the title says share insights about what we've learned from the Lightning network and When we initially had a look at sort of where things were at when I joined the company earlier this year, Lightning was starting to pick up quite significantly in terms of activity as we could see in the data. We have a team of four Lightning developers and I just saw them sharing things in Slack. And being enthusiastic about the progress. And I figured like, I haven't really seen anything like this out there too much. So we started discussing like, hey, could we, you know, like share some of these insights with people? And there are people who have been sharing data, which is great, but not much from larger nodes or bigger companies, anything like that. So we figured, what if we make a report and share what... we think people might be interested in as a first pass sort of to see what is the interest from the industry what is the interest from the market where do they think it might be interesting to learn more about Lightning and the way we operate it or the way they could operate it and then potentially repeat the report in the future with some learnings with more insights but the focus was very much on data sharing because it's very difficult for I believe anyway for people to work on this kind of system and to improve it if they don't really have a lot of statistics about Like, you know, why payments are failing to give an obvious thing of how users are actually experiencing the Lightning Network. You have to do a lot of in-person conversations with people, I guess, a lot of qualitative research. And that's quite time intensive if you're a developer who wants to, you know, build systems. You're not necessarily a [unclear] market researcher.

Sjors Provoost And you might be running a non-custodial service where you're not spying on your customers enough to see what problems they're running into. Or you might be, you know, building a node. And you don't even have a help desk.

Sam Wouters Yeah.

Sjors Provoost So yeah.

Sam Wouters Yeah, definitely. So we were trying to figure out like what would be the best format to share this. The report is relatively brief. I believe it's about 20 pages and lots of images and graphs in there to just make it accessible to people. So to River clients who might not all have necessarily used the Lightning Network before or really learned much about it. So it also keeps them in mind a little bit, makes it accessible for non-technical people. And yeah, the focus was sharing data. And what we're now already seeing is that other people are sort of jumping in, starting to share their insights too, based on some of the metrics that we shared. And it's starting to get a conversation going, which is really what we were hoping to see there. And hopefully this can help developers who want to improve Lightning to gather more data and insights so that they improve Lightning and essentially everyone benefits because it's definitely in our interest to see Lightning succeed. And we're trying to do what we can to help there.

Aaron van Wirdum And to be very clear what you did specifically, this is the data from your own node. You didn't collect data from other nodes or users. This is just... You guys have one of the biggest nodes and this is your experience. That's what it is, right?

Sam Wouters Yeah, we actually have two nodes in the top 10 by capacity. So how much Bitcoin is on there. But initially, the second one was in the process of being built up. So it wasn't up there yet. But by the end, when I was done with the report, we also had number five or six, I believe, by capacity. The data is all from there. I would love to, in the future, start aggregating data for more providers. And I've seen some comments on Twitter of people being like, yeah, but this is just what operators are saying. How can we trust that? And it's like well you could you know you could lie about how many transactions you're processing and make all kinds of things up to make it look bigger than it is but eventually it's going to catch up with you if you pretend to do like 10 times the number of transactions that you're currently doing and everyone else shares their data and nobody gets close to what you're doing then It looks kind of dicey. So mostly I believe at the current stage of Lightning, people are very invested in seeing it succeed in the long run. So like you don't necessarily have to trust everything and you can put like 17 asterisks in the report saying like this is all like self-reported data from all of these users. But across this many users, these are the insights we can gather.

Aaron van Wirdum Okay, so the way we're going to handle the report is we're not going to... deal with it page by page we're gonna select some subtopics that we've found of particular interest maybe in context of a more technical podcast so let's start with the most important question how much money are you making?

Sam Wouters Well, me personally?

Aaron van Wirdum No, no, I'm not asking about your salary. I'm asking about the fees.

Sam Wouters I know it was a joke. Yeah, no. So the focus of the company isn't to make as much money as possible on Lightning. The focus is to help grow it. So currently the returns are about 1.15% per year. And in the report itself, I also compared this to some other users that have sort of shared their data. Some of them earn like 5% to 10% per year which is already a pretty significant number but there's likely some kind of plateau in there where if you add more capacity to your node you're not going to be able to reach those numbers because there's not enough activity on the network for you to get those kind of returns with that much Bitcoin so there's a that's a little asterisk there so for River it's 1.15 at the moment but the focus is much more on reliability of the of the Lightning integration that we have

Aaron van Wirdum Right yeah.

Sjors Provoost I think one of the things you pointed out in the report was to say like companies can optimize for very different things so they could optimize for revenue but if they have clients they may want to and they make money from those clients in very different ways they want to make sure that the clients just have the best experience when they use Lightning even if that's not profitable at all so and that also I guess in the marketplace means that you're competing with people who do not optimize for profit so that means that fees might stay quite low but we'll have to see how that goes yeah.

Sam Wouters Definitely.

Aaron van Wirdum What is the most work so to say, in operating a node? Is it managing liquidity? That would be my guess. That's sort of where most of the effort goes into. Is that right or no?

Sam Wouters Yeah, I think that's fair to say. As a node operator, I mentioned we have a team of four developers. They're not spending all day closing and opening channels and managing liquidity. That would be very extensive. They're very focused on building out the infrastructure that we are offering to other companies. So that's where the majority of the work goes. Managing liquidity as you mentioned and opening and closing channels that does take up some time especially if you want to have successful routes if you want high reliability so there's a bit of vetting involved there in sort of the nodes that you want to connect to because if you just connect to A random hobbyist that will send one transaction per month doesn't really have much liquidity, doesn't really put any effort into it. It's kind of difficult as a company to justify that because you're locking up capital in a very inefficient place with no track record. So there's a little bit of vetting. It's not very strict. And our CEO has also mentioned on Twitter, like we're looking into ways to make this a bit more flexible. So there is a whitelist at the moment. And only if you're on there, you can open and close channels. But it ensures that... there's a high reliability for transactions to get routed and that's ultimately what we want to focus on to get it reliable enough for companies to want to consider accepting Lightning as a payment for example.

Aaron van Wirdum So how does the vetting work what are the factors you you weigh when it comes to vetting?

Sam Wouters Yeah for example some of the things I mentioned like how long has this node been there like is there some kind of track record how many connections do they have how much Bitcoin capacity do they have essentially is there some kind of minimum for the channel size it's not very large but it's not like you're opening and closing channels for just a few dollars because there are already like so many of those out there so I think those are some of the main reasons main things and I'm sure…

Aaron van Wirdum Like how does this actually work is this is there someone sitting there looking at these things or is it automated and by the way I will notice right now so we had a discussion yesterday before we did a podcast where I mentioned [unclear] as well but for me it's definitely true I haven't been keeping up with Lightning very closely at all for the past couple of years like I see I'm somehow way more interested in Bitcoin base chain always have been so it's very possible that I'm asking very beginner questions just so the listeners know and you know.

Sjors Provoost I don't pay super much attention to Lightning, but I do run my own node. But I don't pay much attention to the node either. I will open new channels if I make a payment and it doesn't go through. That's sort of a reminder to me, okay, I should probably open a channel there. Or if people complain that they can't buy my book. But other than that, yeah.

Aaron van Wirdum So the question was is there actually someone sitting there looking at these things or is this some automated process?

Sam Wouters Yeah no it is a human process very much so because there's just like too many things to consider etc it's too early stage in the future I could potentially see it automated but that goes for for many of the processes they're currently manually doing it's it's just a matter of time before a lot of that will get automated and needs to be automated because if you know if companies want to run Lightning it's kind of unlikely that they'll want to have a full-time employee just sitting there managing channels and liquidity. It's just not realistic in many cases. So for your note there, like on Lightning knowledge, like I'm also not a developer. I'm not a technical person. So I do spend time looking at Lightning because... similar to Bitcoin like it gives off a lot of energy like Bitcoin in the early days it gives off a lot of energy there's a lot of people excited about it it's really interesting to learn more about so I try to keep up to date but I'm definitely no expert either.

Aaron van Wirdum So if there if if it's currently such a manual process to manage liquidity for example does that mean in effect that how centralized is the network now is it really dominated by a few big players like River that actually have someone doing this because they can afford to do it?

Sam Wouters Well, if you take a look at some of these node operators that are sharing their data on Twitter publicly, I believe Goldsatz is a quite well-known one who just shares all of his strategies, all of his approaches. There's quite a lot of other Lightning enthusiasts like it who just share how they do things. And for them, since they're earning higher returns, they can also potentially actually be an income stream. But generally they mention like it's a couple hours per week work for his approach, for example, that gives a bit of an idea for people like how much work is this actually. But this is, of course, someone who is very experienced, really knows what they're doing. If you're not that kind of person, there's probably quite a learning curve. It will take you a while to get to that point. So then it's a significant time investment, which, yeah, of course, changes the entire picture for people. But to your question, like is it centralized at the moment? From everything I've seen, like all the data I've looked into, And there was also a really nice visualization that I added to the report from a Twitter user called PyMoment, which was a visualization of the Lightning Network. And you could see how essentially there's dozens and dozens of different routes that any given transaction could take, like through sort of the major nodes out there. And there will be many sub-routes that will be longer, that might be less reliable, etc. So in that regard, for the current size of the Lightning Network, I would say it looks pretty dang decentralized. It could be far more decentralized, of course. To sort of bootstrap the network from the ground up, it makes sense to have a very reliable experience for people so that they can enjoy the benefits of the network. And then as it grows bigger and bigger, I'm sure people will also become more vocal about centralization and that type of thing.

Aaron van Wirdum Right. And I guess the other cost in managing liquidity, so we've just discussed the human cost. The other cost would be on-chain fees, I guess, because you have to open and close channels. Is that right?

Sam Wouters Yeah, for sure. And I think...

Aaron van Wirdum Right now, surely that's very low. So that could change into the future. What are the expectations there?

Sam Wouters Yeah, for sure. Like, I mean, nobody knows how expensive it's going to get in what kind of timeframe. Yeah, he does.

Aaron van Wirdum How expensive is it going to get 20 years from now?

Sjors Provoost Less than 21 million Bitcoin.

Aaron van Wirdum So disappointing. Okay, we have an estimate.

Sam Wouters Yeah.

Aaron van Wirdum Sorry, Sam, go on.

Sam Wouters No, no problem. Think... like what it might be an issue for in the future if you want to get more people who set up routing nodes where they actually earn returns that's like that requires some technical skill it requires a lot of trial and error and setting up channels trying to find optimal routes where you can actually make some money so if you have to open and close a lot of channels to get to the optimal setup but transaction fees are quite high then you could have sort of a upfront cost that you would need to pay to actually get to that success point But in a way, I mentioned this in a report as well, it's not that dissimilar from someone who wants to build up a successful YouTube channel, for example. You need to invest in some equipment. You need to really put in a lot of time and effort to make really good videos. And by doing that, you invest a little bit in it, but it can then give you returns. So I actually see it quite similar to that where the best content creators, they don't talk into their iPhone. They have a proper microphone and some good lighting, etc. And that just makes a massive difference. So I see this play out in the future quite similar for Lightning. However, that argument goes only so far. You know, if transaction fees become hundreds of dollars, then it becomes pretty expensive to become a successful routing operator, a node operator.

Sjors Provoost Well, it goes both ways, right? If transaction fees on-chain are $100, well, that implies that everybody who tries to use Bitcoin has to pay that $100. So that means a lot more people will try to use Lightning and they'll be happy to pay $20 Lightning routing fees.

Sam Wouters Yeah, for sure. But what it changes essentially is that the upfront cost to become a successful routing node, that would rise, that would become potentially less accessible to someone who doesn't have a bit of a budget to invest in this. So that's the only thing. But is that a big threat to the decentralization of the network?

Sjors Provoost But I'm not even sure about that, right? Because it's all about the money, the returns you're making. On the capital that you're investing. So if you're putting up $1,000 and you start making $300 a year, then it's actually better than it is now. So maybe those high fees just mean that you get more return on your investment and you don't have to be a big operator. You could be a small operator.

Sam Wouters Yeah, there's definitely many facets to it, I think.

Aaron van Wirdum You mentioned, and again, well, maybe I should stop mentioning this myself. I was just going to say this might be a very beginning question again. But you said, so there's particular routes you could be on that would make more money than other routes. Would the way to figure that out be just trial and error? Sorry, trial and error. Just try it out and see what works? Or are there other ways to sort of figure it out beforehand?

Sam Wouters From my understanding so far that's how a lot of people do it so when I mentioned earlier that we're trying to share a lot of interesting data that people might be interested in the number one request is obviously from people like what are your most profitable routes but the tricky thing there when you share that publicly is you get all these people who copy the exact same setups And by doing that, the profit for everyone declines and ultimately everyone's disappointed because some people feel like, oh, you lied about this because I'm not making any money. And it's like, well, you know, like 100 people just did the exact same thing. Of course, it's not going to be great.

Sjors Provoost That's arbitrage. So I guess to make a concrete example, let's say Starbucks starts accepting Lightning payments and nobody has a channel to Starbucks, but you do. Then everybody wants to pay. Starbucks goes through you. And so you can just charge like 2% on every transaction. You wouldn't do that, but you could. And then as people find out, oh my God, that's so profitable, I'm going to open a channel to Starbucks too. Then, of course, those fees will go down in competition. But the other problem is they also need to have inbound liquidity because you are part of a route. So you are the last destination before Starbucks, but somebody has to get to you first. So if nobody has a channel to you, then even though you have a giant channel to Starbucks, nobody's going to use it because they can't get to you first. So it's tricky on both sides, I guess.

Sam Wouters Yeah, definitely.

Aaron van Wirdum Okay, so this segment of the podcast was about fees and liquidity and also because that's the segment of the report, of course. Is this what we want to discuss about this or just do you have any questions about fees or about liquidity? Sam, did we forget to mention anything? If not, we'll move on. Okay, so the next point is why Lightning payments can fail. So Sam, why can Lightning payments fail? What did you find?

Sam Wouters Yeah, what's nice about being a larger node in the network and having a good bit of connections is that you start to get to volumes where you can say something statistically meaningful about these failures. Whereas if you're a hobbyist, you've routed maybe 100 transactions in a month, it's a bit difficult to draw conclusions from.

Aaron van Wirdum Wait, so how many transactions do you route per month?

Sam Wouters So in September, I believe off the top of my head, it was 115,000. And so then you can start to say something meaningful about it.

Sjors Provoost Yeah, that's a huge number. So that might be useful to clarify. You're primarily a routing node, as in people connect to you and like payments go through you, but you're not necessarily, you are sometimes, but you're usually not the origin and destination of those payments. So it's really a perspective of somebody sitting in the middle. How often does that work or how often does that fail? Which is different from the end user point of view. So to remind people, if you're making a payment on Lightning, you are the one deciding the route, generally. So that means you have an idea of what the Lightning network looks like and what the capacity of all these channels is, and you construct a whole bunch of routes and you try all of them until it works. And that means you know that it failed or not. If you're receiving coins on Lightning, you're just telling people, please send the money to me. And then either you get something or you don't get something. There's some in-between phases where I think you get something, but then it still doesn't finalize. But you have a lot less information. And when you are a route, then you don't know where the payment is coming from and you don't know where the payment is going because it's all onion routed. So you can only see the next hop and where the previous hop where it came from. And so all you know is that some payment tried to get through. And that's it. But you don't know any context there. But you can still do statistics on that. And that's, I think, what your report is mostly focused on. Yeah, exactly. Having 100,000 plus payments per month is a lot, I think. And then a very low failure rate. Though, though, I think the failure rate doesn't necessarily explain—it doesn't necessarily completely include what the end user is experiencing. Because some payments might not even go through you. And so they might fail. Or maybe the payment is split up in multiple parts. And the part that goes to you is fine. And the part that goes to somebody else is not fine.

Sam Wouters Yeah. And to maybe give some context on the numbers there. So I mentioned 150,000 transactions that was in September. I believe October, like I haven't looked at the data yet, but for September, so 115,000, the success rate was 98.7%, I believe. And I did check a couple months back as well to see if this wasn't a fluke of course and the numbers were typically around that I believe at one point they were 99 [unclear]—sorry, 98.5 so they've sort of been hovering around there for the past few months out of that essentially so 1.3 percent of the payments that…

Aaron van Wirdum Wait does this mean are you bragging now is this thanks to River doing good work? Or do you think this is sort of general?

Sam Wouters So I've checked with some people to like just out of curiosity like do you see these numbers as well some other people have it a bit lower but like they also don't have a sort of a white list of channels that they connect to and some people will mention like yeah if you're you know if you're connecting to all kinds of random nodes then obviously that success rate will drop So it's also not bragging in the sense that we're happy with it. There's progress, but the ultimate goal is just to get to 100% or as close as 100% as possible. Because if we want people globally to use this in a reliable way and use it for business, then you just need to be extremely reliable. Yeah, 98.7% is great. But if one in 100 payments doesn't make it, some people have the patience for that.

Sjors Provoost Well, that depends on really on the details, right? Because it sounds really bad. Like if that was the case for normal credit card payments, that would be a complete disaster. But because Lightning Wallets make multiple attempts to finish a payment, 98% success rate might be perfectly fine because it'll one second later try some other route and it'll go through. So maybe it's perfectly fine, this number. I find it hard to qualify. To assess that, maybe some Lightning expert will tell. It probably depends on the kind of failure. So I think there are Lightning failures where it immediately tells the wallet that's trying to send, hey, I'm sorry, your payment failed. Try again. There's another one where it's like, oh, sorry, your money is stuck in limbo for the next three weeks. Those are the nightmare failures. But I think those, I don't know if you can measure those separately, but I think what you're showing in your chart is one is a timeout. So that I I don't know maybe the listeners can help us explain what what exactly the thing is that times out the only thing I could find was multi -path payments apparently when you have multi-path payments so your wallet sends…

Aaron van Wirdum Well hang on Sjors, we just mentioned there's a high success rate but it's not a 100 and then the failed payments can be divided into several categories right so what are the several…

Sam Wouters Yeah, the two biggest ones, so Sjors just mentioned one, it's a timeout. And from my understanding, the timeout means that after about a minute, which is the standard time that has been built in, after about a minute of trying to find a route, like trying to send this onwards, it just can't seem to figure anything out that definitively gets it to the final point it needs to be. And in the second... option that might actually be the second biggest that I just mentioned the timeout the biggest might be that there's no route it's one of the two they're both about half a percent ish or so so there's just no route available. Let's check the options and it isn't there even though there was supposedly a way to get to the final destination.

Sjors Provoost And so the latter thing suggests that for example you're talking to a mobile wallet and so the next hop in line I guess is a mobile wallet and that mobile wallet is just not there so the original Person making a payment constructed a route which went from you to the mobile wallet and then it's gone.

Sam Wouters Yeah, potentially.

Sjors Provoost Or could it be multiple hops away too? Because you have, of course, no control over the final destination?

Sam Wouters Yeah, indeed. And I think as well, the other reason that gets mentioned quite a bit is Tor. So people using just like the Tor network has been experiencing, I believe, quite a lot of issues over the past months. I'm not sure how long this has been. So there can also be issues there where there was supposedly a node in between that was going to be accepting this transaction, like routing it forward. And if Tor is having issues, then it could be that initially there was a connection established, but it drops for some reason. And that could also cause issues.

Sjors Provoost Yeah. I mean, it seems like for your direct peers, you could try and make sure that you connect to those peers through clearnet. If you know who they are, then you can call them and say, can you give me your clearnet IP address, I guess? They might give it to you or they might not. But if you don't know who they are, then I guess that's part of your criteria, right? Whether you want to connect to a node that's on Tor because you might get lots of failures if there's a problem on the network.

Sam Wouters Yeah, I don't know if that's something that's currently looked at as one of the criteria that you asked about earlier, Aaron. So I'm not sure there. I haven't heard anything about it, so it would actually surprise me. But it is a factor. Is it a very big factor? It's one of the biggest factors at the moment of the 1.3% that isn't making it. And perhaps for other people, this is significantly higher. So it's definitely worth exploring more, I think, for people that sort of understand where is this issue originating.

Sjors Provoost Yeah, we talked about probing yesterday during the preparation. And I guess probing means that somebody is just scanning the network for how much money is actually in every channel. So they just send a fake payment, essentially, to a final destination. And if the fake payment makes it through, then the recipient of the fake payment will not accept it because they don't have the pre-image. So it will fail, but it will fail in a different way than when there's not enough money. So the attacker or scanner, whatever you want to call it. Can see how much money is actually in the channels because that's normally kept private. But if that was happening a lot, you would expect a very, very high payment rate, as you pointed out. And you're not seeing that. So that implies there's no probing?

Sam Wouters There's probably some, I would guess. I don't know for sure. There's also, I believe, probably many of the chain analytics companies that are trying to figure this kind of stuff out to see if they can sort of... reduce the privacy on Lightning and figure out what's going on there but there I couldn't see anything in the data that would indicate like this is happening at mass scale and really impacting sort of success rates etc.

Aaron van Wirdum Okay, I want to recap this, and I'm not sure if I understand this correctly. So do correct me if I'm wrong. So generally, we have a high success rate of payments, or at least sounds high to me. I don't know what would be considered high, but 99% sounds kind of high to me. And then out of the 1%, so the payments that do fail, you can basically categorize them in two categories. So one is timeout. And one is no route available right and then there's a very small sliver of other I'm seeing in the report do we even need to get so what's other that you just don't know or what other?

Sam Wouters Others could be there is indeed there's like a error unspecified like when there's just some kind of issue my guess is one of the issues as well I believe I looked into it was that someone might send an actual manual payment but they make some kind of mistake in in their in their payment or in the instructions etc and it doesn't go through for some reason or the node could be uh offline for just a second or something a power outage you name it something yeah something like this. Could happen but that's that's such a small minority that I really believe focusing on the first two will give the biggest results in getting a higher reliability.

Aaron van Wirdum Okay so the first two were or the main two the two are timeout and no route available so my first question about no route available is why would it even get to you if there's no route available? Did you already discuss this or no?

Sjors Provoost Yeah, it wasn't super clear to me either, but I think it just means that the next hop isn't available, which could be that you have a... So basically Lightning nodes gossip around things about the network. So they'll say, oh, River is connected to these other five nodes and they have channels that are this big. And so somebody makes a payment based on that information. Then the payment gets to River, and River's like, yeah, I'm trying to reach that other guy, but they're not online.

Aaron van Wirdum Right. Okay, so that could be the mobile wallet, which you just mentioned. For example right okay I got that one then yeah and then the other one would be timeout and you mentioned that's probably Tor related I think?

Sam Wouters Could be that's the biggest suspicion I've heard from people so far indeed

Aaron van Wirdum Well I can definitely confirm because we do our meetup in amsterdam and a lot of people try to pay with Lightning and some of the people have set it up themselves with their wallet and then it's going through Tor and that's probably the number one problem we we encounter when people try to pay us through their own sort of set up which uses Tor and

Sjors Provoost Well they're probably using Tor in both directions so they're probably using Tor to connect to their home wallet in the first place and then that wallet itself is connected through Tor to the next hop on the Lightning network route so yeah that can get slow.

Sam Wouters Yeah makes sense maybe to give a little more context actually I was just thinking like we've shared the the number of transactions the payment success rate it's also interesting to know I guess for people like for what kind of size of transactions is that because you know it's really easy to have an extremely high success rate if people are just sending one cent so to speak or one Satoshi worth but the average transaction size was 46 dollars Which to me, the first thing was like, I had to quadruple check if this was correct, because it seemed really high for Lightning given the current on-chain fees. But it gets skewed upwards very high by like, we have high capacity on the nodes. So obviously larger transactions are a bit more likely to come through us and some of the others in the top capacity nodes. But then I had to look at the median size and it was about $5. So that seems a lot more realistic and sort of around what you might expect, given the fact that there will still be large transactions that people just... trying things out. So that gives a little bit of an idea, like this is sort of the median transaction size for which that payment success rate is there. And I had a look at some of the earliest data I could find in terms of payment success rates on Lightning from, so I believe from 2018 or might be 2018. Yeah. A $5 transaction failed 50% of the time ish. So that's quite a jump there.

Aaron van Wirdum Right. And I would assume that the bigger the payments, the bigger the [unclear] will fail…

Sam Wouters That was the assumption initially as well, but due to the size that the Lightning Network is at at the moment, we had a look at as well as how many transactions have we done over a Bitcoin, for example, worth, as there are some large channels. But typically when people try these, they're...

Aaron van Wirdum Over a Bitcoin? Is that what you said?

Sam Wouters Yeah.

Aaron van Wirdum Over the Lightning Network?

Sam Wouters Yeah.

Aaron van Wirdum Okay.

Sam Wouters I believe we've routed four, one of our developers.

Aaron van Wirdum Really?

Sam Wouters Yeah. This was yeah but but.

Aaron van Wirdum I'm guessing that would skew the average?

Sam Wouters Yeah and that's why but but actually yeah hence why I looked at the median to exclude those indeed but yeah looking at the stats there it gave gave some insights in sort of how have things progressed ever since so that's I think relevant context for people to understand when we talk about these numbers.

Sjors Provoost I mean keep in mind that if you're sending a Bitcoin to another node you're probably sending it to some node like through a bunch of channels that are very high highly reliable you're not sending it to some random mobile wallet on Tor so I would imagine that the success rate would be higher.

Sam Wouters Yeah that's actually the point I wanted to make but I kind of lost my train of thought there but indeed like likely when people are sending such large transactions they're going to just manually take a look at you know could I actually get this through is it there or not so I believe at the current stage you can't say a lot about it yet there's probably you know reliability might drop a little bit when you start sort of messing with the threshold of the sort of upper x percentile of the channel sizes but uh yeah for the most part it seems to be pretty reliable.

Sjors Provoost Yeah and if you want to know a little bit more about routing you should listen to the episode I did a year ago with Rene Pickhardt about optimally reliable payment yada yada yada But anyway, a smart way to split transactions into things. And he looks at, he talks about metrics. Like if the channel is, you don't know what the capacity of a channel is. So your best guess might be half, basically. So your best guess might be the half of the money is on one side and the other half of the money is on the other side. And you can do some reasoning from there.

Aaron van Wirdum Also, we did an episode with Joost Jager. That was also about routing, right?

Sjors Provoost Yep.

Aaron van Wirdum Okay well so we discussed why Lightning payments can fail and I think do we discuss everything about that?

Sjors Provoost I think so yeah.

Aaron van Wirdum Okay next topic this will be the last topic of this podcast. Lightning infrastructure for business.

Sjors Provoost Yeah, you mentioned that River is running a Lightning node as a service. So basically instead of the clients having to run their own Lightning nodes, you run it for them and they just use an API that makes their life easier. And you've discovered some things that are important.

Sam Wouters Yeah.

Sjors Provoost When doing that.

Sam Wouters Yeah. So actually River has been offering this for a while, but to just one client to try things out, which is El Salvador's Chivo wallet. We offer the API for them to route any Lightning transactions that people want to do through the app through our nodes. And what this essentially does is it makes life easier for the developers of this app instead of them having to spend part of their time on keeping nodes online. Updating them making sure they always have enough liquidity etc that responsibility is transferred to us which also means that if Lightning payments fail people look at us like well essentially the clients look at Chivo and they're like why is it not working but Chivo looks at us like hey you have to make sure this works as often as possible and thus why I mentioned earlier our focus is very much…

Aaron van Wirdum I want to specify something real quick here if it Chivo user pays a Chivo user it doesn't use Lightning right that's just custodial so we're talking about yeah Lightning payments from Chivo to some other Lightning.

Sam Wouters Specifically.

Aaron van Wirdum Yeah okay go on.

Sam Wouters Exactly yeah so if people want to use the Lightning like essentially if the clients want to use the Lightning network we need it to be as reliable as possible as I mentioned earlier that's really why that's the focus but initially like build this up with this as a first client and then recently sort of came out about it and shared it with the industry as initially a lot of people thought that Strike was doing this but it actually wasn't the case. But that was sort of the first client.

Aaron van Wirdum Wait, so was River doing it from the start or did River jump in later?

Sam Wouters No, from the start.

Aaron van Wirdum Okay.

Sam Wouters So from there on out, like we learned a lot by doing this and then made this product public where now other people and developers, et cetera, can look into this API. If anyone's interested, the website is rls.dev. But you can also find it through the River website. And it's essentially, as mentioned, it's an API so that you can route these payments through. And the first immediate thought that people have like, oh, but that's centralized. But then everything goes through River's nodes or something. That's a problem for the Lightning Network. But I see this very similar to when you tell a friend about Bitcoin and they're pretty excited about it. They're like, all right, I'll buy a little bit. In many cases, they'll buy a bit and put it on an exchange and keep it there. And that's not really the experience that veteran Bitcoiners like to see, but it at least makes them take that first step so they can then start learning more about self-custody, sort of evaluate what is the impact for this on me, on my life, on my Financial decisions. And that's a really first important hurdle to overcome. And in a similar way, if companies want to start taking Lightning seriously and accept it as a payment method, etc., but they don't necessarily want to... have a developer that spends a lot of time on this then they can use such a service and sorry… So if a company wants to use the Lightning Network, but doesn't necessarily want to put a full-time developer on this, then this makes it far more accessible for them to get started. And if they then take it very sort of seriously and it's promising numbers, et cetera, then they can use it to like use that position to then reevaluate. Do we maybe want to run our own infrastructure? So it's the first important step for businesses to want to consider this.

Sjors Provoost Yeah, and something I brought up yesterday, I don't know if the answer to that is known, but you mentioned there's four people working on Lightning. Well, there's four developers, but they're maybe doing even more than just Lightning.

Aaron van Wirdum You mean at River?

Sjors Provoost Yeah, exactly. And that might sound very large, but I have seen companies that run altcoins and they will have a bigger team on their Ethereum node. So by the standards of, you know, compared to running a Bitcoin node and doing on-chain payments, it's definitely more complicated. But is it more complicated than certain altcoins to manage these nodes? I don't know the answer to it. My guess is it's somewhere in between a very, like, very demanding altcoin. And a much simpler altcoin or something like that. So that's another basis of comparison you can have. So it's not that you will only have River to run the only custodial Lightning or the only hosted API for a Lightning node. There could be a lot of companies offering that service, and you can just swap out to a different one if you don't like it.

Sam Wouters Yeah, exactly.

Sjors Provoost Or do it yourself if you have more than four people available or something like that. Maybe the number is lower.

Sam Wouters So initially when Lightning started growing and becoming people who were very vocal about it being a scaling solution for Bitcoin, they sort of started to push exchanges to adopt it without really realizing how much work is involved in doing this. And typically exchanges, they're big enough to be able to do this. So I'd say definitely keep sort of pushing them to take some action. But for a lot of businesses, if they would like to do this, there's a fair bit of work involved. It's not something to overlook. So this just helps it to make it more accessible for them. And then they can essentially play with it themselves, see what demand is like from their customers and get a first good look.

Sjors Provoost Yeah. And sharing the resource of liquidity probably helps in general, right? Because you have to lock up. Quite a bit of liquidity in order to have a good experience so if you can share that burden with multiple businesses that might make sense yeah.

Aaron van Wirdum Okay I would assume in general if you're gonna put in man hours to add something to an exchange you're running an exchange you want to add something it's probably more profitable to just add some shitcoin that people trade rather than yeah add Lightning because people can already use Bitcoin on chain right yeah that's probably the reason that a lot of exchanges haven't integrated Lightning although it is growing now some have some are starting to adopt it.

Sam Wouters Yeah definitely and I think that's totally correct what you're saying and for us it's nice because we're just Bitcoin focused we don't have to worry about all these other coins and integrations and you name everything So it's really exciting to just focus on this and try to help it grow as much as possible.

Sjors Provoost Thank you for coming and explaining all this interesting stuff to us. And dear listeners, thank you for listening to Bitcoin.

Aaron van Wirdum Explained.