Episode 66: The BTCD Bug That Brought Down LND Nodes

Aaron van Wirdum Live from Utrecht, this is Bitcoin.

Sjors Provoost Explained.

Aaron van Wirdum Sjors, I have horrible news.

Sjors Provoost Oh my god, tell me.

Aaron van Wirdum No one, zero people, a great nil, a great total of nil people used the promo code Sjorsnado. From what I understand.

Sjors Provoost That's unsurprising.

Aaron van Wirdum I don't know what that means. It either means we only have hardcore fans that already bought their ticket before we even gave out the code. Or it means no one can remember your name.

Sjors Provoost Or it means they used the other promo code at the end of the show that was spliced in by the editor.

Aaron van Wirdum Oh, was there another one spliced in?

Sjors Provoost Yeah.

Aaron van Wirdum You listen to our shows. I sometimes do, but I didn't notice that. Oh, that wasn't easier. That one was easier to remember?

Sjors Provoost Probably.

Aaron van Wirdum Oh, okay. Okay. That's a bit sad. Or just no one listens to our podcast. That's option three, I guess.

Sjors Provoost No, I think some people listen because some people at the conference talk to me as well that they listened and that they read the book.

Aaron van Wirdum Which book is this, Sjors? Do you want to shill it?

Sjors Provoost Yeah, it's a book I wrote called Bitcoin, a work in progress.

Aaron van Wirdum Very well. Sjors, I have more horrible news. The Lightning Network crashed last week. I know we were both at Bitcoin Amsterdam, or at least maybe it was a couple of days before that. We were in full preparation of Bitcoin Amsterdam. Our mind was occupied, but the Lightning Network crashed. Did you hear about this?

Sjors Provoost Yes, today.

Aaron van Wirdum In preparation for this podcast.

Sjors Provoost Exactly.

Aaron van Wirdum Great. I'm gonna ask you all of the details so I know you've studied hard the past couple of hours to learn everything about this crash. And you're going to inform our dear listeners. Sjors, what happened?

Sjors Provoost Yeah, and I have to say, we were both surprised about how little has been written more thoroughly. Maybe we couldn't find it, but we were expecting to find very large blog posts explaining everything that went wrong and all the implications of that. But so far, we only found a couple of tweets and a few fixes, though I think it's more important than people think it is.

Aaron van Wirdum Yeah, either a blog post, some kind of post-mortem, or even sort of detailed... conversation somewhere or but yeah the only thing we could really find were some tweets so either these discussions are happening somewhere where we're not aware of it or maybe uh I don't know…

Sjors Provoost Or maybe we can get them started.

Aaron van Wirdum Let's see so Sjors, what happens? Uh

Sjors Provoost Basically a lot of Lightning nodes just stopped working uh at a certain block height and at the same time there was somebody on twitter uttering the words oh my god I made this really cool transaction right and then you know he got a lot of replies saying oh my god you broke uh LND.

Aaron van Wirdum Can we can we mention do you have the link in front of you who was this?

Sjors Provoost Yeah he's called Burak at @brqgoo with two o's

Aaron van Wirdum Okay so what did he do what did Burak do what did you do Burak?

Sjors Provoost I just did a 998 of 999 Tapscript multisig, and it only costs 490 in transaction fees.

Aaron van Wirdum Okay, let's start there.

Sjors Provoost This is about as close as... Oops, I accidentally killed it. As we have ever gotten on Bitcoin, even though he didn't kill Bitcoin.

Aaron van Wirdum What was that again? Because I remember the term. What happened there?

Sjors Provoost Oops, I accidentally killed it. It was about the destroyed smart contract that contained, I don't know, a couple hundred million dollars worth of reserves.

Aaron van Wirdum That was an Ethereum, right?

Sjors Provoost Yeah, and apparently somebody accidentally changed that smart contract into a state where nobody could touch it again.

Aaron van Wirdum Right. So an Ethereum that happened.

Sjors Provoost One of those like, I wonder what this button does. I really hope this button doesn't do what I think it does. No, they must have prevented that from happening. Click. Oops.

Aaron van Wirdum Yeah. And then everyone who had their money, their ETH in a smart contract basically lost their ETH, right? That's what happened there. Yeah. ETH were frozen. Okay. So it's not that bad in this case, luckily, although, you know, it's still a problem. So we'll get into that. But first of all, so what did he actually do? You mentioned a 908.

Sjors Provoost 998 out of 999. So it turns out with Bitcoin, you can do a multisig address, which means your multiple keys have to sign, and you can set your own threshold. So traditionally, you might say, I need two signatures out of these three public keys. But there was a limit to that based on various constraints, mostly that the way Bitcoin Core implemented, or the way Satoshi originally implemented the multisig check was extremely inefficient. And that means that if you did anything more than, say, 15 or 16, it would just become too slow to validate any of the blocks.

Aaron van Wirdum But you could do it technically?

Sjors Provoost No, there was a limit on it.

Aaron van Wirdum Do you know what the limit was?

Sjors Provoost I think it was 15 or 16.

Aaron van Wirdum Okay.

Sjors Provoost So a maximum of 15 signatures.

Aaron van Wirdum So then you could do a 14 out of 15 or 2 out of 15?

Sjors Provoost Something like that. And that might have even been like, yes, you can do 15, but you can only do 6 signatures out of that. Some constraint like that. It's not about the block size that it's using, but about the validation time going up quadratically, I think, instead of linearly.

Aaron van Wirdum Right. Okay. So that's sort of the original multisig in Bitcoin, but that was improved upon. Is that right?

Sjors Provoost Exactly. It was improved upon in Taproot. By making it much simpler, basically, instead of having one opcode called the OP_CHECKMULTISIG, it has something called OP_CHECKKEY or something like that. I forgot the name. But basically, you just tell it to check one key to check the next key, etc. And then it just adds up the number. And so what he did was basically push that to the maximum and have these 998 keys in it.

Aaron van Wirdum Okay, so that is the maximum? 999?

Sjors Provoost I'm not sure.

Aaron van Wirdum But x out of 999 I'm not sure.

Sjors Provoost Okay but it was big it was much bigger than anything we've previously seen before on the blockchain but it's perfectly valid under Taproot rules so when you're spending such a thing that means you need to put 998 signatures on the blockchain

Aaron van Wirdum I guess there are two options right either: 999 people have to all agree on how to spend the coins, in which case it looks like a very regular one sig, because that's what Taproot does, right?

Sjors Provoost If you were to use MuSig, yes. So if you use MuSig to create a sort of one key...

Aaron van Wirdum No, that's not what I'm referring to, Sjors. With Taproot, you always have the option that if everyone in the contract agrees, then you should be able to just spend the coin, the UTXO as if it's a very regular transaction, right?

Sjors Provoost You have that option.

Aaron van Wirdum So you don't have to reveal what the actual conditions were under the hood.

Sjors Provoost Ideally, yes. But that's not... You still have to do that yourself. You have to use software to create that joint signature. And that requires something like MuSig. So it's not something... Taproot doesn't solve it for you. Taproot just says you can have one signature, which is just one public key as far as the Bitcoin Core nodes are... As long as validation is concerned, it's just one public key and one signature. But you could have made it using a million keys or whatever you want. But in order to do that, you need something like MuSig or some other algorithm to make that combined signature.

Aaron van Wirdum At this point, I'm confused. Are we talking about the same thing? We might be. We might be. Because I thought MuSig was something else. But yeah, go on.

Sjors Provoost So in Taproot, you have two ways to spend coins. One is to provide a signature for basically the key path, as it's called. And that's just a public key. As far as the blockchain is concerned.

Aaron van Wirdum Yes.

Sjors Provoost And the other way is to satisfy a script. Now, in order to make that key, that public key, you have to make that public key yourself. And you can use all sorts of ways to combine different public keys into one. One of them is MuSig. So MuSig is just a way to combine signatures, regardless of how you use it. You don't have to use it in Taproot.

Aaron van Wirdum Okay I'll believe you then I thought these were two different things sorry to our listeners who already knew this that we had to go over this again but that's fine. Thanks for clarifying Sjors, yeah okay so but in this case the option that was used was not MuSig and by the way when I see MuSig I don't mean a song. We mean, how do you spell it?

Sjors Provoost Yeah, with a G sound, not with a C sound.

Aaron van Wirdum And it stands for multi-sig, I guess, right? Just short, or what does it actually stand for?

Sjors Provoost I have no idea.

Aaron van Wirdum We're getting off the track anyways. So he didn't go for the MuSig option. He went for revealing individual signatures.

Sjors Provoost Now, of course, any of these individual signatures could be a 100,000... multisig, right? It could be. Through MuSig. We don't know that. The blockchain doesn't care. It just sees 998 signatures. Each signature is 64 bytes. So this was a very large transaction. This was, I think, a megabyte or something. Something bigly. And so this is no problem because you can make these Taproot scripts can be as big as you want to. Now, it turns out that with SegWit 0, the old version of SegWit, they could not be as long as you want to. There was a limit of, I think, 10,000 bytes. And this probably had something to do with the way multisig was implemented in all the versions of Bitcoin, but I'm not sure.

Aaron van Wirdum Okay, hang on. So with SegWit, there was... What were the original SegWit, the first SegWit version?

Sjors Provoost Yeah, I mean, Taproot is technically SegWit 1, but to make it easier, yeah, with SegWit, which is SegWit 0.

Aaron van Wirdum Right, yeah.

Sjors Provoost There was a limit.

Aaron van Wirdum There was a limit on how big transactions could be.

Sjors Provoost How big the witness could be.

Aaron van Wirdum How big the witness could be.

Sjors Provoost Okay. So those signatures, you could only fit a certain number of signatures into the witness, namely 10,000 divided by 64, which is some number less than what he was doing today.

Aaron van Wirdum Right. So this would only be revealed when you're spending the coins, right?

Sjors Provoost Yes.

Aaron van Wirdum Okay.

Sjors Provoost Exactly. So...

Aaron van Wirdum So then technically, just to finish that thought, the reason I ask is so that would mean that it would be possible to receive coins that you can then never spend. I mean, you shouldn't. You determine under what conditions you receive coins.

Sjors Provoost Of course, in SegWit, you couldn't make such a multisig anyway. So that wouldn't be an issue. But yeah, it's possible. If you could send somebody coins using a set of conditions that the only way to satisfy that is to put too much stuff in the transaction, namely more than 10 kilobytes.

Aaron van Wirdum Right, yeah.

Sjors Provoost And so maybe what you can do is, you know, one multisig can only do 16, but I think you can just do one multisig and then... do another multisig, and another multisig, and another multisig, and say all of these multisigs have to be satisfied. And then you can have as many transactions as you want, and then you can't satisfy it.

Aaron van Wirdum Yeah.

Sjors Provoost Yeah.

Aaron van Wirdum Anyways, we keep being distracted by some side notes. Let's try to... This is my responsibility. We're going to stay on the track. So SegWit, zero. The original SegWit had a transaction size limit. Taproot or...

Sjors Provoost A witness limit. Witness size limit.

Aaron van Wirdum Oh, yeah. Sorry. Taproot does not. Taproot transactions do not have this limit, right?

Sjors Provoost Yeah, either it doesn't have a limit at all or it has a very large limit in the order of megabytes.

Aaron van Wirdum Right, so now it's possible to create a big transaction like this one, like the 998 signatures out of 999.

Sjors Provoost Yeah, so you're making a very big input of a transaction. And now this is not a problem for Bitcoin Core nodes. It's also not a problem according to the rules of Taproot. However, btcd had an implementation mistake, so we should explain what btcd is.

Aaron van Wirdum Yes.

Sjors Provoost btcd is another implementation of Bitcoin, essentially. There are a couple of implementations. There's Libbitcoin, there's Bitcoin Core, there is Bitcoin Rust, although I don't know if it's a full implementation. There is...

Aaron van Wirdum Bitcoin, I don't know.

Sjors Provoost Yeah, Bitcoin is in JavaScript, I think so, yeah. There's a bunch of them, and some of them implement Taproot, others don't. And basically, all these nodes implement consensus code from scratch. That's always a risk. Though I think Libbitcoin will actually use the Bitcoin part of the consensus code. But in any case, they've all implemented themselves, so they all have to be very careful to make sure that they copy everything in the spec or every bug that's in the Bitcoin Core source code. Otherwise, problems happen. And in this case, in btcd, was... incorrectly checking that this thing was less than 10 kilobyte, because that was a rule for the old version of SegWit and not for the new versions of SegWit. And so it would simply reject the whole block because it thought the transaction was invalid. So it would just freeze at that cool block.

Aaron van Wirdum Right. So btcd was applying a rule that really only applied to SegWit. Let's keep it simple. It applied to SegWit, and it was applying it to Taproot. I'm simplifying, but that's basically what happened, right?

Sjors Provoost And slightly under the hood, btcd has at least two places where it checks certain rules. One is when it receives a block over the wire, and we'll kind of get into later what receiving over the wire means. But basically when it receives a block from another peer, it does some preliminary checks on it. And then once those checks are passed, it goes to a full validation check. Now, that full validation will happen even if you feed the block manually, if you copy-paste it or something like that. But these preliminary checks should only happen when they receive the block over the network. And it was those checks that were enforcing this 10-kilobyte limit on all versions of SegWit, essentially, not just on SegWit 0. Because the way Bitcoin SegWit has been designed is there's multiple versions of SegWit. And if the version number is more than zero, then in the beginning, you would always approve the transaction. It would always be spendable. You would simply not check any rules, including rules about size limits.

Aaron van Wirdum Yeah. So I would propose from now on, let's just say SegWit and Taproot, even though technically it's SegWit zero and SegWit one, right?

Sjors Provoost Yeah, in this case, because it's about the witness. And every version of SegWit will have a witness, but the rules for that witness will change for every version of SegWit. So with regular SegWit, there was a limit of 10,000, but there was no limit defined for any other version of SegWit.

Aaron van Wirdum Yeah.

Sjors Provoost So it turns out this...

Aaron van Wirdum Okay, so btcd, they did the same check kind of twice. And you mentioned that the first check happens when they receive it over the wire. And then you said, we'll get into that. So let's get into that. What do you mean by that?

Sjors Provoost Yeah, and presumably they check it again in the code that checks the consensus. And there they were probably doing it correctly. So in the code that really, you know, once they've received the block and put it in their own database, it might say, okay, is this block actually obeying SegWit? And then it would do it correctly. It would say, what's the version of SegWit? Okay, version number zero. Okay, that's the limit. Or if it was Taproot, it would check some other limit.

Aaron van Wirdum Okay. btcd is an alternative Bitcoin implementation. However, we were talking about Lightning. So how does this tie in with Lightning then?

Sjors Provoost Yeah. So first I'll describe sort of an imaginary scenario because I don't think this actually exists in the wild or at least not very often. Let's say you're running LND, which is the Lightning implementation built by the same people who also built btcd.

Aaron van Wirdum So that's Lightning Labs, right?

Sjors Provoost Yeah. Now you can use Lightning LND using different node software. If you were using it with btcd, then this bug would very clearly occur because your btcd node is talking to other nodes in the network might be Bitcoin Core nodes it sees this block it receives it over the wire and it says this is wrong and then it stops syncing

Aaron van Wirdum You still haven't explained what over the wire is, by the way.

Sjors Provoost Yeah, just over the internet, something like that. So it basically rejects it that way. And then how would this impact Lightning? Your Lightning node wants to see all the new blocks in order to make sure that its channels are still okay, nobody's trying to cheat on it, etc. And so if your Lightning node is not seeing any new transactions anymore, it's going to get increasingly unhappy. At some point, it may even be talking to its peers on the Lightning Network, and its peers are saying, no, no, no, there are much newer blocks than what you're talking about. Why don't you follow the newest blocks? And maybe the peers will say, hey, the fees have gone up. Why don't you increase the fees on the penalty transactions? Because Lightning nodes will constantly renegotiate what the penalty transactions should be, etc. But you're noticing, no, no, no, the blockchain hasn't moved. Why are you talking about these higher fees? And at some point saying, fuck you. I'm going to disconnect from you. I'm going to close all the channels now. And so some people on the internet were reporting that they were seeing their channels getting closed by the other side because of some miscommunication because their node was basically stuck. And that can cascade, of course, if lots of nodes start closing the channels at the same time.

Aaron van Wirdum Okay, yeah. So we were explaining or you were explaining that if LND uses btcd to read the blockchain, to get the blockchain, then the problem is obvious. And then you hinted at some problems.

Sjors Provoost Yeah.

Aaron van Wirdum So do all LND nodes use btcd? Is this a requirement for LND?

Sjors Provoost I think they... They do under some circumstances, and they don't under others. So generally, most people, I think, run Bitcoin Core as the node. However, if you're running a mobile app, then you may want to not download all the blocks from your mobile app. You want to use these filters, BIP158, Neutrino filters that we talked about in some other episode. So those filters basically give you a quick summary of the block, and then you only request the blocks that you want. And that also, under the hood, is btcd. So what's possible on the mobile app is that your btcd node on the mobile app will get stuck because of this. Because the mobile app will fetch a block, it will fetch the magic bad block, because you happen to have asked for that one, and then your mobile app will stop too. But that does not really explain what we saw people remark. Because people were complaining that, first of all, they were complaining that their node stopped, their Lightning node stopped. And I don't think a mobile app would necessarily stop, but maybe it would. But they were also complaining about channels getting closed. And mobile apps tend to be very patient in this stuff. Like you can have your, I don't know, your Phoenix wallet or Breez wallet or whatever. You can ignore it for weeks and it'll start annoying you, warn you, but it generally doesn't start closing channels on you. So that doesn't seem to be what's going on. When people have forced closes of their channels, I think they're actually running their own node. They might be using something like Umbrel or something. And at the same time, today I saw on Twitter, on October 10th, Umbrel said, hey guys, there's a new update. Please install this hotfix because there's a problem with Lightning. So then my question is, why would Umbrel nodes have a problem with this? And this is where I would have loved to have read a good postmortem because now I have to guess because I don't know exactly how Umbrel works. But as far as I know, software like Umbrel uses Bitcoin Core to download all the blocks. So when people say over the wire... I would assume that the wire is Bitcoin Core receiving blocks from other nodes out there. And Bitcoin Core doesn't have this bug. So then the question is, why are we still running into this bug, even though we're using Bitcoin Core to validate the blocks? You know, the Lightning software shouldn't be doing extra consensus checks. So this is where I think what's going on is that on your little node, you have Bitcoin Core running and you have LND running. But LND, under the hood, uses bits and pieces of btcd. It just uses some of the building blocks from btcd. And one of those building blocks...

Aaron van Wirdum For example, for Neutrino.

Sjors Provoost Yeah, for Neutrino. And then, again, there's two sort of paths that I would imagine could have gone wrong. One path is that the node, the LND software, is talking to your full node, not directly in the sense it is talking using these filters. So it might be asking your node, so it's on the same device, but your LND software is asking the node for the filters on the same device, and then given the filters, it asks the node for the blocks. And those blocks are still going over a wire. Even though the wire is inside your own computer, it's still a wire from the point of view of the software. And so then it would also run into this bug where the block briefly goes on the wire from your Bitcoin Core node to this little piece of btcd, really. And so that's a problem because you don't want to have these extra valid... You don't want to be reliant on some other implementation than Bitcoin Core for your consensus checks because there could be other bugs in there. That's one possibility. The other possibility is that it's even different there. Maybe LND talks directly to Bitcoin Core and just asks for every block, which would make sense. I mean, Lightning nodes, if it's on the same machine anyway, it might as well just ask every block from Bitcoin Core, just in raw form. And that could also still be interpreted as if it was going over the wire. So maybe it was still running those same checks, even though there was nothing going over a wire. In any case, somehow some of the btcd consensus code is, you know, network code, but network is kind of, you know, important for consensus too. Somehow it's getting run, even though you're not running btcd as the node, which is kind of worrying from an architectural point of view. Because again, in general, other implementations than Bitcoin Core can cause problems when they have even the slightest different interpretation of what the rules are.

Aaron van Wirdum Right okay so we so to summarize that we know what the bug in btcd was what we're unsure about and we've tried to look for it is why it affected the Lightning network or to the extent that it did why it affects LND nodes.

Sjors Provoost Why why it affected nodes even though you'd think they were not running btcd at least not the full version of btcd right so that's that's an open question I think we've we've postulated some possibilities I would also observe…

Aaron van Wirdum Someone has the answer.

Sjors Provoost Let us know.

Aaron van Wirdum Let us know.

Sjors Provoost We'll mention maybe by the time this airs it's already been explained by somebody and they're like oh you guys are wrong you didn't read this post it's because you wrote it after we recorded it so one more thing is that I think some people framed this as a Taproot-related bug, but it really wasn't, because I work on this other project called Fork Monitor, which is by BitMEX Research, and Fork Monitor checks different nodes to see if they have any problems compared to each other. And because I was also busy with the conference, I completely ignored what Fork Monitor was doing, but it actually turned out that Fork Monitor, the btcd node on Fork Monitor, was also stuck at this magical block, or one block before the magical block. But just nobody noticed it. Or maybe people did notice it and told me about it, and I ignored it anyway because I didn't know there was this drama going on.

Aaron van Wirdum Busy partying in Amsterdam.

Sjors Provoost Exactly. That has to happen. But basically, this particular node and fork monitor has not been updated for a while because I was being extremely lazy. It was running a very old version. Actually, it was running a version that was from before they had Taproot support. So the node that stopped did not support Taproot, which means this bug is not related to Taproot at all. And in fact, somebody could have triggered this bug before Taproot was even created. Depending on when the bug was introduced, probably when SegWit was introduced, I'm guessing this check was written when SegWit was introduced. So this bug has been sitting there for like five years.

Aaron van Wirdum But then someone had to make a SegWit version one transaction before there was a SegWit version one.

Sjors Provoost Yes, but that's possible. And we've talked about this in the earlier episode about accidental hard forks or what is a hard fork or what isn't a hard fork. You can use features that don't exist yet in Bitcoin Core. And the thing is, you're giving away your money, but you can give away your money in a way that, you know, causes other nodes to crash or causes half the Lightning network to crash in this case. Or maybe not half the network. I don't know how big the explosion was. This is why I'd like to see a postmortem. Like how many nodes actually went down because of this?

Aaron van Wirdum Right. Yes, me too. And so are we done with this part? Are we done with the speculation part?

Sjors Provoost Yep, I think so.

Aaron van Wirdum Okay, so then I guess to sort of close this off. And you did allude to it already a little bit but what is actually the problem or the harm or how does this affect users so to say because I would assume that at least initially a Lightning node would actually continue to operate like normal even if it's not on the latest block because it doesn't require blockchain confirmations to make transactions or to forward payments, right?

Sjors Provoost Yeah, but a lot of it depends on the internal details. I think one of the tweets I read was saying that nodes couldn't open new channels or close them, which you think should be possible, right? Because you don't need to wait for a new block to close a channel. But there may be some implementation detail that causes those two things to depend on each other. Even though that's not intentional. So that's not really related, I guess. I mean, that's more of a matter of robustness. Like what does a node do when it stops syncing? What does a Lightning node do when the underlying node stops? That needs to be thought through a bit. Because one of the things that could go wrong is, let's say somebody's trying to cheat on you. And then it depends on all your settings. If you set things that it waits for a month and you can punish the cheater for a whole month, then as long as you figure out there's a problem within a couple of days and upgrade, you're fine. But let's say you've really tightened all the screws to maximize your fee revenue or whatever. And you've set it so that if you don't respond within 12 hours, let's say. Now, let's say 24 hours that's a bit of a better example let's say you don't respond within 24 hours if somebody can cheat on you that means you meet you need to make sure your your Lightning node is not down for 24 hours

Aaron van Wirdum Yeah so to just to clarify this real quick anyone who knows a bit about Lightning knows this but maybe some of our listeners don't so the idea with Lightning is you're updating the balance between each other people have balances with each other and what's stopping people from out from broadcasting older channel states where they get more money than the latest channel state is that you can respond by basically taking all of the coins, right?

Sjors Provoost I think so, at least a fair chunk of it. Yeah, so it's a deterrent.

Aaron van Wirdum Yes, it's a deterrent. However, in order to see if someone is broadcasting an old transaction or if someone is trying to steal from you, you kind of need to know what's going on on the blockchain. So if you can't see that, you can't respond and then someone can steal from you. That's a problem you're explaining here, right?

Sjors Provoost Exactly. Now, to add some nuance to that, it doesn't have to be in the blockchain. You can look at the mempool. We've explained the mempool in earlier episodes. Typically, your mempool is set to 300 megabytes. That's the default on Bitcoin Core.

Aaron van Wirdum Hang on, you say you can look at the mempool, but...

Sjors Provoost You can see if somebody's cheating by looking at the cheat transaction inside the mempool.

Aaron van Wirdum Yeah, but you can't see it with your eyes, right? So I would assume that software is written in a certain way. Do they look at the mempool?

Sjors Provoost Yeah, so that's another question I don't know the answer to. I think it would be wise if your Lightning software checks what's happening in the mempool. I don't know if that means it should immediately respond to it. Maybe that's overzealous, you know, because maybe it was a mistake and the other side corrects it with, you know, bumps it into a non-[unclear]. I don't know what the game theory is there. But if we run through this example, let's say, so let's say your node is no longer catching up with blocks because some exploit caused it to stop. It caused it to stop syncing new blocks. Then that means you will see transactions appear in the mempool. And your mempool will get very big very quickly. If it's two megabytes per block within 150 blocks, you have a full mempool, and any new transaction that comes in is probably going to be ignored. It kind of overflows. Depends, maybe it's paying a high fee, maybe not. That means that's about a day, 150 blocks. So that means if your timeouts are set to about a day, you could be in a situation where somebody's cheating on you. That transaction is in other people's mempool, but not in your mempool because your mempool is already overflowed through the bug.

Aaron van Wirdum Yeah, and because there's no new blocks coming in from your perspective. So every transaction just stays in the mempool.

Sjors Provoost So unless you have a giant alarm system, you might not even know that this is going on and you've already been cheated. But let's say there is a big alarm system, then you have another problem because you still have this one-day deadline. So even though you know there's a problem, you can't act because the node is broken and you don't know how to fix your own node because you'd have to be able to write consensus code. So if there's not a patch within that 24 hours, you're still in trouble. So that's kind of one of the things I would be worried about here. Now, that's not a problem if you set the timeouts as something more reasonable. But, yeah, if you're reliant on somebody making a hot fix within 24 hours while you're on a plane, it's not good. So, you know, maybe this is completely exaggerated, but I'm just trying to illustrate why these things could be bad and why I think it's good to have a good post-mortem on thinking, in general, thinking through all these scenarios. And maybe the Lightning developers are thinking through all those scenarios.

Aaron van Wirdum Yeah, I was going to say, I mean, it might be a hard attack to pull off or a niche risk, but you've got to think about it, right?

Sjors Provoost Yeah, I think so.

Aaron van Wirdum Okay. What about routing? Would routing be affected? I think routing was affected. I'm not sure how, but I think there were less routes available, I guess, because nodes were going down.

Sjors Provoost It could be the same reason. It could be that a node says, hey, I'm too far behind the real world, or I'm not seeing any new blocks. So generally, nodes have a heuristic. At least Bitcoin Core has a heuristic. I think that if you don't see a new block within two hours, some alarm bells go off internally. It tries to make more connections. Perhaps Lightning software has similar things that after a few hours of no update, it says, you know what? I'm not sure what's going on in the world. I'm going to stop forwarding payments now. So maybe it just refuses to forward any new payments until it's up to date again. Things like that could happen.

Aaron van Wirdum Right.

Sjors Provoost So that would definitely cause some systemic effects where you would certainly find that all your routes are not working or a lot of them because some of them are running as a piece of software that's down.

Aaron van Wirdum Yep. Okay Sjors, is that it?

Sjors Provoost I think so.

Aaron van Wirdum Yeah I think so. So clearly this was a little bit of a guessing episode or speculation informed speculation how you want to call it we're trying to figure out what went wrong and we weren't exactly sure and we've we weren't really finding the answer we're looking for so again if anyone knows more than us Let us know. Maybe we'll cover it in the next episode.

Sjors Provoost Who knows?

Aaron van Wirdum We'll definitely mention it in the next episode. I don't know if we'll do a whole other episode about it. But please let us know if you know anything more. Sjors, is Lightning still reckless?

Sjors Provoost It depends on what you're doing. I think generally it's fine.

Aaron van Wirdum Such a nuanced answer. Just say yes. Reckless. Lightning is reckless. Everyone's reckless. That's what we want to hear.

Sjors Provoost It's not too bad. I mean, Bitcoin is still an experiment, right? So I guess that's all. Then thank you for listening to Bitcoin.

Aaron van Wirdum Explained.