WEBVTT - Automatic transcript; recognition may contain errors. Reviewed for technical wording and speaker labels. Times match the published episode. 00:00:20.500 --> 00:00:22.880 Live from Utrecht, this is Bitcoin. 00:00:23.700 --> 00:00:24.380 Explained. 00:00:25.200 --> 00:00:26.620 Sjors, I have horrible news. 00:00:27.020 --> 00:00:28.440 Oh my god, tell me. 00:00:29.920 --> 00:00:30.500 No one, 00:00:31.960 --> 00:00:37.660 zero people, a great nil, a great total of nil people used the 00:00:37.660 --> 00:00:41.670 promo code Sjorsnado. From what I understand. 00:00:42.030 --> 00:00:42.930 That's unsurprising. 00:00:43.840 --> 00:00:48.540 I don't know what that means. It either means we only have hardcore fans that 00:00:48.540 --> 00:00:54.240 already bought their ticket before we even gave out the code. Or it means no one 00:00:54.240 --> 00:00:55.300 can remember your name. 00:00:55.950 --> 00:00:59.550 Or it means they used the other promo code at the end of the show that was 00:00:59.550 --> 00:01:00.970 spliced in by the editor. 00:01:01.600 --> 00:01:03.400 Oh, was there another one spliced in? 00:01:03.560 --> 00:01:03.700 Yeah. 00:01:03.880 --> 00:01:07.220 You listen to our shows. I sometimes do, but I didn't notice that. Oh, that 00:01:07.220 --> 00:01:09.260 wasn't easier. That one was easier to remember? 00:01:09.500 --> 00:01:09.820 Probably. 00:01:10.300 --> 00:01:14.400 Oh, okay. Okay. That's a bit sad. Or just no one listens to our podcast. That's 00:01:14.400 --> 00:01:15.260 option three, I guess. 00:01:15.320 --> 00:01:18.080 No, I think some people listen because some people at the conference talk to me 00:01:18.080 --> 00:01:20.400 as well that they listened and that they read the book. 00:01:21.980 --> 00:01:23.940 Which book is this, Sjors? Do you want to shill it? 00:01:24.100 --> 00:01:26.700 Yeah, it's a book I wrote called Bitcoin, a work in progress. 00:01:27.510 --> 00:01:33.270 Very well. Sjors, I have more horrible news. The Lightning Network crashed last 00:01:33.270 --> 00:01:37.590 week. I know we were both at Bitcoin Amsterdam, or at least maybe it was a couple 00:01:37.590 --> 00:01:41.370 of days before that. We were in full preparation of Bitcoin Amsterdam. Our mind 00:01:41.370 --> 00:01:45.830 was occupied, but the Lightning Network crashed. Did you hear about this? 00:01:46.370 --> 00:01:47.110 Yes, today. 00:01:47.950 --> 00:01:49.850 In preparation for this podcast. 00:01:50.770 --> 00:01:51.250 Exactly. 00:01:51.690 --> 00:01:57.420 Great. I'm gonna ask you all of the details so I know you've studied 00:01:57.420 --> 00:02:01.380 hard the past couple of hours to learn everything about this crash. 00:02:02.650 --> 00:02:05.990 And you're going to inform our dear listeners. Sjors, what happened? 00:02:06.390 --> 00:02:10.910 Yeah, and I have to say, we were both surprised about how little has been written 00:02:10.910 --> 00:02:15.410 more thoroughly. Maybe we couldn't find it, but we were expecting to find very 00:02:15.410 --> 00:02:19.850 large blog posts explaining everything that went wrong and all the implications 00:02:19.850 --> 00:02:24.290 of that. But so far, we only found a couple of tweets and a few fixes, though I 00:02:24.290 --> 00:02:26.130 think it's more important than people think it is. 00:02:26.550 --> 00:02:30.850 Yeah, either a blog post, some kind of post-mortem, or even sort of detailed... 00:02:31.740 --> 00:02:35.700 conversation somewhere or but yeah the only thing we could really find were some 00:02:35.700 --> 00:02:40.520 tweets so either these discussions are happening somewhere where we're not aware 00:02:40.520 --> 00:02:44.350 of it or maybe uh I don't know… 00:02:44.350 --> 00:02:45.930 Or maybe we can get them started. 00:02:45.930 --> 00:02:49.820 Let's see so Sjors, what happens? Uh 00:02:49.820 --> 00:02:55.560 Basically a lot of Lightning nodes just stopped working uh at a certain block 00:02:55.560 --> 00:03:00.890 height and at the same time there was somebody on twitter uttering the words oh 00:03:00.890 --> 00:03:05.890 my god I made this really cool transaction right and then you know he got a lot 00:03:05.890 --> 00:03:08.650 of replies saying oh my god you broke uh LND. 00:03:08.650 --> 00:03:13.010 Can we can we mention do you have the link in front of you who was this? 00:03:13.170 --> 00:03:18.870 Yeah he's called Burak at @brqgoo 00:03:18.870 --> 00:03:20.490 with two o's 00:03:20.490 --> 00:03:25.290 Okay so what did he do what did Burak do what did you do Burak? 00:03:25.740 --> 00:03:31.080 I just did a 998 of 999 Tapscript 00:03:31.080 --> 00:03:34.820 multisig, and it only costs 490 in transaction fees. 00:03:35.570 --> 00:03:36.970 Okay, let's start there. 00:03:37.150 --> 00:03:41.530 This is about as close as... Oops, I accidentally killed it. As we have ever 00:03:41.530 --> 00:03:43.910 gotten on Bitcoin, even though he didn't kill Bitcoin. 00:03:44.390 --> 00:03:47.470 What was that again? Because I remember the term. What happened there? 00:03:47.490 --> 00:03:51.570 Oops, I accidentally killed it. It was about the destroyed smart contract that 00:03:51.570 --> 00:03:55.310 contained, I don't know, a couple hundred million dollars worth of reserves. 00:03:55.630 --> 00:03:56.770 That was an Ethereum, right? 00:03:56.970 --> 00:04:01.510 Yeah, and apparently somebody accidentally changed that smart contract into a 00:04:01.510 --> 00:04:02.770 state where nobody could touch it again. 00:04:03.360 --> 00:04:04.960 Right. So an Ethereum that happened. 00:04:05.240 --> 00:04:08.060 One of those like, I wonder what this button does. I really hope this button 00:04:08.060 --> 00:04:10.640 doesn't do what I think it does. No, they must have prevented that from 00:04:10.640 --> 00:04:11.980 happening. Click. Oops. 00:04:12.680 --> 00:04:16.820 Yeah. And then everyone who had their money, their ETH in a smart contract 00:04:16.820 --> 00:04:20.040 basically lost their ETH, right? That's what happened there. Yeah. ETH were 00:04:20.040 --> 00:04:24.500 frozen. Okay. So it's not that bad in this case, luckily, although, you know, 00:04:24.580 --> 00:04:27.780 it's still a problem. So we'll get into that. But first of all, so what did he 00:04:27.780 --> 00:04:31.420 actually do? You mentioned a 908. 00:04:32.230 --> 00:04:37.650 998 out of 999. So it turns out with Bitcoin, you can do a multisig address, 00:04:37.870 --> 00:04:41.690 which means your multiple keys have to sign, and you can set your own threshold. 00:04:41.950 --> 00:04:45.950 So traditionally, you might say, I need two signatures out of these three public 00:04:45.950 --> 00:04:50.930 keys. But there was a limit to that based on various constraints, mostly that the 00:04:50.930 --> 00:04:56.090 way Bitcoin Core implemented, or the way Satoshi originally implemented the 00:04:56.090 --> 00:04:58.750 multisig check was extremely inefficient. 00:05:00.160 --> 00:05:05.500 And that means that if you did anything more than, say, 15 or 16, it would just 00:05:05.500 --> 00:05:07.420 become too slow to validate any of the blocks. 00:05:07.540 --> 00:05:08.820 But you could do it technically? 00:05:09.060 --> 00:05:09.940 No, there was a limit on it. 00:05:09.960 --> 00:05:11.060 Do you know what the limit was? 00:05:11.060 --> 00:05:12.140 I think it was 15 or 16. 00:05:12.400 --> 00:05:12.600 Okay. 00:05:12.940 --> 00:05:14.740 So a maximum of 15 signatures. 00:05:14.960 --> 00:05:18.240 So then you could do a 14 out of 15 or 2 out of 15? 00:05:18.420 --> 00:05:21.860 Something like that. And that might have even been like, yes, you can do 15, but 00:05:21.860 --> 00:05:25.200 you can only do 6 signatures out of that. Some constraint like that. 00:05:26.480 --> 00:05:30.310 It's not about the block size that it's using, but about the validation time 00:05:30.310 --> 00:05:33.490 going up quadratically, I think, instead of linearly. 00:05:33.870 --> 00:05:38.370 Right. Okay. So that's sort of the original multisig in Bitcoin, but that was 00:05:38.370 --> 00:05:40.110 improved upon. Is that right? 00:05:40.350 --> 00:05:44.570 Exactly. It was improved upon in Taproot. By making it much simpler, basically, 00:05:44.730 --> 00:05:50.410 instead of having one opcode called the OP_CHECKMULTISIG, it has something 00:05:50.410 --> 00:05:54.590 called OP_CHECKKEY or something like that. I forgot the name. But basically, you 00:05:54.590 --> 00:06:00.370 just tell it to check one key to check the next key, etc. And then it just adds 00:06:00.370 --> 00:06:06.040 up the number. And so what he did was basically push that to the maximum and 00:06:06.040 --> 00:06:08.900 have these 998 keys in it. 00:06:09.040 --> 00:06:11.880 Okay, so that is the maximum? 999? 00:06:12.510 --> 00:06:13.270 I'm not sure. 00:06:13.270 --> 00:06:15.710 But x out of 999 I'm not sure. 00:06:15.710 --> 00:06:19.410 Okay but it was big it was much bigger than anything we've previously seen before 00:06:19.410 --> 00:06:23.570 on the blockchain but it's perfectly valid under Taproot rules so when you're 00:06:23.570 --> 00:06:28.470 spending such a thing that means you need to put 998 signatures on the blockchain 00:06:28.470 --> 00:06:30.850 I guess there are two options right either: 00:06:32.430 --> 00:06:37.090 999 people have to all agree on how to spend the coins, in which case it looks 00:06:37.090 --> 00:06:40.530 like a very regular one sig, because that's what Taproot does, right? 00:06:40.530 --> 00:06:46.230 If you were to use MuSig, yes. So if you use MuSig to create a sort of 00:06:46.230 --> 00:06:47.930 one key... 00:06:47.930 --> 00:06:52.550 No, that's not what I'm referring to, Sjors. With Taproot, you always have the 00:06:52.550 --> 00:06:56.450 option that if everyone in the contract agrees, then you should be able to just 00:06:56.450 --> 00:07:00.690 spend the coin, the UTXO as if it's a very regular transaction, right? 00:07:00.770 --> 00:07:01.210 You have that option. 00:07:01.210 --> 00:07:04.630 So you don't have to reveal what the actual conditions were under the hood. 00:07:04.950 --> 00:07:08.650 Ideally, yes. But that's not... You still have to do that yourself. You have to 00:07:08.650 --> 00:07:12.770 use software to create that joint signature. And that requires something like MuSig. 00:07:14.060 --> 00:07:17.100 So it's not something... Taproot doesn't solve it for you. Taproot just says you 00:07:17.100 --> 00:07:21.580 can have one signature, which is just one public key as far as the Bitcoin Core 00:07:21.580 --> 00:07:26.540 nodes are... As long as validation is concerned, it's just one public key and one 00:07:26.540 --> 00:07:30.800 signature. But you could have made it using a million keys or whatever you want. 00:07:30.960 --> 00:07:34.920 But in order to do that, you need something like MuSig or some other algorithm to 00:07:34.920 --> 00:07:36.880 make that combined signature. 00:07:37.340 --> 00:07:42.640 At this point, I'm confused. Are we talking about the same thing? We might be. We 00:07:42.640 --> 00:07:46.420 might be. Because I thought MuSig was something else. But yeah, go on. 00:07:46.500 --> 00:07:50.680 So in Taproot, you have two ways to spend coins. One is to provide a signature 00:07:50.680 --> 00:07:56.560 for basically the key path, as it's called. And that's just a public key. 00:07:57.170 --> 00:07:58.590 As far as the blockchain is concerned. 00:07:58.830 --> 00:07:58.990 Yes. 00:07:59.170 --> 00:08:03.270 And the other way is to satisfy a script. Now, in order to make that key, that 00:08:03.270 --> 00:08:08.090 public key, you have to make that public key yourself. And you can use all sorts 00:08:08.090 --> 00:08:12.710 of ways to combine different public keys into one. One of them is MuSig. So MuSig 00:08:12.710 --> 00:08:16.350 is just a way to combine signatures, regardless of how you use it. You don't have 00:08:16.350 --> 00:08:17.150 to use it in Taproot. 00:08:19.280 --> 00:08:23.060 Okay I'll believe you then I thought these were two different things sorry to our 00:08:23.060 --> 00:08:26.340 listeners who already knew this that we had to go over this again but that's 00:08:26.340 --> 00:08:31.180 fine. Thanks for clarifying Sjors, yeah okay so but in this case the option that 00:08:31.180 --> 00:08:37.140 was used was not MuSig and by the way when I see MuSig I don't mean a song. 00:08:37.440 --> 00:08:39.040 We mean, how do you spell it? 00:08:39.480 --> 00:08:42.660 Yeah, with a G sound, not with a C sound. 00:08:42.900 --> 00:08:47.180 And it stands for multi-sig, I guess, right? Just short, or what does it actually 00:08:47.180 --> 00:08:47.620 stand for? 00:08:47.400 --> 00:08:48.000 I have no idea. 00:08:48.040 --> 00:08:53.520 We're getting off the track anyways. So he didn't go for the MuSig option. He 00:08:53.520 --> 00:08:56.120 went for revealing individual signatures. 00:08:56.560 --> 00:09:01.940 Now, of course, any of these individual signatures could be a 100,000... 00:09:02.850 --> 00:09:07.610 multisig, right? It could be. Through MuSig. We don't know that. The blockchain 00:09:07.610 --> 00:09:13.550 doesn't care. It just sees 998 signatures. Each signature is 64 bytes. So 00:09:13.550 --> 00:09:16.650 this was a very large transaction. This was, I think, a megabyte or something. 00:09:18.630 --> 00:09:19.570 Something bigly. 00:09:20.850 --> 00:09:26.170 And so this is no problem because you can make these Taproot scripts can be as 00:09:26.170 --> 00:09:30.130 big as you want to. Now, it turns out that with SegWit 0, the old version of 00:09:30.130 --> 00:09:33.290 SegWit, they could not be as long as you want to. There was a limit of, I think, 00:09:33.350 --> 00:09:34.390 10,000 bytes. 00:09:35.930 --> 00:09:40.170 And this probably had something to do with the way multisig was implemented in 00:09:40.170 --> 00:09:42.230 all the versions of Bitcoin, but I'm not sure. 00:09:42.490 --> 00:09:46.510 Okay, hang on. So with SegWit, there was... What were the original SegWit, the 00:09:46.510 --> 00:09:47.750 first SegWit version? 00:09:47.890 --> 00:09:51.050 Yeah, I mean, Taproot is technically SegWit 1, but to make it easier, yeah, with 00:09:51.050 --> 00:09:52.350 SegWit, which is SegWit 0. 00:09:52.490 --> 00:09:52.950 Right, yeah. 00:09:53.380 --> 00:09:54.240 There was a limit. 00:09:54.300 --> 00:09:56.900 There was a limit on how big transactions could be. 00:09:57.040 --> 00:09:58.500 How big the witness could be. 00:09:58.660 --> 00:09:59.640 How big the witness could be. 00:09:59.680 --> 00:10:03.420 Okay. So those signatures, you could only fit a certain number of signatures into 00:10:03.420 --> 00:10:09.240 the witness, namely 10,000 divided by 64, which is some number less than what he 00:10:09.240 --> 00:10:09.900 was doing today. 00:10:10.220 --> 00:10:14.060 Right. So this would only be revealed when you're spending the coins, right? 00:10:14.460 --> 00:10:14.740 Yes. 00:10:14.980 --> 00:10:15.300 Okay. 00:10:15.860 --> 00:10:16.180 Exactly. 00:10:17.540 --> 00:10:18.040 So... 00:10:18.500 --> 00:10:23.160 So then technically, just to finish that thought, the reason I ask is so that 00:10:23.160 --> 00:10:28.390 would mean that it would be possible to receive coins that you can then never 00:10:28.390 --> 00:10:32.530 spend. I mean, you shouldn't. You determine under what conditions you receive coins. 00:10:32.570 --> 00:10:36.990 Of course, in SegWit, you couldn't make such a multisig anyway. So that wouldn't 00:10:36.990 --> 00:10:42.210 be an issue. But yeah, it's possible. If you could send somebody coins using a 00:10:42.210 --> 00:10:46.710 set of conditions that the only way to satisfy that is to put too much stuff in 00:10:46.710 --> 00:10:48.470 the transaction, namely more than 10 kilobytes. 00:10:48.610 --> 00:10:48.890 Right, yeah. 00:10:49.190 --> 00:10:53.670 And so maybe what you can do is, you know, one multisig can only do 16, but I 00:10:53.670 --> 00:10:58.230 think you can just do one multisig and then... do another multisig, and another 00:10:58.230 --> 00:11:01.030 multisig, and another multisig, and say all of these multisigs have to be 00:11:01.030 --> 00:11:04.130 satisfied. And then you can have as many transactions as you want, and then you 00:11:04.130 --> 00:11:04.930 can't satisfy it. 00:11:05.210 --> 00:11:05.350 Yeah. 00:11:05.610 --> 00:11:05.750 Yeah. 00:11:06.050 --> 00:11:11.590 Anyways, we keep being distracted by some side notes. Let's try to... This is my 00:11:11.590 --> 00:11:16.410 responsibility. We're going to stay on the track. So SegWit, zero. The original 00:11:16.410 --> 00:11:20.430 SegWit had a transaction size limit. Taproot or... 00:11:20.430 --> 00:11:22.350 A witness limit. Witness size limit. 00:11:22.430 --> 00:11:22.910 Oh, yeah. Sorry. 00:11:24.210 --> 00:11:28.810 Taproot does not. Taproot transactions do not have this limit, right? 00:11:29.030 --> 00:11:32.490 Yeah, either it doesn't have a limit at all or it has a very large limit in the 00:11:32.490 --> 00:11:33.250 order of megabytes. 00:11:33.490 --> 00:11:37.470 Right, so now it's possible to create a big transaction like this one, like the 00:11:37.470 --> 00:11:40.510 998 signatures out of 999. 00:11:40.830 --> 00:11:45.630 Yeah, so you're making a very big input of a transaction. And now this is not a 00:11:45.630 --> 00:11:49.610 problem for Bitcoin Core nodes. It's also not a problem according to the rules of Taproot. 00:11:50.820 --> 00:11:55.360 However, btcd had an implementation mistake, so we should explain what btcd is. 00:11:55.480 --> 00:11:55.680 Yes. 00:11:56.400 --> 00:12:00.780 btcd is another implementation of Bitcoin, essentially. There are a couple of 00:12:00.780 --> 00:12:04.960 implementations. There's Libbitcoin, there's Bitcoin Core, there is Bitcoin Rust, 00:12:05.120 --> 00:12:08.900 although I don't know if it's a full implementation. There is... 00:12:08.900 --> 00:12:09.660 Bitcoin, I don't know. 00:12:09.780 --> 00:12:13.500 Yeah, Bitcoin is in JavaScript, I think so, yeah. There's a bunch of them, and 00:12:13.500 --> 00:12:15.440 some of them implement Taproot, others don't. 00:12:17.520 --> 00:12:21.640 And basically, all these nodes implement consensus code from scratch. That's 00:12:21.640 --> 00:12:26.040 always a risk. Though I think Libbitcoin will actually use the Bitcoin part of 00:12:26.040 --> 00:12:30.040 the consensus code. But in any case, they've all implemented themselves, so they 00:12:30.040 --> 00:12:33.880 all have to be very careful to make sure that they copy everything in the spec or 00:12:33.880 --> 00:12:38.600 every bug that's in the Bitcoin Core source code. Otherwise, problems happen. And 00:12:38.600 --> 00:12:44.390 in this case, in btcd, was... incorrectly checking 00:12:44.390 --> 00:12:49.120 that this thing was less than 10 kilobyte, because that was a rule for the old 00:12:49.120 --> 00:12:53.120 version of SegWit and not for the new versions of SegWit. And so it would simply 00:12:53.120 --> 00:12:56.870 reject the whole block because it thought the transaction was invalid. So it 00:12:56.870 --> 00:12:58.990 would just freeze at that cool block. 00:12:59.310 --> 00:13:05.170 Right. So btcd was applying a rule that really only applied to SegWit. 00:13:05.290 --> 00:13:08.650 Let's keep it simple. It applied to SegWit, and it was applying it to Taproot. 00:13:09.030 --> 00:13:11.750 I'm simplifying, but that's basically what happened, right? 00:13:11.810 --> 00:13:17.610 And slightly under the hood, btcd has at least two places where it 00:13:17.610 --> 00:13:21.850 checks certain rules. One is when it receives a block over the wire, and we'll 00:13:21.850 --> 00:13:27.130 kind of get into later what receiving over the wire means. But basically when it 00:13:27.130 --> 00:13:32.840 receives a block from another peer, it does some preliminary checks on it. And 00:13:32.840 --> 00:13:37.640 then once those checks are passed, it goes to a full validation check. Now, that 00:13:37.640 --> 00:13:40.800 full validation will happen even if you feed the block manually, if you 00:13:40.800 --> 00:13:45.420 copy-paste it or something like that. But these preliminary checks should only 00:13:45.420 --> 00:13:49.180 happen when they receive the block over the network. And it was those checks that 00:13:49.180 --> 00:13:55.040 were enforcing this 10-kilobyte limit on all versions of SegWit, essentially, not 00:13:55.040 --> 00:13:59.800 just on SegWit 0. Because the way Bitcoin SegWit has been designed is there's 00:13:59.800 --> 00:14:05.610 multiple versions of SegWit. And if the version number is more than zero, then in 00:14:05.610 --> 00:14:08.750 the beginning, you would always approve the transaction. It would always be 00:14:08.750 --> 00:14:12.850 spendable. You would simply not check any rules, including rules about size limits. 00:14:14.130 --> 00:14:19.250 Yeah. So I would propose from now on, let's just say SegWit and Taproot, even 00:14:19.250 --> 00:14:22.410 though technically it's SegWit zero and SegWit one, right? 00:14:23.050 --> 00:14:28.830 Yeah, in this case, because it's about the witness. And every version of SegWit 00:14:28.830 --> 00:14:32.740 will have a witness, but the rules for that witness will change for every version 00:14:32.740 --> 00:14:37.740 of SegWit. So with regular SegWit, there was a limit of 10,000, but there was no 00:14:37.740 --> 00:14:39.680 limit defined for any other version of SegWit. 00:14:39.960 --> 00:14:40.160 Yeah. 00:14:40.510 --> 00:14:41.930 So it turns out this... 00:14:41.930 --> 00:14:46.030 Okay, so btcd, they did the same check kind of twice. And you mentioned that the 00:14:46.030 --> 00:14:49.310 first check happens when they receive it over the wire. And then you said, we'll 00:14:49.310 --> 00:14:51.550 get into that. So let's get into that. What do you mean by that? 00:14:51.670 --> 00:14:55.130 Yeah, and presumably they check it again in the code that checks the consensus. 00:14:55.270 --> 00:14:58.690 And there they were probably doing it correctly. So in the code that really, you 00:14:58.690 --> 00:15:01.330 know, once they've received the block and put it in their own database, it might 00:15:01.330 --> 00:15:04.850 say, okay, is this block actually obeying SegWit? And then it would do it 00:15:04.850 --> 00:15:08.170 correctly. It would say, what's the version of SegWit? Okay, version number zero. 00:15:08.390 --> 00:15:12.120 Okay, that's the limit. Or if it was Taproot, it would check some other limit. 00:15:14.270 --> 00:15:18.250 Okay. btcd is an alternative Bitcoin implementation. However, we were talking 00:15:18.250 --> 00:15:21.230 about Lightning. So how does this tie in with Lightning then? 00:15:21.570 --> 00:15:25.270 Yeah. So first I'll describe sort of an imaginary scenario because I don't think 00:15:25.270 --> 00:15:28.970 this actually exists in the wild or at least not very often. Let's say you're 00:15:28.970 --> 00:15:33.370 running LND, which is the Lightning implementation built by the same people who 00:15:33.370 --> 00:15:34.510 also built btcd. 00:15:34.930 --> 00:15:36.270 So that's Lightning Labs, right? 00:15:36.570 --> 00:15:40.970 Yeah. Now you can use Lightning LND using different node software. If you were 00:15:40.970 --> 00:15:46.270 using it with btcd, then this bug would very clearly occur because your btcd node 00:15:46.270 --> 00:15:49.710 is talking to other nodes in the network might be Bitcoin Core nodes it sees this 00:15:49.710 --> 00:15:54.390 block it receives it over the wire and it says this is wrong and then it stops syncing 00:15:55.130 --> 00:15:58.010 You still haven't explained what over the wire is, by the way. 00:15:58.230 --> 00:16:03.230 Yeah, just over the internet, something like that. So it basically rejects it 00:16:03.230 --> 00:16:07.930 that way. And then how would this impact Lightning? Your Lightning node wants to 00:16:07.930 --> 00:16:10.990 see all the new blocks in order to make sure that its channels are still okay, 00:16:11.010 --> 00:16:15.770 nobody's trying to cheat on it, etc. And so if your Lightning node is not seeing 00:16:15.770 --> 00:16:18.990 any new transactions anymore, it's going to get increasingly unhappy. 00:16:20.230 --> 00:16:24.350 At some point, it may even be talking to its peers on the Lightning Network, and 00:16:24.350 --> 00:16:27.070 its peers are saying, no, no, no, there are much newer blocks than what you're 00:16:27.070 --> 00:16:31.010 talking about. Why don't you follow the newest blocks? And maybe the peers will 00:16:31.010 --> 00:16:35.810 say, hey, the fees have gone up. Why don't you increase the fees on the penalty 00:16:35.810 --> 00:16:39.670 transactions? Because Lightning nodes will constantly renegotiate what the 00:16:39.670 --> 00:16:40.970 penalty transactions should be, etc. 00:16:42.350 --> 00:16:45.570 But you're noticing, no, no, no, the blockchain hasn't moved. Why are you talking 00:16:45.570 --> 00:16:49.170 about these higher fees? And at some point saying, fuck you. I'm going to 00:16:49.170 --> 00:16:52.310 disconnect from you. I'm going to close all the channels now. And so some people 00:16:52.310 --> 00:16:55.030 on the internet were reporting that they were seeing their channels getting 00:16:55.030 --> 00:16:59.370 closed by the other side because of some miscommunication because their node was 00:16:59.370 --> 00:17:04.650 basically stuck. And that can cascade, of course, if lots of nodes start closing 00:17:04.650 --> 00:17:05.670 the channels at the same time. 00:17:11.440 --> 00:17:12.100 Okay, yeah. 00:17:13.300 --> 00:17:18.840 So we were explaining or you were explaining that if LND uses 00:17:18.840 --> 00:17:23.620 btcd to read the blockchain, to get the blockchain, then the problem is obvious. 00:17:23.920 --> 00:17:26.040 And then you hinted at some problems. 00:17:26.420 --> 00:17:26.580 Yeah. 00:17:29.120 --> 00:17:33.900 So do all LND nodes use btcd? Is this a requirement for LND? 00:17:34.060 --> 00:17:39.570 I think they... They do under some circumstances, and they don't under others. So 00:17:39.570 --> 00:17:45.490 generally, most people, I think, run Bitcoin Core as the node. However, if you're 00:17:45.490 --> 00:17:49.530 running a mobile app, then you may want to not download all the blocks from your 00:17:49.530 --> 00:17:53.310 mobile app. You want to use these filters, BIP158, Neutrino filters that we 00:17:53.310 --> 00:17:56.670 talked about in some other episode. So those filters basically give you a quick 00:17:56.670 --> 00:18:00.550 summary of the block, and then you only request the blocks that you want. And 00:18:00.550 --> 00:18:05.610 that also, under the hood, is btcd. So what's possible on the mobile app 00:18:05.640 --> 00:18:11.040 is that your btcd node on the mobile app will get stuck because of this. 00:18:11.300 --> 00:18:15.980 Because the mobile app will fetch a block, it will fetch the magic bad block, 00:18:16.000 --> 00:18:20.000 because you happen to have asked for that one, and then your mobile app will stop 00:18:20.000 --> 00:18:24.920 too. But that does not really explain what we saw people remark. Because people 00:18:24.920 --> 00:18:27.320 were complaining that, first of all, they were complaining that their node 00:18:27.320 --> 00:18:30.240 stopped, their Lightning node stopped. And I don't think a mobile app would 00:18:30.240 --> 00:18:33.820 necessarily stop, but maybe it would. But they were also complaining about 00:18:33.820 --> 00:18:37.920 channels getting closed. And mobile apps tend to be very patient in this stuff. 00:18:38.080 --> 00:18:42.980 Like you can have your, I don't know, your Phoenix wallet or Breez wallet or 00:18:42.980 --> 00:18:48.660 whatever. You can ignore it for weeks and it'll start annoying you, warn you, but 00:18:48.660 --> 00:18:53.570 it generally doesn't start closing channels on you. So that doesn't seem to be 00:18:53.570 --> 00:18:57.310 what's going on. When people have forced closes of their channels, I think 00:18:57.310 --> 00:18:59.690 they're actually running their own node. They might be using something like 00:18:59.690 --> 00:19:04.680 Umbrel or something. And at the same time, today I saw on Twitter, on October 00:19:04.680 --> 00:19:10.060 10th, Umbrel said, hey guys, there's a new update. Please install this hotfix 00:19:10.060 --> 00:19:14.150 because there's a problem with Lightning. So then my question is, why would 00:19:14.150 --> 00:19:17.640 Umbrel nodes have a problem with this? And this is where I would have loved to 00:19:17.640 --> 00:19:22.220 have read a good postmortem because now I have to guess because I don't know 00:19:22.220 --> 00:19:27.220 exactly how Umbrel works. But as far as I know, software like Umbrel uses Bitcoin 00:19:27.220 --> 00:19:32.790 Core to download all the blocks. So when people say over the wire... I would 00:19:32.790 --> 00:19:37.710 assume that the wire is Bitcoin Core receiving blocks from other nodes out there. 00:19:38.070 --> 00:19:42.390 And Bitcoin Core doesn't have this bug. So then the question is, why are we still 00:19:42.390 --> 00:19:45.610 running into this bug, even though we're using Bitcoin Core to validate the 00:19:45.610 --> 00:19:50.000 blocks? You know, the Lightning software shouldn't be doing extra consensus 00:19:50.000 --> 00:19:55.400 checks. So this is where I think what's going on is that on your little node, 00:19:55.560 --> 00:20:00.660 you have Bitcoin Core running and you have LND running. But LND, under the hood, 00:20:00.740 --> 00:20:04.180 uses bits and pieces of btcd. It just uses some of the building blocks from btcd. 00:20:04.600 --> 00:20:06.300 And one of those building blocks... 00:20:06.300 --> 00:20:07.760 For example, for Neutrino. 00:20:08.180 --> 00:20:12.720 Yeah, for Neutrino. And then, again, there's two sort of paths that I would 00:20:12.720 --> 00:20:18.400 imagine could have gone wrong. One path is that the node, the LND software, 00:20:18.700 --> 00:20:24.500 is talking to your full node, not directly in the sense it is talking using these 00:20:24.500 --> 00:20:30.200 filters. So it might be asking your node, so it's on the same device, but your 00:20:30.200 --> 00:20:33.660 LND software is asking the node for the filters on the same device, and then 00:20:33.660 --> 00:20:37.010 given the filters, it asks the node for the blocks. And those blocks are still 00:20:37.010 --> 00:20:40.910 going over a wire. Even though the wire is inside your own computer, it's still a 00:20:40.910 --> 00:20:44.990 wire from the point of view of the software. And so then it would also run into 00:20:44.990 --> 00:20:49.190 this bug where the block briefly goes on the wire from your Bitcoin Core node to 00:20:49.190 --> 00:20:55.060 this little piece of btcd, really. And so that's a problem because 00:20:55.060 --> 00:20:58.820 you don't want to have these extra valid... You don't want to be reliant on some 00:20:58.820 --> 00:21:02.060 other implementation than Bitcoin Core for your consensus checks because there 00:21:02.060 --> 00:21:07.570 could be other bugs in there. That's one possibility. The other possibility is 00:21:07.570 --> 00:21:13.530 that it's even different there. Maybe LND talks directly to Bitcoin Core and 00:21:13.530 --> 00:21:17.630 just asks for every block, which would make sense. I mean, Lightning nodes, if 00:21:17.630 --> 00:21:20.230 it's on the same machine anyway, it might as well just ask every block from 00:21:20.230 --> 00:21:25.490 Bitcoin Core, just in raw form. And that could also still be interpreted as if it 00:21:25.490 --> 00:21:28.410 was going over the wire. So maybe it was still running those same checks, even 00:21:28.410 --> 00:21:34.330 though there was nothing going over a wire. In any case, somehow some of the 00:21:34.330 --> 00:21:40.250 btcd consensus code is, you know, network code, but network is kind of, you 00:21:40.250 --> 00:21:41.750 know, important for consensus too. 00:21:43.290 --> 00:21:48.250 Somehow it's getting run, even though you're not running btcd as the node, which 00:21:48.250 --> 00:21:53.210 is kind of worrying from an architectural point of view. Because again, in 00:21:53.210 --> 00:21:56.990 general, other implementations than Bitcoin Core can cause problems when they 00:21:56.990 --> 00:21:59.870 have even the slightest different interpretation of what the rules are. 00:22:00.880 --> 00:22:06.640 Right okay so we so to summarize that we know what the 00:22:06.640 --> 00:22:11.990 bug in btcd was what we're unsure about and we've tried to look for it 00:22:11.990 --> 00:22:17.770 is why it affected the Lightning network or to the extent that it 00:22:17.770 --> 00:22:20.110 did why it affects LND nodes. 00:22:20.110 --> 00:22:25.110 Why why it affected nodes even though you'd think they were not running btcd at 00:22:25.110 --> 00:22:29.310 least not the full version of btcd right so that's that's an open question I 00:22:29.310 --> 00:22:32.310 think we've we've postulated some possibilities I would also observe… 00:22:32.310 --> 00:22:33.390 Someone has the answer. 00:22:33.390 --> 00:22:34.690 Let us know. 00:22:34.690 --> 00:22:35.270 Let us know. 00:22:35.270 --> 00:22:38.590 We'll mention maybe by the time this airs it's already been explained by somebody 00:22:38.590 --> 00:22:41.470 and they're like oh you guys are wrong you didn't read this post it's because you 00:22:41.470 --> 00:22:45.290 wrote it after we recorded it so one more thing is that I think some people 00:22:45.970 --> 00:22:51.490 framed this as a Taproot-related bug, but it really 00:22:51.490 --> 00:22:56.050 wasn't, because I work on this other project called Fork Monitor, which is by 00:22:56.050 --> 00:23:00.170 BitMEX Research, and Fork Monitor checks different nodes to see if they have any 00:23:00.170 --> 00:23:04.250 problems compared to each other. And because I was also busy with the conference, 00:23:04.510 --> 00:23:08.170 I completely ignored what Fork Monitor was doing, but it actually turned out that 00:23:08.170 --> 00:23:12.960 Fork Monitor, the btcd node on Fork Monitor, was also stuck at this magical 00:23:12.960 --> 00:23:17.380 block, or one block before the magical block. But just nobody noticed it. Or 00:23:17.380 --> 00:23:21.200 maybe people did notice it and told me about it, and I ignored it anyway because 00:23:21.200 --> 00:23:23.200 I didn't know there was this drama going on. 00:23:23.300 --> 00:23:24.440 Busy partying in Amsterdam. 00:23:24.700 --> 00:23:28.520 Exactly. That has to happen. But basically, this particular node and fork monitor 00:23:28.520 --> 00:23:32.140 has not been updated for a while because I was being extremely lazy. It was 00:23:32.140 --> 00:23:35.720 running a very old version. Actually, it was running a version that was from 00:23:35.720 --> 00:23:40.090 before they had Taproot support. So the node that stopped did not support 00:23:40.090 --> 00:23:44.740 Taproot, which means this bug is not related to Taproot at all. And in fact, 00:23:44.940 --> 00:23:49.680 somebody could have triggered this bug before Taproot was even created. Depending 00:23:49.680 --> 00:23:53.780 on when the bug was introduced, probably when SegWit was introduced, I'm guessing 00:23:53.780 --> 00:23:58.480 this check was written when SegWit was introduced. So this bug has been sitting 00:23:58.480 --> 00:23:59.780 there for like five years. 00:23:59.980 --> 00:24:05.820 But then someone had to make a SegWit version one transaction before there was a 00:24:05.820 --> 00:24:06.600 SegWit version one. 00:24:06.720 --> 00:24:09.980 Yes, but that's possible. And we've talked about this in the earlier episode 00:24:09.980 --> 00:24:14.200 about accidental hard forks or what is a hard fork or what isn't a hard fork. You 00:24:14.200 --> 00:24:18.110 can use features that don't exist yet in Bitcoin Core. And the thing is, you're 00:24:18.110 --> 00:24:22.710 giving away your money, but you can give away your money in a way that, you know, 00:24:22.830 --> 00:24:26.230 causes other nodes to crash or causes half the Lightning network to crash in this 00:24:26.230 --> 00:24:30.750 case. Or maybe not half the network. I don't know how big the explosion was. This 00:24:30.750 --> 00:24:34.190 is why I'd like to see a postmortem. Like how many nodes actually went down 00:24:34.190 --> 00:24:34.770 because of this? 00:24:35.720 --> 00:24:41.620 Right. Yes, me too. And so are we done with this part? Are we done 00:24:41.620 --> 00:24:42.520 with the speculation part? 00:24:42.880 --> 00:24:43.620 Yep, I think so. 00:24:43.880 --> 00:24:48.370 Okay, so then I guess to sort of close this off. And you did allude to it already 00:24:48.370 --> 00:24:53.870 a little bit but what is actually the problem or the harm or 00:24:53.870 --> 00:24:59.820 how does this affect users so to say because I would assume that at 00:24:59.820 --> 00:25:04.620 least initially a Lightning node would actually continue to operate like normal 00:25:04.620 --> 00:25:08.750 even if it's not on the latest block because it doesn't require blockchain 00:25:08.750 --> 00:25:12.210 confirmations to make transactions or to forward payments, right? 00:25:12.610 --> 00:25:15.610