WEBVTT - Automatic transcript; recognition may contain errors. Reviewed for technical wording and speaker labels. Times match the published episode. 00:00:19.700 --> 00:00:22.060 Live from Utrecht, this is Bitcoin. 00:00:22.800 --> 00:00:23.280 Explained. 00:00:23.520 --> 00:00:25.080 There you go, Sjors. How are you? 00:00:25.420 --> 00:00:25.760 Excellent. 00:00:26.640 --> 00:00:29.040 Back in good old Utrecht, we were both in London. 00:00:29.280 --> 00:00:29.740 That's right. 00:00:29.940 --> 00:00:34.200 Planning to record some episodes there, but then everybody got COVID. 00:00:34.780 --> 00:00:38.000 Yes, and also it's a conference, so it's always a bit difficult to record. 00:00:39.490 --> 00:00:44.750 So back in Utrecht, and we're going to discuss a blog post. Well, we're going 00:00:44.750 --> 00:00:50.560 to... discuss a concept that was described in a blog post by Jameson Lopp. 00:00:50.680 --> 00:00:51.220 That's right. 00:00:52.420 --> 00:00:57.780 He actually published a couple of blog posts about backwards compatibility, 00:00:57.780 --> 00:01:02.160 I guess, or syncing old nodes. That's really what the blog posts were about. 00:01:02.380 --> 00:01:02.660 Exactly. 00:01:03.140 --> 00:01:08.880 Yeah, so Bitcoin Core, of course, releases two new clients each year or at least 00:01:08.880 --> 00:01:11.160 two new major releases each year, right? 00:01:11.420 --> 00:01:11.860 Yeah, roughly. 00:01:12.280 --> 00:01:18.220 And these blog posts were about syncing older Bitcoin Core nodes to see 00:01:18.220 --> 00:01:21.820 how fast they would actually sync to the current chain tip right? 00:01:21.820 --> 00:01:22.440 That's right. 00:01:22.440 --> 00:01:26.480 And I think you've done similar research a couple years ago is that right? 00:01:26.480 --> 00:01:32.400 Yeah so back in 2017 um what I did was I took a couple of Amazon AWS 00:01:32.400 --> 00:01:37.640 machines so these are just cloud machines, and install different versions of 00:01:37.640 --> 00:01:42.080 Bitcoin Core on those machines, and then just have them sync the Bitcoin 00:01:42.080 --> 00:01:45.040 blockchain as far as possible, and then I would just measure how long it would 00:01:45.040 --> 00:01:49.430 take, and those pretty big differences. And he's basically done that again, 00:01:49.670 --> 00:01:53.350 probably slightly different, but the same idea and with more recent versions. 00:01:53.990 --> 00:01:54.510 Right. 00:01:54.850 --> 00:01:56.810 And also with some older versions. 00:01:57.780 --> 00:02:02.600 Yeah, so to be very specific, I already mentioned this, but there were basically 00:02:02.600 --> 00:02:07.180 two blog posts by Jameson, and the first one was really about syncing old nodes, 00:02:07.240 --> 00:02:11.080 and I would say the second one was actually mainly about backwards compatibility 00:02:11.080 --> 00:02:15.000 and forwards compatibility, which, you know, it's a related issue. 00:02:15.690 --> 00:02:21.570 Yeah, and he also zoomed in on some of the really old nodes specifically, like 00:02:21.570 --> 00:02:27.450 the version 0.3, 0.4, 0.5 from many, many years ago and 00:02:27.450 --> 00:02:29.470 how difficult it is to get these things to work at all. 00:02:30.150 --> 00:02:33.210 Yes, well, that's the backwards compatibility part, right? 00:02:34.280 --> 00:02:38.420 Yeah, arguably. I mean, we can get into whether it's backwards compatibility or 00:02:38.420 --> 00:02:40.640 whether it's some other problems that he was running into. 00:02:42.040 --> 00:02:46.680 Which we will get into. Yes all right. First of all for the people that don't 00:02:46.680 --> 00:02:51.070 know this, what does that actually mean? What is backwards compatibility? Or 00:02:51.070 --> 00:02:55.030 Sjors, forwards compatibility, because sometimes the terms are sort of used 00:02:55.030 --> 00:02:56.130 interchangeably. 00:02:56.610 --> 00:03:00.390 Yeah, so I'm also going to be sloppy about it. The general idea is that you want 00:03:00.390 --> 00:03:04.610 —uh, you don't want to force people to upgrade their nodes at least definitely not 00:03:04.610 --> 00:03:09.870 immediately. So older nodes should work well with newer nodes. And if you know if 00:03:09.870 --> 00:03:13.930 you change… if you do this soft fork, you know the the new nodes will be able to 00:03:13.930 --> 00:03:18.530 the old nodes will just keep working fine while everybody… while other people 00:03:18.530 --> 00:03:21.560 upgrade to a new node. um Yeah, 00:03:22.300 --> 00:03:27.040 Exactly. So like I mentioned, Bitcoin Core releases two new major releases every 00:03:27.040 --> 00:03:32.580 year. And the goal is that these new releases won't fall out of sync with older 00:03:32.580 --> 00:03:36.700 nodes or the other way around, depending on how you look at it. The goal is to 00:03:36.700 --> 00:03:42.080 keep everyone in consensus, including older nodes. Why do we think this is 00:03:42.080 --> 00:03:43.620 important? 00:03:44.470 --> 00:03:49.270 Well, there are multiple reasons for it. Because you want to keep the system as 00:03:49.270 --> 00:03:53.650 voluntary as possible. There might be reasons why people are worried about 00:03:53.650 --> 00:03:57.590 installing new versions of Bitcoin Core. Maybe they want more people to review 00:03:57.590 --> 00:04:02.910 it. If suddenly you have only two days to upgrade your node, maybe there's some 00:04:02.910 --> 00:04:06.450 malware in it. But if you wait a year, it's more likely somebody would have 00:04:06.450 --> 00:04:11.490 already found it. It's also just maybe not possible to upgrade. Maybe you're busy 00:04:11.490 --> 00:04:13.040 doing other things. you know. 00:04:13.040 --> 00:04:17.080 Yeah, maybe you want to [unclear] the code yourself right? Yeah, it's at least 00:04:17.080 --> 00:04:20.840 possible someone might want to do that, and that can take a little bit of time, 00:04:20.840 --> 00:04:24.360 and therefore it's important that they can keep using Bitcoin in the meantime. 00:04:24.360 --> 00:04:28.020 Yeah, and and companies you know, that are running exchanges and whatever, they 00:04:28.020 --> 00:04:32.420 might be busy, they don't necessarily have… you know they don't necessarily can 00:04:32.420 --> 00:04:37.380 do everything for you on January 1st of some year. So it's nice if they don't 00:04:37.380 --> 00:04:39.720 have to upgrade. Though it's always a good idea to upgrade because you tend to 00:04:39.720 --> 00:04:42.940 stay up to date on security fixes and performance improvements. 00:04:43.520 --> 00:04:48.240 Yeah, I would argue there's also kind of... And we'll move on after this. But I 00:04:48.240 --> 00:04:52.440 would argue there's also kind of a philosophical reason. Like in our previous 00:04:52.440 --> 00:04:56.220 episode, we were discussing like very hypothetical situations where there's a 00:04:56.220 --> 00:05:01.700 chain split. And like if you're... If you're asking the question, what is 00:05:01.700 --> 00:05:06.240 actually Bitcoin, then it makes sense that older nodes, if they are still 00:05:06.240 --> 00:05:10.800 syncing, someone never upgraded their node, why wouldn't that be Bitcoin? That 00:05:10.800 --> 00:05:15.580 was a big theme during the block size wars. Can you just change the protocol? And 00:05:15.580 --> 00:05:17.440 there was this argument about backwards compatibility. 00:05:19.310 --> 00:05:23.230 It's a philosophical argument that if old nodes still sync, then obviously that's 00:05:23.230 --> 00:05:23.950 still Bitcoin, right? 00:05:24.270 --> 00:05:28.970 Yeah. But as we'll find out now, there is some nuance to that argument because 00:05:28.970 --> 00:05:33.990 trying to run a node from 12 years ago is not possible, probably. 00:05:34.250 --> 00:05:39.310 For sure. Well, let's get into that right now, I think. Let's start there. 00:05:39.970 --> 00:05:45.830 So Jameson did these experiments with some older nodes. And when we 00:05:45.830 --> 00:05:47.730 say older, how old are we talking? 00:05:48.280 --> 00:05:53.200 I think as old as 2012, maybe even 2011. 00:05:53.640 --> 00:05:58.960 Yeah, that era. 2011, 2012, 2013. It was 00:05:58.960 --> 00:06:04.320 pre-Bitcoin 0.7, right? 00:06:05.080 --> 00:06:08.360 Which ones [unclear]? 4, 5, 6, 7? 00:06:08.500 --> 00:06:09.220 Even 3. 00:06:09.860 --> 00:06:16.260 3, 4, 5, 6, 7. Or 3, 4, 5, 6 in this latest blog post, right? Yeah. 00:06:18.540 --> 00:06:20.980 And these did not sync. Did all of them not sync? 00:06:23.180 --> 00:06:27.820 I think at least until 0.5, it could not get to sync. 00:06:28.160 --> 00:06:28.380 Right. 00:06:28.600 --> 00:06:33.420 And that also reflects my own experience when I tried that in 2017. 00:06:36.530 --> 00:06:39.470 Nodes older than 0.5 will not sync. 00:06:39.610 --> 00:06:40.750 0.5 wouldn't sync either. 00:06:41.010 --> 00:06:45.930 Right. Okay. And then we get into some of the reasons as far as we can tell 00:06:45.930 --> 00:06:51.720 because... My reading of the blog post is not entirely clear every time why 00:06:51.720 --> 00:06:56.920 exactly. It's clear where the nodes stop syncing, but it's not necessarily clear 00:06:56.920 --> 00:06:58.940 why exactly it stops syncing. Is that right? 00:06:59.460 --> 00:07:04.160 Yeah, but we can get into some of the reasons, and eventually we'll find out that 00:07:04.160 --> 00:07:09.260 with 0.5, I wasn't able to sync it all the way, he wasn't able to sync it all the 00:07:09.260 --> 00:07:14.960 way, but there's some recent developments that may still be ways to get it to 00:07:14.960 --> 00:07:16.730 sync. We simply don't know yet. 00:07:17.010 --> 00:07:22.620 Okay, so where does it stop or what's the reason as far as we know? What's the 00:07:22.620 --> 00:07:23.080 story here? 00:07:23.360 --> 00:07:27.440 Yeah, so there's a couple of things and those really all boil down to the theme 00:07:27.440 --> 00:07:30.920 of dependencies. So that's a fun topic always. 00:07:31.180 --> 00:07:34.760 I think we discussed this in a previous episode, but maybe you should... 00:07:34.760 --> 00:07:38.760 Very, very long time ago. So our very first episode, I think, or maybe the second 00:07:38.760 --> 00:07:44.240 episode, we talked about dependencies like OpenSSL. And how, you know, when 00:07:44.240 --> 00:07:45.580 those dependencies change… 00:07:45.600 --> 00:07:47.800 It's very basic. What is a dependency? 00:07:45.800 --> 00:07:46.400 [unclear]… 00:07:48.260 --> 00:07:52.780 Or a library. Yeah. So dependency is a piece of software that you include in your software, 00:07:54.030 --> 00:07:58.950 basically. So in the case of OpenSSL, it's a very well-known dependency. 00:08:00.380 --> 00:08:04.620 It's a cryptographic library. So it means that Satoshi did not have to implement 00:08:04.620 --> 00:08:08.640 all of the elliptic curve cryptography, he could just 00:08:08.640 --> 00:08:14.000 use OpenSSL and let it handle all the cryptography. 00:08:14.160 --> 00:08:19.560 Right, so OpenSSL is being developed by another group of developers and 00:08:19.560 --> 00:08:24.320 their code is essentially used in Bitcoin Core. 00:08:24.690 --> 00:08:25.330 That's right. 00:08:25.530 --> 00:08:30.930 And then it's called a dependency or the problem is if these other developers 00:08:30.930 --> 00:08:35.190 change something that might affect your own software, right? That's sort of the problem. 00:08:35.390 --> 00:08:39.830 Yeah, exactly. And this can be especially a problem when the dependency... 00:08:40.570 --> 00:08:44.970 secretly, or not secretly, but like accidentally imposes rules that you don't 00:08:44.970 --> 00:08:45.850 know are rules. 00:08:46.070 --> 00:08:46.410 Right. 00:08:46.890 --> 00:08:51.150 And, well, there's a couple of, there's probably three different incidents of 00:08:51.150 --> 00:08:52.930 that historically, at least three. 00:08:53.550 --> 00:08:55.030 You mean in Bitcoin or? 00:08:55.110 --> 00:08:55.690 Yeah, in Bitcoin. 00:08:55.850 --> 00:08:56.350 Yeah, okay. 00:08:56.630 --> 00:08:59.490 I mean, in general, dependencies, of course, cause all sorts of headaches. I 00:08:59.490 --> 00:09:05.250 think it was in the news last week that somebody in the NPM, the 00:09:05.250 --> 00:09:09.670 Node Package Manager, which is a JavaScript library that a lot of... Any website 00:09:09.670 --> 00:09:13.970 that looks even a little bit fancy probably uses that in the background. And it 00:09:13.970 --> 00:09:18.750 turns out that if you were a developer and you were based in Russia or Belarus, 00:09:19.070 --> 00:09:24.670 your computer would get wiped if you updated the dependency. So dependencies are 00:09:24.670 --> 00:09:29.030 a nightmare in general, but we won't go into too much detail about that nightmare. 00:09:29.330 --> 00:09:30.730 Yeah, let's stick to the Bitcoin. 00:09:30.990 --> 00:09:35.270 Yeah, so for Bitcoin, we can start with one dependency, which is called 00:09:35.270 --> 00:09:35.870 BerkeleyDB. 00:09:37.580 --> 00:09:42.940 BerkeleyDB is a database technology that was initially used to store, I think, 00:09:43.000 --> 00:09:46.100 information about the UTXO set, or at least information about Bitcoin 00:09:46.100 --> 00:09:50.340 transactions. So the node was using that dependency, which is built by other 00:09:50.340 --> 00:09:55.090 people. And when the Bitcoin blockchain was small, when blocks were small, 00:09:56.700 --> 00:10:01.160 Every block would just add a bunch of transactions to the database. I'm kind of 00:10:01.160 --> 00:10:04.380 simplifying it partially because I don't know the precise details. But let's say 00:10:04.380 --> 00:10:07.760 a block contains 100 transactions and it would just add those 100 transactions to 00:10:07.760 --> 00:10:13.400 the database and the database would be happy. However, it turns out that if you 00:10:13.400 --> 00:10:16.600 crank up the number of transactions you start putting in the database, so if the 00:10:16.600 --> 00:10:21.240 blocks get bigger, at some point the database won't allow you to add these 00:10:21.240 --> 00:10:27.200 blocks. However, that never happened in practice because Bitcoin switched 00:10:27.200 --> 00:10:32.530 to a new database type in about 2013 called LevelDB. 00:10:32.830 --> 00:10:33.150 Right. 00:10:34.230 --> 00:10:38.950 And this LevelDB, you know, it's much faster and there was other reasons why it 00:10:38.950 --> 00:10:43.910 was better. But what happens is at that point, blocks started getting even bigger 00:10:43.910 --> 00:10:47.610 and LevelDB had no problem with it. But because we said we want to be backwards 00:10:47.610 --> 00:10:51.920 compatible. Some people were still running nodes that use BerkeleyDB and those 00:10:51.920 --> 00:10:56.320 nodes started causing problems because they started hitting those limits. And so 00:10:56.320 --> 00:11:00.440 the database would say, hey, this block is too large. Kind of. 00:11:00.760 --> 00:11:06.440 Right. So if a big block, and by big we mean one megabyte, I guess, or 00:11:06.440 --> 00:11:07.800 maybe close to one megabyte… 00:11:07.920 --> 00:11:11.020 Yeah, or 300 kilobytes. I don't know where it went wrong. 00:11:11.160 --> 00:11:15.660 Whatever it was. If a big block would have been mined, say, in 2012, would that 00:11:15.660 --> 00:11:20.450 have meant that Bitcoin... Core nodes it wasn't even called Bitcoin Core back 00:11:20.450 --> 00:11:24.310 then, Bitcoin nodes then would have crashed is that sort of… 00:11:24.310 --> 00:11:27.270 I think they would have not have crashed, but they would have rejected the block, 00:11:27.270 --> 00:11:31.790 but it was worse than that because it was not deterministic, so it's like some 00:11:31.790 --> 00:11:36.630 Bitcoin Core nodes would have stopped and others would not have. Um so because 00:11:36.630 --> 00:11:41.350 you were hitting internal, basically internal intricacies of the database 00:11:41.350 --> 00:11:46.270 package, it wasn't like a clear coded limit, just running into the internal 00:11:46.270 --> 00:11:51.850 magic. And so what ended up happening was a short chain split where the decision 00:11:51.850 --> 00:11:57.410 was made to undo the update and then everybody was told to either update to a new 00:11:57.410 --> 00:12:02.540 version of Bitcoin Core or change a slight setting in that database. 00:12:03.750 --> 00:12:09.690 Right, but my… question and you've answered it but just to be clear, essentially 00:12:09.690 --> 00:12:14.210 what you're saying is there was actually a problem in Bitcoin clients back then. 00:12:14.210 --> 00:12:19.210 We just never noticed it because all blocks were almost empty or very small. Yeah 00:12:19.210 --> 00:12:24.950 and only if you start to try to sync it now, now the blocks are big, it's 00:12:24.950 --> 00:12:28.330 that the problem is really revealed. So there was kind of a bug in old nodes right? 00:12:28.330 --> 00:12:32.410 Yeah but it got revealed as soon as they upgraded it, and my guess would be that 00:12:32.410 --> 00:12:36.230 you know, the miners are also running the same software to create the blocks, and 00:12:36.230 --> 00:12:38.650 so they wouldn't have created those big blocks in the first place. So maybe 00:12:38.650 --> 00:12:42.640 that's why it happened at the same time. Or maybe, you know, it could have been 00:12:42.640 --> 00:12:48.100 deliberate. But it turns out that had that upgrade not happened, we would have 00:12:48.100 --> 00:12:53.400 run into another bottleneck in that same software later on. And that's what me, I 00:12:53.400 --> 00:12:57.320 think both me and Jameson ran into. That if you keep running that old software... 00:12:57.320 --> 00:13:00.860 Sorry, this is the same dependency. Also BerkeleyDB? We're still talking about BerkeleyDB. 00:13:00.860 --> 00:13:05.840 So if you stick to BerkeleyDB, if you decide, you know, you go back in your time 00:13:05.840 --> 00:13:08.760 machine to 2013, you say, you know what, I don't want to install this update. 00:13:09.150 --> 00:13:12.210 I'll just change this configuration as people recommend, and then I'll just keep 00:13:12.210 --> 00:13:16.830 going. If you do that, then I think in 2015 or something like that, you start 00:13:16.830 --> 00:13:21.410 running into blocks that will still crash your database. And that's the new 00:13:21.410 --> 00:13:27.390 development where I posted a question on Stack Overflow years and years ago 00:13:27.390 --> 00:13:31.090 asking I'm running you know, I can't get past this block with this old version of 00:13:31.090 --> 00:13:35.130 the software. Can anybody help? And nobody replied to that until yesterday. 00:13:35.130 --> 00:13:39.450 Andrew Chow actually replied to it and had some suggestions on how to get past 00:13:39.450 --> 00:13:43.860 those error messages. This is why it's still a mystery. We it's possible that 00:13:43.860 --> 00:13:49.000 with a few other changes in the settings you know, it might still be able to 00:13:49.000 --> 00:13:53.800 reach the tip of the modern blockchain. My guess is it will run into other 00:13:53.800 --> 00:13:57.760 limitations in that database because it's just not been tested with the giant 00:13:57.760 --> 00:13:58.940 blockchain that we have today. 00:13:59.860 --> 00:14:00.100 Right. 00:14:01.800 --> 00:14:07.400 Okay. So again, like the previous example, essentially there was always a problem 00:14:07.400 --> 00:14:13.010 in Bitcoin code back then and it was just never revealed because the blocks 00:14:13.010 --> 00:14:16.230 never were created that were big enough to actually reveal the problem. 00:14:16.470 --> 00:14:16.690 Exactly. 00:14:16.990 --> 00:14:21.380 It's only if you... sync an old node that the problem will reveal itself. So 00:14:21.380 --> 00:14:23.800 we've had two of these problems, two twice. 00:14:24.220 --> 00:14:28.280 We've had one that people really noticed it in the wild that caused an actual 00:14:28.280 --> 00:14:32.940 split. And we've had one that you can notice yourself if you try to use very old software. 00:14:33.260 --> 00:14:36.920 Right. And you mentioned there might be even more if we get past this new block 00:14:36.920 --> 00:14:41.740 and someone tries syncing again and new problems. So basically this Berkeley DB 00:14:41.740 --> 00:14:46.160 database, that was kind of a problem and we got rid of it. And now the new... 00:14:46.910 --> 00:14:48.630 Nodes perform better for that reason. 00:14:48.990 --> 00:14:49.070 Yeah. 00:14:49.290 --> 00:14:49.770 Is that right? 00:14:50.010 --> 00:14:54.330 Yeah. And so there's a second dependency that's been a cause of headaches. And we 00:14:54.330 --> 00:14:57.070 have discussed that one in the earlier podcast, which is OpenSSL. 00:14:57.250 --> 00:15:00.090 We discussed it in this podcast, I think, a minute ago. 00:15:00.230 --> 00:15:06.150 But specifically, the problem there is that OpenSSL, the very old versions of 00:15:06.150 --> 00:15:11.410 OpenSSL, so the ones that Satoshi installed, those were very permissive in terms 00:15:11.410 --> 00:15:15.140 of how the signature could be encoded. So you couldn't make a fake signature, but 00:15:15.140 --> 00:15:20.520 the way you write a signature down, it could have like 75 characters or maybe 00:15:20.520 --> 00:15:25.440 more. And later versions of OpenSSL said, no, no, no, we're going to make it more 00:15:25.440 --> 00:15:30.340 precise. You have to only use, I don't know, 73 characters for your signature and 00:15:30.340 --> 00:15:35.690 no more. And it happened to be, if I understand correctly, that 00:15:35.690 --> 00:15:41.310 there never was any signature with that many characters in 00:15:41.440 --> 00:15:47.410 practice. No, I think there was. So let me see if I get 00:15:47.410 --> 00:15:51.490 the problem. So yeah, so I think there was a signature that was bigger, but 00:15:51.490 --> 00:15:54.630 nobody ran into the problem because if you had installed an old version of 00:15:54.630 --> 00:15:59.790 Bitcoin Core back in the day, you would have had the old version of OpenSSL and 00:15:59.790 --> 00:16:03.410 therefore you would have been able to sync just fine with these bigger 00:16:03.410 --> 00:16:08.840 signatures. But if now you try to install that old version, 00:16:09.440 --> 00:16:14.300 today, on a modern computer, you're going to have a modern version of OpenSSL and 00:16:14.300 --> 00:16:17.380 you're going to produce a version of Bitcoin Core that will not accept these old blocks. 00:16:17.680 --> 00:16:21.500 Right, yeah. So this is kind of the opposite problem as the previous one. The 00:16:21.500 --> 00:16:25.360 previous problem we just mentioned with Berkeley is essentially that Bitcoin Core 00:16:25.360 --> 00:16:28.860 clients were always, you know, they always had these bugs essentially, but the 00:16:28.860 --> 00:16:33.080 bugs were never revealed because blocks were never big enough to reveal them. And 00:16:33.080 --> 00:16:37.640 now the problem is kind of different because Jameson was doing these tests. He 00:16:37.640 --> 00:16:42.960 had to rebuild old clients. And while he was rebuilding the old Bitcoin Core 00:16:42.960 --> 00:16:48.220 clients, the only way he knew to do it or... was by... 00:16:48.430 --> 00:16:49.540 I don't know how this works. 00:16:49.660 --> 00:16:51.800 Yeah, he was using the most recent version of the dependencies. 00:16:52.080 --> 00:16:55.360 Yeah, so then you get a new dependency. So now we have sort of the original 00:16:55.360 --> 00:17:00.260 dependency problem that I discussed is that the dependency changes and now the 00:17:00.260 --> 00:17:04.520 original client doesn't work anymore. However, so here the problem would be the 00:17:04.520 --> 00:17:09.040 opposite way that if you were still running that old software, you never changed 00:17:09.040 --> 00:17:13.580 it, you're still using your old computer from 2012 or whatever. you would still 00:17:13.580 --> 00:17:17.420 actually sync to the current blockchain because then you're also still using this 00:17:17.420 --> 00:17:20.420 old OpenSSL code, right? 00:17:20.480 --> 00:17:26.000 Yeah, and in fact, there was a soft fork at some point that restricted the size 00:17:26.000 --> 00:17:29.540 of signatures in the consensus rules. But again, because it was a soft fork, your 00:17:29.540 --> 00:17:34.640 old node would be fine with the new rules, but it would not detect a violation of 00:17:34.640 --> 00:17:37.820 the new rules if somebody produced a signature that we no longer consider 00:17:37.820 --> 00:17:42.070 invalid. But there was, again, a transition period, I think it might have been 00:17:42.070 --> 00:17:47.930 2013 or even 2015, where this was actually a problem for people who were 00:17:47.930 --> 00:17:52.610 compiling their own nodes. They actually had to be careful to when they compile 00:17:52.610 --> 00:17:56.210 their own nodes to either use an old version of OpenSSL or 00:17:56.210 --> 00:18:02.210 apply a little patch to a Bitcoin Core that would work around the newer 00:18:02.210 --> 00:18:03.330 versions of OpenSSL. 00:18:04.150 --> 00:18:09.770 Right. Does this have anything to do with the 2013 blockchain split that actually 00:18:09.770 --> 00:18:11.310 happened for like six blocks? 00:18:11.410 --> 00:18:13.810 No, that was the one we just talked about. That was a Berkeley DB. 00:18:14.010 --> 00:18:15.570 Oh, that was a Berkeley DB problem. 00:18:15.810 --> 00:18:21.270 And I think this is later. I think this is 2015, but I'm not sure. In any case, 00:18:21.490 --> 00:18:26.280 this was a problem in reality for people who compile their own nodes. It was not 00:18:26.280 --> 00:18:29.680 a problem for people who download the nodes because the downloaded versions would 00:18:29.680 --> 00:18:33.900 all have the old version of OpenSSL in it or the newer version would have the 00:18:33.900 --> 00:18:36.940 workaround in it. It was never a problem when you download Bitcoin Core. It is a 00:18:36.940 --> 00:18:40.640 problem when you compile it yourself. And it was even a problem back then. 00:18:41.460 --> 00:18:46.240 Right. At the risk of getting sidetracked a little bit, would it be possible 00:18:46.240 --> 00:18:52.100 to... you know, basically rebuild an old client also using like an old library? 00:18:52.320 --> 00:18:54.920 I guess you'd have to find it somewhere and that's the hard part then? 00:18:55.220 --> 00:18:59.260 Yeah, no, exactly. I mean, a lot of the open source libraries are still available 00:18:59.260 --> 00:19:03.980 somewhere. It gets a bit more. I think that's what Jameson did. He installed like 00:19:03.980 --> 00:19:07.880 a super old version of Ubuntu and then tried to also find some of the very old 00:19:07.880 --> 00:19:11.560 packages. But the problem is, of course, most operating systems that you would be 00:19:11.560 --> 00:19:15.810 building on will have the most recent versions of the libraries. And obviously 00:19:15.810 --> 00:19:19.470 you do not want to actually use a computer with an outdated version of OpenSSL 00:19:19.470 --> 00:19:23.170 because you'd have heartbleeds and all these like very bad security 00:19:23.170 --> 00:19:27.830 vulnerabilities on your computer. But there are ways to do it, even safe ways to do it. 00:19:29.440 --> 00:19:29.840 Yeah. 00:19:30.220 --> 00:19:30.520 Right. 00:19:30.820 --> 00:19:36.140 So this is a problem for, say, an historian like a thousand years from now. If 00:19:36.140 --> 00:19:39.960 you're a historian and you want to really study Bitcoin in the old days, you're 00:19:39.960 --> 00:19:42.460 going to have a problem because a thousand years from now, you'll probably still 00:19:42.460 --> 00:19:44.440 find the old source code of Bitcoin. 00:19:46.450 --> 00:19:50.470 But you'll have a very hard time building a computer, even a virtual computer, 00:19:50.710 --> 00:19:55.010 that will behave the same as a computer from today. 00:19:55.390 --> 00:20:01.200 Right. So we've now discussed three reasons why old nodes stalled. 00:20:01.340 --> 00:20:06.040 I think they were three, right? Have we covered them all? Right. And just to 00:20:06.040 --> 00:20:11.580 reiterate, in two of these cases, they were actually bugs in the old code, or 00:20:11.580 --> 00:20:12.300 essentially, right? 00:20:13.580 --> 00:20:17.780 Compared to... you can't sync to new blocks because... Do we call them bugs? 00:20:17.920 --> 00:20:18.460 I would. 00:20:18.720 --> 00:20:19.560 Yeah, I think so. 00:20:19.720 --> 00:20:22.360 Yeah, all right. So in two cases, there were actually old bugs and old code that 00:20:22.360 --> 00:20:27.510 just weren't revealed until, you know, much later when no one was using it 00:20:27.510 --> 00:20:31.930 anymore, essentially. And in one case, it was because of the dependency changed. 00:20:32.290 --> 00:20:32.430 Yeah. 00:20:32.570 --> 00:20:35.110 And the old code would have actually synced to the new blockchain. 00:20:35.430 --> 00:20:35.550 Yeah. 00:20:36.130 --> 00:20:41.310 All right. Let's get into the next part of the podcast then, which is 00:20:41.310 --> 00:20:42.570 syncing. 00:20:44.200 --> 00:20:47.080 So syncing is probably one of the main 00:20:48.360 --> 00:20:52.140 user experience bottlenecks. Can we call it that? 00:20:52.500 --> 00:20:55.920 Yeah, because it's the first thing you do when you install Bitcoin, when you 00:20:55.920 --> 00:20:59.880 install Bitcoin Core. And the first thing you do is sit and wait for a long time. 00:21:00.320 --> 00:21:04.700 And so it's not a great experience. Unless you really like progress bars. 00:21:04.940 --> 00:21:08.120 Yeah, and the reason for that is, of course, because you're essentially... 00:21:09.720 --> 00:21:13.020 rechecking or learning about the entire history of the blockchain. You're 00:21:13.020 --> 00:21:16.760 downloading and checking the entire history of the blockchain in order to get up 00:21:16.760 --> 00:21:20.780 to date with what's the state of Bitcoin today, who owns what. 00:21:20.980 --> 00:21:24.460 Exactly. And in earlier episodes, we've talked about some ways to either speed 00:21:24.460 --> 00:21:29.940 that up or to skip it altogether with things like AssumeUTXO. But for now, and 00:21:29.940 --> 00:21:33.200 probably for the foreseeable future, yeah, you need to check every single block, 00:21:33.420 --> 00:21:37.200 not just to know if it's correct, even just to know which coins actually exist. 00:21:39.080 --> 00:21:39.560 Right. 00:21:41.190 --> 00:21:47.100 And the goal for every new Bitcoin Core release is essentially to make this 00:21:47.100 --> 00:21:48.320 a bit faster, I think. 00:21:48.560 --> 00:21:54.380 Not for every Bitcoin Core release, but there's this repeating theme. 00:21:54.380 --> 00:21:59.560 It's not even a goal. It's just that it gets better very often with little 00:21:59.560 --> 00:22:00.860 changes or with bigger changes. 00:22:01.140 --> 00:22:03.980 Yeah, well, actually, when we say better... 00:22:04.320 --> 00:22:04.940 Faster. 00:22:04.940 --> 00:22:09.580 Yeah, well when we say faster, and I guess this is part of the research, the 00:22:09.580 --> 00:22:13.260 thing is, every… by the time there's new Bitcoin Core release the blockchain 00:22:13.260 --> 00:22:15.420 itself has also become bigger. 00:22:15.420 --> 00:22:15.960 Yes. 00:22:15.960 --> 00:22:21.760 So does it actually take more time year over year to sync, or do the performance 00:22:21.760 --> 00:22:25.220 improvements keep up? That’s sort of one of the questions. 00:22:25.220 --> 00:22:30.340 I think there have been times when the performance improvements kept up. I don't 00:22:30.340 --> 00:22:31.280 think that's the case anymore. 00:22:31.280 --> 00:22:31.900 Right. 00:22:33.180 --> 00:22:37.040 So it is getting worse. So there's a couple of things that are good. There's 00:22:37.040 --> 00:22:40.560 Moore's law. That means new hardware is just faster. If you have a new computer, 00:22:40.680 --> 00:22:43.420 you'll sync faster. If you have more RAM, you'll sync faster. If you have a 00:22:43.420 --> 00:22:49.120 faster disk. And the other is these improvements. But in the long run, it's 00:22:49.120 --> 00:22:54.220 very hard for them. I think when I looked at it in 2017, it was basically 00:22:54.220 --> 00:22:58.600 just almost constant. Despite dramatic improvements in the performance of the 00:22:58.600 --> 00:23:04.130 code, it was just getting... it was taking about as long. But of course, the 00:23:04.130 --> 00:23:08.330 hardware is getting faster too. When Jameson and I were running these tests, we 00:23:08.330 --> 00:23:12.550 were running the same modern hardware to run the tests on, just to have a bit of 00:23:12.550 --> 00:23:18.140 a fair comparison. But I think people like Luke Dashjr have done projections on 00:23:18.140 --> 00:23:21.840 it, like how long is it going to take until it actually starts getting faster 00:23:22.540 --> 00:23:27.300 just based on Moore's law. And I think the estimates are like 2040 or later when 00:23:27.300 --> 00:23:32.500 this growth in the blockchain is no longer as bad. They won't outpace anymore the 00:23:32.500 --> 00:23:33.060 Moore's law. 00:23:39.540 --> 00:23:45.420 Okay, sure. So tell our listeners something interesting about this. 00:23:45.460 --> 00:23:46.540 How long does it take to sync? 00:23:46.880 --> 00:23:51.640 So that really depends on your computer and how well you configure it, the 00:23:51.640 --> 00:23:54.800 Bitcoin Core node. So with the default settings, it's not as fast as it can be. 00:23:55.580 --> 00:24:00.760 There was a test by Jonas Schnelli, a former Bitcoin Core developer 00:24:00.760 --> 00:24:06.460 and maintainer. And I think two years ago, he got an Apple M1 00:24:06.460 --> 00:24:12.400 chip, one of those Mac minis. But he only had four gigabytes of RAM. And the 00:24:12.400 --> 00:24:16.480 first time he tried it, he was able to sync the whole chain in 12 hours, I think. 00:24:16.700 --> 00:24:22.100 And then somebody suggested a few better settings to him. And it was six hours. 00:24:22.650 --> 00:24:26.130 So that was like state-of-the-art hardware six hours two years ago. 00:24:27.370 --> 00:24:31.250 But if he had used more RAM, he might have been able to do it a little bit 00:24:31.250 --> 00:24:32.430 faster, maybe in four hours. 00:24:33.880 --> 00:24:39.580 But if you have a super slow Raspberry Pi, it might take you two or three weeks. 00:24:39.800 --> 00:24:40.320 Sure. 00:24:40.620 --> 00:24:46.620 And if you're running, say, a web server, so say you want to install a BTCPay 00:24:46.620 --> 00:24:52.510 server somewhere, and you want on that server to sync the node from scratch, it's 00:24:52.510 --> 00:24:57.610 going to take you weeks. Because these web servers typically are very low 00:24:57.610 --> 00:24:58.050 performance. 00:24:59.370 --> 00:25:03.810 Once it's caught up, it's fine. It'll catch up with all the new blocks. But 00:25:03.810 --> 00:25:08.350 catching it up from the Genesis block, it has low memory, usually only has one or 00:25:08.350 --> 00:25:12.210 two CPUs, and it's all virtual, so it's pretty horrible. 00:25:13.300 --> 00:25:17.760 Yeah this is one of these things that you really kind of want to see in a graph 00:25:17.760 --> 00:25:23.600 and I'll try to remember to include the links to the actual blog posts 00:25:23.600 --> 00:25:27.140 in the show notes, but can you tell us anything interesting about the 00:25:27.140 --> 00:25:28.280 improvements over the years? 00:25:28.680 --> 00:25:29.100 Yeah. 00:25:29.160 --> 00:25:30.700 How much has it improved? 00:25:30.980 --> 00:25:36.660 Pretty dramatic. So one of the, like, I think Jameson found like a factor of 40, 00:25:36.920 --> 00:25:37.980 depending on how you measure it. 00:25:38.320 --> 00:25:39.260 40 since when? 00:25:39.480 --> 00:25:41.340 Since these very old versions. 00:25:41.630 --> 00:25:45.160 Since like 2012. 0.4 or whatever. Yeah. Okay. 00:25:45.620 --> 00:25:49.720 But it's probably more than that. And there's a couple of reasons. I mean, some 00:25:49.720 --> 00:25:52.500 of you might call cheating, but it's not because it actually improves the 00:25:52.500 --> 00:25:58.450 experience for the user. So we did an episode about AssumeValid which is where 00:25:58.450 --> 00:26:03.930 basically the developers pick a block that's before a release, and then the 00:26:03.930 --> 00:26:07.550 Bitcoin Core node doesn't verify the signatures up until that block. It verifies 00:26:07.550 --> 00:26:11.350 everything else, but not the signatures, and that saves, I don't know, a factor 00:26:11.350 --> 00:26:12.090 two at least. 00:26:12.210 --> 00:26:15.070 Yeah, well, I wouldn't call that cheating, but it's definitely a shortcut. 00:26:15.650 --> 00:26:20.350 Yeah, I mean, you can turn it off if you don't like it for all these reasons, but 00:26:20.350 --> 00:26:25.760 it speeds things up a lot. Another improvement was, well, we talked about it, 00:26:25.860 --> 00:26:29.920 LevelDB. So switching from BerkeleyDB, the database that had all these problems, 00:26:30.270 --> 00:26:32.330 but it was also slower to LevelDB. 00:26:32.590 --> 00:26:37.530 By the way, you just mentioned, sorry, I'm interrupting you, but on the previous 00:26:37.530 --> 00:26:41.990 point, you mentioned AssumeValid, but Jameson actually did not. 00:26:43.030 --> 00:26:43.790 No, exactly. 00:26:43.480 --> 00:26:47.550 [unclear] AssumeValid, right? He intentionally left that out for the reason that 00:26:47.550 --> 00:26:51.890 it is a shortcut, and he wants to make a fair comparison, which I think that's 00:26:51.890 --> 00:26:52.590