Syncing Old Bitcoin nodes - Episode 55

Aaron van Wirdum Live from Utrecht, this is Bitcoin.

Sjors Provoost Explained.

Aaron van Wirdum There you go, Sjors. How are you?

Sjors Provoost Excellent.

Aaron van Wirdum Back in good old Utrecht, we were both in London.

Sjors Provoost That's right.

Aaron van Wirdum Planning to record some episodes there, but then everybody got COVID.

Sjors Provoost Yes, and also it's a conference, so it's always a bit difficult to record.

Aaron van Wirdum So back in Utrecht, and we're going to discuss a blog post. Well, we're going to... discuss a concept that was described in a blog post by Jameson Lopp.

Sjors Provoost That's right.

Aaron van Wirdum He actually published a couple of blog posts about backwards compatibility, I guess, or syncing old nodes. That's really what the blog posts were about.

Sjors Provoost Exactly.

Aaron van Wirdum Yeah, so Bitcoin Core, of course, releases two new clients each year or at least two new major releases each year, right?

Sjors Provoost Yeah, roughly.

Aaron van Wirdum And these blog posts were about syncing older Bitcoin Core nodes to see how fast they would actually sync to the current chain tip right?

Sjors Provoost That's right.

Aaron van Wirdum And I think you've done similar research a couple years ago is that right?

Sjors Provoost Yeah so back in 2017 um what I did was I took a couple of Amazon AWS machines so these are just cloud machines, and install different versions of Bitcoin Core on those machines, and then just have them sync the Bitcoin blockchain as far as possible, and then I would just measure how long it would take, and those pretty big differences. And he's basically done that again, probably slightly different, but the same idea and with more recent versions.

Aaron van Wirdum Right.

Sjors Provoost And also with some older versions.

Aaron van Wirdum Yeah, so to be very specific, I already mentioned this, but there were basically two blog posts by Jameson, and the first one was really about syncing old nodes, and I would say the second one was actually mainly about backwards compatibility and forwards compatibility, which, you know, it's a related issue.

Sjors Provoost Yeah, and he also zoomed in on some of the really old nodes specifically, like the version 0.3, 0.4, 0.5 from many, many years ago and how difficult it is to get these things to work at all.

Aaron van Wirdum Yes, well, that's the backwards compatibility part, right?

Sjors Provoost Yeah, arguably. I mean, we can get into whether it's backwards compatibility or whether it's some other problems that he was running into.

Aaron van Wirdum Which we will get into. Yes all right. First of all for the people that don't know this, what does that actually mean? What is backwards compatibility? Or Sjors, forwards compatibility, because sometimes the terms are sort of used interchangeably.

Sjors Provoost Yeah, so I'm also going to be sloppy about it. The general idea is that you want —uh, you don't want to force people to upgrade their nodes at least definitely not immediately. So older nodes should work well with newer nodes. And if you know if you change… if you do this soft fork, you know the the new nodes will be able to the old nodes will just keep working fine while everybody… while other people upgrade to a new node. um Yeah,

Aaron van Wirdum Exactly. So like I mentioned, Bitcoin Core releases two new major releases every year. And the goal is that these new releases won't fall out of sync with older nodes or the other way around, depending on how you look at it. The goal is to keep everyone in consensus, including older nodes. Why do we think this is important?

Sjors Provoost Well, there are multiple reasons for it. Because you want to keep the system as voluntary as possible. There might be reasons why people are worried about installing new versions of Bitcoin Core. Maybe they want more people to review it. If suddenly you have only two days to upgrade your node, maybe there's some malware in it. But if you wait a year, it's more likely somebody would have already found it. It's also just maybe not possible to upgrade. Maybe you're busy doing other things. you know.

Aaron van Wirdum Yeah, maybe you want to [unclear] the code yourself right? Yeah, it's at least possible someone might want to do that, and that can take a little bit of time, and therefore it's important that they can keep using Bitcoin in the meantime.

Sjors Provoost Yeah, and and companies you know, that are running exchanges and whatever, they might be busy, they don't necessarily have… you know they don't necessarily can do everything for you on January 1st of some year. So it's nice if they don't have to upgrade. Though it's always a good idea to upgrade because you tend to stay up to date on security fixes and performance improvements.

Aaron van Wirdum Yeah, I would argue there's also kind of... And we'll move on after this. But I would argue there's also kind of a philosophical reason. Like in our previous episode, we were discussing like very hypothetical situations where there's a chain split. And like if you're... If you're asking the question, what is actually Bitcoin, then it makes sense that older nodes, if they are still syncing, someone never upgraded their node, why wouldn't that be Bitcoin? That was a big theme during the block size wars. Can you just change the protocol? And there was this argument about backwards compatibility. It's a philosophical argument that if old nodes still sync, then obviously that's still Bitcoin, right?

Sjors Provoost Yeah. But as we'll find out now, there is some nuance to that argument because trying to run a node from 12 years ago is not possible, probably.

Aaron van Wirdum For sure. Well, let's get into that right now, I think. Let's start there. So Jameson did these experiments with some older nodes. And when we say older, how old are we talking?

Sjors Provoost I think as old as 2012, maybe even 2011.

Aaron van Wirdum Yeah, that era. 2011, 2012, 2013. It was pre-Bitcoin 0.7, right? Which ones [unclear]? 4, 5, 6, 7?

Sjors Provoost Even 3.

Aaron van Wirdum 3, 4, 5, 6, 7. Or 3, 4, 5, 6 in this latest blog post, right? Yeah. And these did not sync. Did all of them not sync?

Sjors Provoost I think at least until 0.5, it could not get to sync.

Aaron van Wirdum Right.

Sjors Provoost And that also reflects my own experience when I tried that in 2017.

Aaron van Wirdum Nodes older than 0.5 will not sync.

Sjors Provoost 0.5 wouldn't sync either.

Aaron van Wirdum Right. Okay. And then we get into some of the reasons as far as we can tell because... My reading of the blog post is not entirely clear every time why exactly. It's clear where the nodes stop syncing, but it's not necessarily clear why exactly it stops syncing. Is that right?

Sjors Provoost Yeah, but we can get into some of the reasons, and eventually we'll find out that with 0.5, I wasn't able to sync it all the way, he wasn't able to sync it all the way, but there's some recent developments that may still be ways to get it to sync. We simply don't know yet.

Aaron van Wirdum Okay, so where does it stop or what's the reason as far as we know? What's the story here?

Sjors Provoost Yeah, so there's a couple of things and those really all boil down to the theme of dependencies. So that's a fun topic always.

Aaron van Wirdum I think we discussed this in a previous episode, but maybe you should...

Sjors Provoost Very, very long time ago. So our very first episode, I think, or maybe the second episode, we talked about dependencies like OpenSSL. And how, you know, when those dependencies change…

Aaron van Wirdum It's very basic. What is a dependency?

Sjors Provoost [unclear]… Or a library. Yeah. So dependency is a piece of software that you include in your software, basically. So in the case of OpenSSL, it's a very well-known dependency. It's a cryptographic library. So it means that Satoshi did not have to implement all of the elliptic curve cryptography, he could just use OpenSSL and let it handle all the cryptography.

Aaron van Wirdum Right, so OpenSSL is being developed by another group of developers and their code is essentially used in Bitcoin Core.

Sjors Provoost That's right.

Aaron van Wirdum And then it's called a dependency or the problem is if these other developers change something that might affect your own software, right? That's sort of the problem.

Sjors Provoost Yeah, exactly. And this can be especially a problem when the dependency... secretly, or not secretly, but like accidentally imposes rules that you don't know are rules.

Aaron van Wirdum Right.

Sjors Provoost And, well, there's a couple of, there's probably three different incidents of that historically, at least three.

Aaron van Wirdum You mean in Bitcoin or?

Sjors Provoost Yeah, in Bitcoin.

Aaron van Wirdum Yeah, okay.

Sjors Provoost I mean, in general, dependencies, of course, cause all sorts of headaches. I think it was in the news last week that somebody in the NPM, the Node Package Manager, which is a JavaScript library that a lot of... Any website that looks even a little bit fancy probably uses that in the background. And it turns out that if you were a developer and you were based in Russia or Belarus, your computer would get wiped if you updated the dependency. So dependencies are a nightmare in general, but we won't go into too much detail about that nightmare.

Aaron van Wirdum Yeah, let's stick to the Bitcoin.

Sjors Provoost Yeah, so for Bitcoin, we can start with one dependency, which is called BerkeleyDB. BerkeleyDB is a database technology that was initially used to store, I think, information about the UTXO set, or at least information about Bitcoin transactions. So the node was using that dependency, which is built by other people. And when the Bitcoin blockchain was small, when blocks were small, Every block would just add a bunch of transactions to the database. I'm kind of simplifying it partially because I don't know the precise details. But let's say a block contains 100 transactions and it would just add those 100 transactions to the database and the database would be happy. However, it turns out that if you crank up the number of transactions you start putting in the database, so if the blocks get bigger, at some point the database won't allow you to add these blocks. However, that never happened in practice because Bitcoin switched to a new database type in about 2013 called LevelDB.

Aaron van Wirdum Right.

Sjors Provoost And this LevelDB, you know, it's much faster and there was other reasons why it was better. But what happens is at that point, blocks started getting even bigger and LevelDB had no problem with it. But because we said we want to be backwards compatible. Some people were still running nodes that use BerkeleyDB and those nodes started causing problems because they started hitting those limits. And so the database would say, hey, this block is too large. Kind of.

Aaron van Wirdum Right. So if a big block, and by big we mean one megabyte, I guess, or maybe close to one megabyte…

Sjors Provoost Yeah, or 300 kilobytes. I don't know where it went wrong.

Aaron van Wirdum Whatever it was. If a big block would have been mined, say, in 2012, would that have meant that Bitcoin... Core nodes it wasn't even called Bitcoin Core back then, Bitcoin nodes then would have crashed is that sort of…

Sjors Provoost I think they would have not have crashed, but they would have rejected the block, but it was worse than that because it was not deterministic, so it's like some Bitcoin Core nodes would have stopped and others would not have. Um so because you were hitting internal, basically internal intricacies of the database package, it wasn't like a clear coded limit, just running into the internal magic. And so what ended up happening was a short chain split where the decision was made to undo the update and then everybody was told to either update to a new version of Bitcoin Core or change a slight setting in that database.

Aaron van Wirdum Right, but my… question and you've answered it but just to be clear, essentially what you're saying is there was actually a problem in Bitcoin clients back then. We just never noticed it because all blocks were almost empty or very small. Yeah and only if you start to try to sync it now, now the blocks are big, it's that the problem is really revealed. So there was kind of a bug in old nodes right?

Sjors Provoost Yeah but it got revealed as soon as they upgraded it, and my guess would be that you know, the miners are also running the same software to create the blocks, and so they wouldn't have created those big blocks in the first place. So maybe that's why it happened at the same time. Or maybe, you know, it could have been deliberate. But it turns out that had that upgrade not happened, we would have run into another bottleneck in that same software later on. And that's what me, I think both me and Jameson ran into. That if you keep running that old software...

Aaron van Wirdum Sorry, this is the same dependency. Also BerkeleyDB? We're still talking about BerkeleyDB.

Sjors Provoost So if you stick to BerkeleyDB, if you decide, you know, you go back in your time machine to 2013, you say, you know what, I don't want to install this update. I'll just change this configuration as people recommend, and then I'll just keep going. If you do that, then I think in 2015 or something like that, you start running into blocks that will still crash your database. And that's the new development where I posted a question on Stack Overflow years and years ago asking I'm running you know, I can't get past this block with this old version of the software. Can anybody help? And nobody replied to that until yesterday. Andrew Chow actually replied to it and had some suggestions on how to get past those error messages. This is why it's still a mystery. We it's possible that with a few other changes in the settings you know, it might still be able to reach the tip of the modern blockchain. My guess is it will run into other limitations in that database because it's just not been tested with the giant blockchain that we have today.

Aaron van Wirdum Right. Okay. So again, like the previous example, essentially there was always a problem in Bitcoin code back then and it was just never revealed because the blocks never were created that were big enough to actually reveal the problem.

Sjors Provoost Exactly.

Aaron van Wirdum It's only if you... sync an old node that the problem will reveal itself. So we've had two of these problems, two twice.

Sjors Provoost We've had one that people really noticed it in the wild that caused an actual split. And we've had one that you can notice yourself if you try to use very old software.

Aaron van Wirdum Right. And you mentioned there might be even more if we get past this new block and someone tries syncing again and new problems. So basically this Berkeley DB database, that was kind of a problem and we got rid of it. And now the new... Nodes perform better for that reason.

Sjors Provoost Yeah.

Aaron van Wirdum Is that right?

Sjors Provoost Yeah. And so there's a second dependency that's been a cause of headaches. And we have discussed that one in the earlier podcast, which is OpenSSL.

Aaron van Wirdum We discussed it in this podcast, I think, a minute ago.

Sjors Provoost But specifically, the problem there is that OpenSSL, the very old versions of OpenSSL, so the ones that Satoshi installed, those were very permissive in terms of how the signature could be encoded. So you couldn't make a fake signature, but the way you write a signature down, it could have like 75 characters or maybe more. And later versions of OpenSSL said, no, no, no, we're going to make it more precise. You have to only use, I don't know, 73 characters for your signature and no more. And it happened to be, if I understand correctly, that there never was any signature with that many characters in practice. No, I think there was. So let me see if I get the problem. So yeah, so I think there was a signature that was bigger, but nobody ran into the problem because if you had installed an old version of Bitcoin Core back in the day, you would have had the old version of OpenSSL and therefore you would have been able to sync just fine with these bigger signatures. But if now you try to install that old version, today, on a modern computer, you're going to have a modern version of OpenSSL and you're going to produce a version of Bitcoin Core that will not accept these old blocks.

Aaron van Wirdum Right, yeah. So this is kind of the opposite problem as the previous one. The previous problem we just mentioned with Berkeley is essentially that Bitcoin Core clients were always, you know, they always had these bugs essentially, but the bugs were never revealed because blocks were never big enough to reveal them. And now the problem is kind of different because Jameson was doing these tests. He had to rebuild old clients. And while he was rebuilding the old Bitcoin Core clients, the only way he knew to do it or... was by... I don't know how this works.

Sjors Provoost Yeah, he was using the most recent version of the dependencies.

Aaron van Wirdum Yeah, so then you get a new dependency. So now we have sort of the original dependency problem that I discussed is that the dependency changes and now the original client doesn't work anymore. However, so here the problem would be the opposite way that if you were still running that old software, you never changed it, you're still using your old computer from 2012 or whatever. you would still actually sync to the current blockchain because then you're also still using this old OpenSSL code, right?

Sjors Provoost Yeah, and in fact, there was a soft fork at some point that restricted the size of signatures in the consensus rules. But again, because it was a soft fork, your old node would be fine with the new rules, but it would not detect a violation of the new rules if somebody produced a signature that we no longer consider invalid. But there was, again, a transition period, I think it might have been 2013 or even 2015, where this was actually a problem for people who were compiling their own nodes. They actually had to be careful to when they compile their own nodes to either use an old version of OpenSSL or apply a little patch to a Bitcoin Core that would work around the newer versions of OpenSSL.

Aaron van Wirdum Right. Does this have anything to do with the 2013 blockchain split that actually happened for like six blocks?

Sjors Provoost No, that was the one we just talked about. That was a Berkeley DB.

Aaron van Wirdum Oh, that was a Berkeley DB problem.

Sjors Provoost And I think this is later. I think this is 2015, but I'm not sure. In any case, this was a problem in reality for people who compile their own nodes. It was not a problem for people who download the nodes because the downloaded versions would all have the old version of OpenSSL in it or the newer version would have the workaround in it. It was never a problem when you download Bitcoin Core. It is a problem when you compile it yourself. And it was even a problem back then.

Aaron van Wirdum Right. At the risk of getting sidetracked a little bit, would it be possible to... you know, basically rebuild an old client also using like an old library? I guess you'd have to find it somewhere and that's the hard part then?

Sjors Provoost Yeah, no, exactly. I mean, a lot of the open source libraries are still available somewhere. It gets a bit more. I think that's what Jameson did. He installed like a super old version of Ubuntu and then tried to also find some of the very old packages. But the problem is, of course, most operating systems that you would be building on will have the most recent versions of the libraries. And obviously you do not want to actually use a computer with an outdated version of OpenSSL because you'd have heartbleeds and all these like very bad security vulnerabilities on your computer. But there are ways to do it, even safe ways to do it. Yeah.

Aaron van Wirdum Right.

Sjors Provoost So this is a problem for, say, an historian like a thousand years from now. If you're a historian and you want to really study Bitcoin in the old days, you're going to have a problem because a thousand years from now, you'll probably still find the old source code of Bitcoin. But you'll have a very hard time building a computer, even a virtual computer, that will behave the same as a computer from today.

Aaron van Wirdum Right. So we've now discussed three reasons why old nodes stalled. I think they were three, right? Have we covered them all? Right. And just to reiterate, in two of these cases, they were actually bugs in the old code, or essentially, right? Compared to... you can't sync to new blocks because... Do we call them bugs? I would.

Sjors Provoost Yeah, I think so.

Aaron van Wirdum Yeah, all right. So in two cases, there were actually old bugs and old code that just weren't revealed until, you know, much later when no one was using it anymore, essentially. And in one case, it was because of the dependency changed.

Sjors Provoost Yeah.

Aaron van Wirdum And the old code would have actually synced to the new blockchain.

Sjors Provoost Yeah.

Aaron van Wirdum All right. Let's get into the next part of the podcast then, which is syncing. So syncing is probably one of the main user experience bottlenecks. Can we call it that?

Sjors Provoost Yeah, because it's the first thing you do when you install Bitcoin, when you install Bitcoin Core. And the first thing you do is sit and wait for a long time. And so it's not a great experience. Unless you really like progress bars.

Aaron van Wirdum Yeah, and the reason for that is, of course, because you're essentially... rechecking or learning about the entire history of the blockchain. You're downloading and checking the entire history of the blockchain in order to get up to date with what's the state of Bitcoin today, who owns what.

Sjors Provoost Exactly. And in earlier episodes, we've talked about some ways to either speed that up or to skip it altogether with things like AssumeUTXO. But for now, and probably for the foreseeable future, yeah, you need to check every single block, not just to know if it's correct, even just to know which coins actually exist.

Aaron van Wirdum Right. And the goal for every new Bitcoin Core release is essentially to make this a bit faster, I think.

Sjors Provoost Not for every Bitcoin Core release, but there's this repeating theme. It's not even a goal. It's just that it gets better very often with little changes or with bigger changes.

Aaron van Wirdum Yeah, well, actually, when we say better...

Sjors Provoost Faster.

Aaron van Wirdum Yeah, well when we say faster, and I guess this is part of the research, the thing is, every… by the time there's new Bitcoin Core release the blockchain itself has also become bigger.

Sjors Provoost Yes.

Aaron van Wirdum So does it actually take more time year over year to sync, or do the performance improvements keep up? That’s sort of one of the questions.

Sjors Provoost I think there have been times when the performance improvements kept up. I don't think that's the case anymore.

Aaron van Wirdum Right.

Sjors Provoost So it is getting worse. So there's a couple of things that are good. There's Moore's law. That means new hardware is just faster. If you have a new computer, you'll sync faster. If you have more RAM, you'll sync faster. If you have a faster disk. And the other is these improvements. But in the long run, it's very hard for them. I think when I looked at it in 2017, it was basically just almost constant. Despite dramatic improvements in the performance of the code, it was just getting... it was taking about as long. But of course, the hardware is getting faster too. When Jameson and I were running these tests, we were running the same modern hardware to run the tests on, just to have a bit of a fair comparison. But I think people like Luke Dashjr have done projections on it, like how long is it going to take until it actually starts getting faster just based on Moore's law. And I think the estimates are like 2040 or later when this growth in the blockchain is no longer as bad. They won't outpace anymore the Moore's law.

Aaron van Wirdum Okay, sure. So tell our listeners something interesting about this. How long does it take to sync?

Sjors Provoost So that really depends on your computer and how well you configure it, the Bitcoin Core node. So with the default settings, it's not as fast as it can be. There was a test by Jonas Schnelli, a former Bitcoin Core developer and maintainer. And I think two years ago, he got an Apple M1 chip, one of those Mac minis. But he only had four gigabytes of RAM. And the first time he tried it, he was able to sync the whole chain in 12 hours, I think. And then somebody suggested a few better settings to him. And it was six hours. So that was like state-of-the-art hardware six hours two years ago. But if he had used more RAM, he might have been able to do it a little bit faster, maybe in four hours. But if you have a super slow Raspberry Pi, it might take you two or three weeks.

Aaron van Wirdum Sure.

Sjors Provoost And if you're running, say, a web server, so say you want to install a BTCPay server somewhere, and you want on that server to sync the node from scratch, it's going to take you weeks. Because these web servers typically are very low performance. Once it's caught up, it's fine. It'll catch up with all the new blocks. But catching it up from the Genesis block, it has low memory, usually only has one or two CPUs, and it's all virtual, so it's pretty horrible.

Aaron van Wirdum Yeah this is one of these things that you really kind of want to see in a graph and I'll try to remember to include the links to the actual blog posts in the show notes, but can you tell us anything interesting about the improvements over the years?

Sjors Provoost Yeah.

Aaron van Wirdum How much has it improved?

Sjors Provoost Pretty dramatic. So one of the, like, I think Jameson found like a factor of 40, depending on how you measure it.

Aaron van Wirdum 40 since when?

Sjors Provoost Since these very old versions.

Aaron van Wirdum Since like 2012. 0.4 or whatever. Yeah. Okay.

Sjors Provoost But it's probably more than that. And there's a couple of reasons. I mean, some of you might call cheating, but it's not because it actually improves the experience for the user. So we did an episode about AssumeValid which is where basically the developers pick a block that's before a release, and then the Bitcoin Core node doesn't verify the signatures up until that block. It verifies everything else, but not the signatures, and that saves, I don't know, a factor two at least.

Aaron van Wirdum Yeah, well, I wouldn't call that cheating, but it's definitely a shortcut.

Sjors Provoost Yeah, I mean, you can turn it off if you don't like it for all these reasons, but it speeds things up a lot. Another improvement was, well, we talked about it, LevelDB. So switching from BerkeleyDB, the database that had all these problems, but it was also slower to LevelDB.

Aaron van Wirdum By the way, you just mentioned, sorry, I'm interrupting you, but on the previous point, you mentioned AssumeValid, but Jameson actually did not.

Sjors Provoost No, exactly.

Aaron van Wirdum [unclear] AssumeValid, right? He intentionally left that out for the reason that it is a shortcut, and he wants to make a fair comparison, which I think that's how I would have done it.

Sjors Provoost Yeah, it's a reasonable thing, and that's also what I did.

Aaron van Wirdum So you mentioned a 40x improvement, but that's without AssumeValid. So with AssumeValid, it would be even more.

Sjors Provoost Yeah, exactly. I think so. And I did that in my blog post from 2017. I did two things. I think I showed the difference between AssumeValid and Not AssumeValid, and it was quite dramatic. And then I also showed... the differences you know, with AssumeValid turned off between the older and the newer versions which is also dramatic. So in terms of things that were improved, like we said, AssumeValid, LevelDB. Another thing is uh, this is done many many years ago, as validating signatures in parallel. So if you're if you're validating a block, one of the things you need to do is actually check the signatures. And if you have multiple processor cores on your computer, which you probably do, one of the improvements was just to make these multiple processors verifying multiple signatures at the same time.

Aaron van Wirdum Right. So that obviously doesn't help if you use AssumeValid, or at least it doesn't help until that AssumeValid… But after that it works, or if you have... AssumeValid switched off, then it helps as well.

Sjors Provoost In general, when you're syncing the node, you'll notice with the default settings, you'll notice in the beginning it's mostly disk usage. And then after a while, your CPU starts getting really hot. When you go past the AssumeValid block, your CPU starts running at 100% and gets really hot.

Aaron van Wirdum Right.

Sjors Provoost Because then it has to validate all these signatures. Another thing that was done that was probably at least marginal, but maybe even more, is just the download blocks in parallel. So rather than getting one block at a time, you would get more of them at the same time from multiple peers.

Aaron van Wirdum Right.

Sjors Provoost We talked about that. The other, I think it was almost a 5X improvement, was getting rid of OpenSSL and using libsecp256k1 that we did a whole episode on.

Aaron van Wirdum Oh, yeah. So we haven't mentioned that, but so that's an entirely, like that's a Bitcoin native library, right?

Sjors Provoost Yeah.

Aaron van Wirdum Am I describing that right?

Sjors Provoost Yeah. It's basically, instead of using OpenSSL for all the signatures, it uses libsecp, which Pieter Wuille wrote at the time. And it was initially designed as a performance improvement, but it also helped dealing with these bugs that we just talked about. So kind of nice...

Aaron van Wirdum Yeah, so we got rid of OpenSSL altogether, and that's also a performance improvement. It's also faster.

Sjors Provoost Yes, exactly. Another thing that happened in, I think, 2017 was that the UTXO database, just the database structure was changed. If you structure a database in a more intelligent way, it can perform better.

Aaron van Wirdum Right.

Sjors Provoost That, I think, was in turn the cause of... this very scary inflation bug in 2018. So, you know, it's nice to improve performance, but you want to be extremely careful. But I think this performance improvement was actually necessary because the performance was so bad that somebody could attack Bitcoin by exploiting the bad performance.

Aaron van Wirdum Right. How?

Sjors Provoost By just creating very unfavorable transactions so that nodes get slower and slower and slower.

Aaron van Wirdum That would get so slow that it takes longer to verify a block than to create a block, essentially.

Sjors Provoost I guess that would be the absolute worst case. I don't... I mean, not to... I mean, yeah, blocks come in every 10 minutes, so if a node cannot verify a block within 10 minutes, you're doomed. I don't think it was that bad.

Aaron van Wirdum Well, there were some older clients. I read this maybe in a Reddit comment under one of these blog posts that some of the very old clients would actually have that problem now. They couldn't sync because it just takes too long to download and verify a block. And by that time, there's more blocks.

Sjors Provoost Yeah, I would not be surprised. And keep in mind that these would also be running on older hardware. So if you started a node and you put it in your nuclear bunker and you can't get back into your nuclear bunker, then at some point it's just not going to be able to keep up anymore. But, you know, let's see another one. There's lots of very small improvements. There's not even a complete list of them. For example, part of the block consists of checking hashes. It's SHA-256 hashes. And it turns out that many modern computer chips have special instructions on them to do SHA-256. They don't have those instructions because the chip makers like Bitcoin, but because SHA-256 is used in a lot of places.

Aaron van Wirdum Right.

Sjors Provoost And so if you actually take advantage of those special instructions, which involves writing some machine code... then it'll get faster. And I think very recently, one of those things was done for ARMv8 chips, which are used in Raspberry Pis, but also used in the Apple M1 chip. So that'll probably speed it up.

Aaron van Wirdum But you're talking about hardware now. Has there been Bitcoin Core codes released that takes advantage of this?

Sjors Provoost Yeah, exactly. So when a specific chip has a feature that could make Bitcoin Core faster, then Bitcoin Core developers could if they want to write special machine code for that chip. So the software, when you compile it, will just check whether or not you have this chip.

Aaron van Wirdum And they did, or you're not sure?

Sjors Provoost Yeah, they have. I mean, it's been done over time for different chips, for different optimizations.

Aaron van Wirdum Got it.

Sjors Provoost So the most recent one I'm aware of is for this ARMv8, which I think is what's used by the Apple M1 and by many modern Raspberry Pis, maybe not the old ones. Another thing, I don't know, some magic math thing in libsecp that made 15% faster signature validation. Those things, I can't review them because I just see magic. The only thing I can see is like, hey, the test still passed and I can still sync the blockchain without getting a consensus error. So I guess it's correct.

Aaron van Wirdum I think SegWit could be considered an improved... Well...

Sjors Provoost No, it's a mitigation of not making it worse. So the block size was increased with SegWit, but the damage caused by that block size increase was mitigated because the SegWit rules put some restrictions on the transaction they can put into it. Because basically, I think there's some things that you can do in transactions that take quadratic time to validate. So if you add five of these things, it's 25 times more expensive to verify it. And I think SegWit got rid of those problems so that you can double the block size and only double the time it takes to validate rather than quadruple or, you know, [unclear] the time it takes to validate.

Aaron van Wirdum Well, that's not what I was referring to. What I was referring to, actually, this was one of the things that Lopp mentioned in his blog post is that actually older nodes will sync faster post the SegWit activation point.

Sjors Provoost Well, because they don't verify any of the signatures for SegWit.

Aaron van Wirdum Right. But it still counts, don't you think? Well, it's also kind of a shortcut.

Sjors Provoost Yeah, but I mean, if you want to go that route, I can think of some even scarier proposals where we just, you know, make all the blocks empty for all nodes and just put the real transaction somewhere else.

Aaron van Wirdum Well, yeah, now that you mentioned empty blocks, that's also maybe interesting to mention.

Sjors Provoost That's definitely a way to make really fast validation.

Aaron van Wirdum At the first I think, 200,000 blocks, so basically, the first four years, for the first four years it doesn't really matter that much which node you're running because all blocks were empty anyways, right?

Sjors Provoost Yeah, I'm sure it got a little bit better. The the newer nodes are even better at running through empty blocks.

Aaron van Wirdum Yeah, but but…

Sjors Provoost Generally when you start up a node, the first couple hundred thousand blocks will fly by in 15 minutes, and then it just starts taking a really long time.

Aaron van Wirdum Yeah, exactly. All right, well, so you've listed a number of performance improvements over time. And do you want to keep going?

Sjors Provoost No, that was the whole list I had.

Aaron van Wirdum Okay.

Sjors Provoost But there's probably more out there.

Aaron van Wirdum Yeah, Bitcoin Core developers are squeaking out… squeezing out new improvements.

Sjors Provoost Yeah, but it's getting more and more marginal. I mean, if you make a 5% improvement, but the chain gets 20% bigger, then it's not enough.

Aaron van Wirdum Okay, Sjors, well. I guess that's the podcast then or not?

Sjors Provoost Yep, I think so. So thank you for listening to Bitcoin Explained.

Aaron van Wirdum There you go.