Episode 70: Bitcoin Core v24 Bug

Aaron van Wirdum Live from Utrecht, this is Bitcoin.

Sjors Provoost Explained.

Aaron van Wirdum Sjors, welcome back.

Sjors Provoost Thank you.

Aaron van Wirdum I saw you were promoting your book everywhere in the world over the past couple of weeks.

Sjors Provoost Absolutely.

Aaron van Wirdum Where did you go?

Sjors Provoost I went to a place called New York, a place called Nashville, and a place called Austin. And those are all in the United States.

Aaron van Wirdum That sounds very exotic.

Sjors Provoost They are.

Aaron van Wirdum And you were promoting your book. Which book is this, Sjors? Do you want to promote it on our show as well?

Sjors Provoost Yeah. So in case nobody's heard of it, it is called... Bitcoin, a work in progress. And if you want to find it, because of the way that Amazon actually messed up the indexing, so they basically took the colon in the name, Bitcoin, a work in progress, and treated that one as a title and the other as a subtitle. So you want to look for Sjors Bitcoin, and then you'll find it.

Aaron van Wirdum Great sales pitch.

Sjors Provoost Or you can go to btcwip.com and you'll find it too.

Aaron van Wirdum So I saw you promoted on Marty Bent’s podcast.

Sjors Provoost Exactly.

Aaron van Wirdum TFTC.

Sjors Provoost Yeah, that was a bit unplanned because we happened to be in the same building. And they said, hey, do you want to come on the podcast? And then we can talk about the book. So we talked about that, Tales of the Crypt.

Aaron van Wirdum Yeah, I saw that. So that's why there was no Bitcoin Explained episode for the past couple of weeks. But we're back, our dear listener, with this Christmas special of Bitcoin Explained. Actually I guess by the time this is released it's not gonna be Christmas anymore but we're recording it…

Sjors Provoost Yeah the last episode of 2022.

Aaron van Wirdum Pretty close to Christmas anyways. Yeah well I mean I guess it kind of depends on how I'm not 100 sure if our great super talented editors actually working between Christmas and new year's we'll have to…

Sjors Provoost Well it's the last episode that was recorded in 2022.

Aaron van Wirdum Yes that's for sure. So we got a very special for you today Dear listener, I hope you're strapped in. This is going to be a major one. Sjors is going to educate you about the most important development of the year.

Sjors Provoost Yes, we have scraped the absolute bottom of the absolute last barrel that we could find after like an hour of brainstorming topics.

Aaron van Wirdum We wanted to bring you something before the end of the year, because otherwise you would have had to manage probably five to six weeks without our voice. And we didn't want to do that to you, dear listener. So here we are. Back with an episode about a bug that was kind of in Bitcoin Core 24.0 but okay first of all Sjors, let's let's back up a little bit here. Let's take it seriously it's a serious issue. Sjors, we had an episode on Bitcoin Core 24.0.

Sjors Provoost Yes.

Aaron van Wirdum And then that episode well no we aired the episode. But then the release of Bitcoin Core 24 was delayed by quite a bit and that was because a discussion about full RBF. There was an option to switch on full RBF in the Bitcoin Core 24 node. And that sort of pushed back the actual release of Bitcoin Core 24 and then finally Bitcoin Core 24 was released but then it kind of wasn't.

Sjors Provoost Yeah that was a bug in it but the release was announced but if you went to the website bitcoincore.org you would actually still find version 23 so only some advanced users would have been able to download the actual release.

Aaron van Wirdum And then so hang on is if Bitcoin Core 24 is not on bitcoincore.org. Is it really released?

Sjors Provoost In a sense, yes, because there was a tag made on GitHub, which is signed, and there was an announcement. So people would be able to download it if they went straight to the FTP-like download site, where you just see the list of files, but not if you went to the main website, because that page takes a little bit longer for people to look at, and then that gets released as well. The page that points to the release.

Aaron van Wirdum Wait, so does the page usually refresh automatically to include the latest release?

Sjors Provoost No, there’s a pull… the website is on GitHub too. So people make a pull request to add a page that points to the new release. And maybe some release notes and other remarks. And that always takes a few days. And within that few days, the bug was found. And so that was postponed.

Aaron van Wirdum Okay, I see. So that was technically an official Bitcoin Core 24 release. But that release never showed up on bitcoincore.org. What you do find on bitcoincore.org right now is Bitcoin Core 24.0.1. And this is because there was a bug in the original. Technically officially released Bitcoin Core 24 release. Is that right?

Sjors Provoost Yeah, exactly. And when something like that happens, then the next release can be just called 24.0.1, basically not immediately 24.1.

Aaron van Wirdum Yeah. Well, why?

Sjors Provoost Because it's like a quick enough fix, basically.

Aaron van Wirdum Okay.

Sjors Provoost Probably nobody's going to be running the old version.

Aaron van Wirdum Right. So that gets us to… Yeah so hang on so first of all so there was a bug in 24 but that was not caught in the release process that's usually what… That's why there is the release candidates process, right?

Sjors Provoost Yeah. So the idea is the release candidates, you hope that people test it and find bugs. But of course, not everybody's doing that and not everybody will be testing every single possible feature in every way that you can break it. So that was not found in time.

Aaron van Wirdum Right. So a bug was found after it was officially released, but before it was published on the bitcoincore.org website. Okay, Sjors, What's the bug? What happened? What went wrong?

Sjors Provoost Well, let's make a little suspension here. And I'm going to read some boosts from the previous episode first.

Aaron van Wirdum Oh, yeah, we almost forgot about that. People are paying us money. We shouldn't forget about that at all. Go ahead, Sjors. And also tell our other listeners how they can pay us even more money.

Sjors Provoost Yeah, so if you have an app that supports the podcasting 2.0 standard, there's multiple apps, Fountain is one, then you can actually stream sats to us. And if you Google something like podcasting 2.0 streaming sats and you'll find podcasts that explain that in more detail and then you can also send us a message which is called a boost and we got well we got a bunch of messages I'll just read two.

Aaron van Wirdum These are the top paying ones?

Sjors Provoost Yes, though we don't promise to take the top-paying… I'll read the amount and the name, etc. So we have at Banana Man, or Banana Man, depending on if he's British or American, paying 12,000 sats, saying episode 69. Nice.

Aaron van Wirdum Nice.

Sjors Provoost Okay. And then we have...

Aaron van Wirdum Oh, that's it?

Sjors Provoost Yes. And then we have a question from at user 6025638884, well, la la la la.

Aaron van Wirdum No, no, this is important, Sjors. He's paying money. You better read his name right.

Sjors Provoost No, these are default usernames that are created by the app. So that's fine. He or she is paying 10,000 sats and has a question, which is related to the topic we talked about earlier, naming RBF. What makes a non-RBF transaction new to a node? And when is it just another version? Is it e.g. considered a new transaction if some of the input UTXO changes?

Aaron van Wirdum Sorry, can you repeat the question?

Sjors Provoost What makes a non-RBF transaction new?

Aaron van Wirdum What makes a non-RBF transaction new? All right, so what makes a transaction new?

Sjors Provoost Rather than another version of it.

Aaron van Wirdum Okay, yeah. Do you want to answer that?

Sjors Provoost Yeah, so my guess is, I mean, it depends on how your wallet wants to interpret these things. In essence, a transaction spends outputs. So if any output is spent by another transaction, then that is a different transaction.

Aaron van Wirdum I mean, I guess technically on a consensus level, any change to a transaction makes it a new transaction, right?

Sjors Provoost Exactly.

Aaron van Wirdum But there's some nuance here maybe when it comes to relaying.

Sjors Provoost Well, what makes it new versus another one? You could have... If you have one input and one transaction and then another input and another transaction, then those are different transactions. They don't exclude each other. But the case we're interested here is when you're replacing a transaction, that means that you have another transaction that spends the same input or at least one of the same inputs so that only one of those two can exist. And then the question is, you know, what is new and what is old? Well, part of that just depends on what you saw the first. As a node before RBF, which would just be like, if I already know a transaction, then whatever else comes in [unclear]. But that could be different for different nodes because you might receive them in different order. But whether you call it another version or whether you call it new, it just depends on the interpretation of the wallet, how it wants to display it. So it could, you know, I think in Bitcoin Core, it'll put a cross through the old one and then just put the new one in its place.

Aaron van Wirdum Yeah, I guess this question sort of derives probably from our conversation about these kinds of topics where we're using terms like new and version, but in actual Bitcoin protocol, there's no such distinction, I guess I would say, right?

Sjors Provoost Yeah, exactly. So at least I think not because you have the nVersion field, but I think with RBF, it's just a flag. It's not a number that goes up. The only thing that goes up is the fee. So if a wallet wants to be a little bit smart about it, you could look at which one pays the highest fee. And then that has to be, that's sort of the newer one. That's probably intended as the newer one. So if you want to make a good wallet UI, you might want to show something that it increased the fee rather than showing two transactions. But that's really just up to the wallet authors, how they want to interpret what they're seeing on the chain. The blockchain doesn't really care. Well, the mempool doesn't really care. The blockchain definitely doesn't care.

Aaron van Wirdum Yeah, I guess it's just way more binary. Either a transaction is valid or it's not. And if it's not, that can be because it's conflicting with the previous transaction.

Sjors Provoost Yeah, but then valid in the mempool depends on your mempool policy, right?

Aaron van Wirdum Yeah. All right. Well, that was maybe a very confusing answer to... I mean, it's a good question. It's a good question.

Sjors Provoost I would also recommend people, if you have questions, also ask them on Stack Overflow.

Aaron van Wirdum Yeah.

Sjors Provoost Bitcoin or Stack Exchange because you'll get better answers.

Aaron van Wirdum Like I said, I think the question kind of derives from our conversation about it. And it doesn't really map on the actual Bitcoin protocol. So that makes it a little bit hard to answer. But hopefully our attempt is at least worth something. How much did they pay for this answer?

Sjors Provoost 10,000 satoshis.

Aaron van Wirdum All right. Do you think this answer was worth 10,000 satoshis?

Sjors Provoost I think it was worth way more than 10,000 satoshis.

Aaron van Wirdum Okay, cool. So then we're back to the main topic, Sjors.

Sjors Provoost Yeah.

Aaron van Wirdum Okay, listeners, strap in again. Here you finally get the answer you've been waiting for. Sjors, there was a bug in Bitcoin Core 24. That's why there was a Bitcoin Core 24.0.1 release. What was this bug?

Sjors Provoost Yeah, it was a bit of a foot gun, basically. You could, well, not, I don't know if foot gun is the right word. You could lose a bunch of money. So in Bitcoin Core, when you're sending money, you can... Either let the wallet decide automatically which of your coins to use to send the money or you can manually specify exactly which coins you want to use or some combination of both where you say well I want to pay one Bitcoin to this guy and I want to use this specific 0.3 Bitcoin coin that I have But other than that, you figure out which coins to take from the wallet.

Aaron van Wirdum Right, yeah. When you're saying coins, the technical term here is UTXOs. I guess people that know that know what you mean anyway. So my addition doesn't really help anyone.

Sjors Provoost No, that's fine. Yeah, so basically it's like the analogy is you have a wallet and there's a bunch of coins and paper notes in it and you have to pay something and you... Take one euro coin out of it you say I definitely want to use this but then you have some other guy going through your wallet and deciding which other coins to use to add up to 10.

Aaron van Wirdum Well if you want to use the analogy sometimes when you're abroad in a country where you don't know which coins are what and you don't want to waste time you just sort of show all your coins to the merchant and then they'll just pick…

Sjors Provoost The coin selection for you?

Aaron van Wirdum Exactly that happens sometimes.

Sjors Provoost I have done that a few times.

Aaron van Wirdum So that would be the analogy here. Yeah.

Sjors Provoost Yeah so anyway there was a bug in that where if you want to stick to the analogy, I think what happened is you would pick one of those coins and then it would basically, well, let's not use the analogy, you would basically select a couple of coins and then due to the bug, it would use those coins twice to calculate what was going on.

Aaron van Wirdum Ah wait so okay so you got a number of coins UTXOs and when you're manually selecting them then for the fee calculation it would… Falsely assume that you selected them double?

Sjors Provoost Yeah exactly and for some reason the net result of that mistake is that you might be paying a very very large fee because you think you're paying say I don't know one cent of fees but actually because you've counted this 10 euros twice it somehow includes it twice. Okay I don't completely understand how it happened. But that was basically the nature of the bug. So you'd be paying a very large fee. Now, fortunately, there is a warning system. If you pay a really, really high fee, it will abort. But there's another feature in the wallet where... So it's kind of a different kind of feature where you say, I want to pay this address. And I don't care about the precise amount. I just want to send this coin or this set of coins. You select the coins that you want to send over. So this can be useful for privacy. If you want to, you know, you receive your salary or whatever, and that's one coin, essentially. And you want to send only your salary to an exchange and none of the other coins in your wallet. You want to mix it. So you select your salary and you send it to the exchange. And if you use that feature in combination with this bug, you would lose even more money and the warning system didn't work correctly. As you notice, I'm being a bit vague because the actual bug... Is not described very well. That is the actual implication of the bug is not described very well. The bug itself is described as what was wrong in the code and that's pretty obvious. So yeah, there was a pretty bad ways to lose some money. So don't use 24.0, just use 24.0.1.

Aaron van Wirdum All right. Well, so yeah, this sounds like a bug, a glitch. And I think you mentioned, I'm still going to ask, so how did this get into the code? How does... Why is this in there?

Sjors Provoost Yeah, so this is interesting, of course, to look at what exactly went wrong here. So the bug itself was like one line of code that should not be there. I think it was a command called break, which jumps out of a loop, and that command was a mistake. Now, how does that get there? It was part of a refactoring. So basically, refactoring means you take an existing piece of code that may be ugly, and you rewrite the same code but prettier, in a way that it doesn't change any of the behavior. And so that's good because cleaner, more readable code in the future will lead to fewer bugs. However, you might make a mistake replicating the original behavior. And that's what happened here. So I think in practice here, there was a bunch of different functions that were responsible for removing. So if you take these coins that you've manually selected, then it would have to remove those from... Like another set of coins so that you don't select coins twice, something like that. And right, because the coin selection algorithm, it looks through the coins in your wallet. It tries to see which coins can I add to this transaction, but it shouldn't be adding coins that are already in your transaction. So it kind of had to do the subtraction there. And that code was doing the subtraction only once. I think it was only subtracting one coin that you selected, but not all of them. So this is the kind of bug that you might not see if you only select one coin. You'd only see if you select two coins. So it's, you know, difficult to run into this. So that was, and the cleanup there, like, I don't know, the code was doing 12 separate steps to remove… To remove these coins, and with the new version, it would only need two steps to remove the coin. So it was just cleaner. But I had the mistake in there, and the mistake was not caught. It wasn't caught during the code review by the people looking at it. There was also no test that failed. So there was no test where this precise scenario of adding two coins, et cetera, et cetera, was there. Because either could have caught it. Somebody could have caught it in a review, or some automated test could have caught it, but it didn't. And so the way that that is fixed… Is, well, first you fix the original bug, and second you add tests that would have caught it, so that at least the same bug could not happen again. That's pretty much what happened. And the other reason it wasn't noticed is that it was only in the release and not in the master branch. So that's an interesting aspect there.

Aaron van Wirdum Okay, right. Okay, so the master branch is the main thing that Bitcoin Core developers work on. And once in a while, Bitcoin Core developers say, okay, from here on out, this is the new release. And they keep working on the master branch for new features. And the new release is going to be released. And somehow this bug was only in the release.

Sjors Provoost Yes, exactly. Because what happened, the bug was introduced in this refactor. And then after that bug was introduced, the split to the release was made. And then somewhat later on master, the bug was accidentally fixed, not deliberately fixed. And the way you accidentally fix something is you do another refactor and something, and then you actually make another mistake because you're not replicating the original behavior. But because nobody wanted that original behavior, nobody would have noticed that either. So yeah, it was accidentally fixed on master. And that also has to do with the odds of finding it. Because who is testing Bitcoin Core? Well, part of it is people running the master branch. Those are probably other Bitcoin Core developers that are just trying snapshots of the master branch. You know, are not supposed to be very safe. It's supposed to be correct, but it's more likely to have bugs. But because the bug was not in the master branch, nobody testing the master branch would have noticed. It's only people testing the release branch, and that might be a much smaller group of people in practice.

Aaron van Wirdum Right.

Sjors Provoost And also, like we described, it's not the easiest thing to trip on. But if you do trip on it, it's very painful, so it was definitely worth fixing.

Aaron van Wirdum Right. Okay. Well, there you have it. That's why the new release is Bitcoin Core 24.0.1 instead of just Bitcoin Core 24. You are enlightened, our dear listener.

Sjors Provoost Yes, there's two other bugs in there too, but they're less important and in very different areas.

Aaron van Wirdum Oh, that's kind of a depressing note to close the episode on.

Sjors Provoost There's always bugs. So what happens is after a release, the developers pile up existing bug fixes. For a future release. So there'll be a version 25.0, which will have the new master branch on it, but there'll also be a version 24.1, 24.2, which will have only sort of bugs, bug fixes in it that are important enough to put on the old releases.

Aaron van Wirdum Wait, wait, wait, wait. Are there still bugs in Bitcoin Core 24.0.1 as far as we know right now?

Sjors Provoost I don't know.

Aaron van Wirdum I thought that's what you said, but maybe I misheard.

Sjors Provoost No, the general process is that we'll collect bug fixes over time. So maybe immediately after a release, there is no known bug. Or maybe there's a known bug, but there's no fix for it yet. But then over time, say the next six months or a year, bugs will get fixed. We'll find out there were bugs in version 24. And then you can do two things to your users. You can say, well, tough, just download version 25 if you want that bug fixed. But then they have to download all these new changes in version 25, which people don't like. So there will also be version 24.1, which only fixes the bugs, but doesn't have any of the other new changes. That's called backporting.

Aaron van Wirdum Sjors, I just want you to tell our listeners that Bitcoin Core 24.0.1 is squeaky clean and we're ready for the new year. We're ready for Bitcoin Core. No, we're ready for 2023 with Bitcoin Core 24.0.1.

Sjors Provoost Well, all I can tell you is that currently there's a milestone called 24.1 and it doesn't have any bugs in it.

Aaron van Wirdum There we go. End of the episode. Thank you for listening to Bitcoin,

Sjors Provoost Explained.