Episode 102: Bitcoin Core 32.1 — Live

Sjors Provoost So with that, I think we should say...

Aaron van Wirdum Live from Amsterdam, this is Bitcoin.

Sjors Provoost Explained.

Aaron van Wirdum Hey, Sjors.

Sjors Provoost Hello.

Aaron van Wirdum This is the first time I can say that without lying.

Sjors Provoost Hey, Sjors?

Aaron van Wirdum No, yeah well, no, that we're live. That's that.

Sjors Provoost Oh, okay. Yes.

Aaron van Wirdum So I don't know if you even remember why I started saying that.

Sjors Provoost Why, you said live from Utrecht?

Aaron van Wirdum Yeah, I mean, we never actually record live, right?

Sjors Provoost Yes.

Aaron van Wirdum It's always recorded and even edited and... But yeah, CK at the time, our manager at Bitcoin Magazine, he said we needed a... That was the only pointer we got from him when we made our podcast. He said you need a way, like a standard way to open the podcast and a standard way to close the podcast. And otherwise, you know, you have free reign. And we didn't prepare anything. And I think at the time was just the first thing that came to mind. So we stuck with it.

Sjors Provoost Well, the first episode we ever recorded was a complete flop because we didn't prepare it. Because we thought like, ah, PSBT bugs in hardware wallets, that's not too complicated. We can explain it off the cuff. And like within two sentences, it was like, I have no idea how this works.

Aaron van Wirdum Oh, that's right. And also... I think the original concept for our podcast, we wanted to do two topics per episode. We want to do two topics per episode and because I at that time I was still kind of working as technical editor I was still on my late sort of stage of being the technical editor before writing a book and becoming editor-in-chief and and so I was still covering technical stuff like very regularly and I think the original idea was I will explain a topic you will explain a topic to me and I'll explain a topic to you because sometimes maybe I had something that you didn't know

Sjors Provoost You had an idea of explaining a topic to me?

Aaron van Wirdum Yeah, you don't even remember. That's how far we drifted from that. We completely abandoned that concept and now it's just me. Sjors knows everything and I know nothing basically.

Sjors Provoost Well, you do know a lot, but the recipe is usually we have like a call before and we did that today as well. to kind of go through it a bit and then hopefully the real thing will be better, but we'll see how it goes.

Aaron van Wirdum Yeah, and just out of interest. So I assume most people have at least listened to it. Is there anyone in the room that has never listened to our podcast? That doesn't know our podcast? Okay, that's actually unsurprisingly, but it's still a big number.

Sjors Provoost It also doesn't add up because I asked the opposite question and we got like 10 hands.

Aaron van Wirdum so a lot of neutrals is there anyone who knew that we do

Sjors Provoost So a lot of people have good OPSEC.

Aaron van Wirdum still make a podcast sometimes because we used to make it like weekly at one point and now it's three or four a year maybe anyway okay so

Sjors Provoost Yeah, I mean, part of the problem was just it became exponentially... In the beginning, we did episodes about topics that I already knew something about,

Aaron van Wirdum yeah yeah

Sjors Provoost but at some point you run out, and then I had to spend like a day preparing the podcast, which makes no sense unless you're like Peter McCormack famous.

Aaron van Wirdum yeah yeah and and also we want to make it in person and I haven't been living in

Sjors Provoost Yes,

Aaron van Wirdum the netherlands for the last year and a half two years so that doesn't help So anyways, we'll make a new episode. So yeah, the general concept for those to know basically every episode we pick a topic about Bitcoin, a technical topic that we then explain. And or we've also made, so now actually most of our podcasts are if there's a new Bitcoin Core release and then we kind of go over what's new in the new Bitcoin Core release. That's what we'll be doing today, right?

Sjors Provoost exactly.

Aaron van Wirdum So we're going to talk about Bitcoin Core 32. I don't even, is it even, is it released?

Sjors Provoost Point one, yes.

Aaron van Wirdum I don't even know that. When was it released?

Sjors Provoost So v32.0 was never released. V32.1 was released, I don't know, two weeks ago, one week ago.

Aaron van Wirdum Okay, so we actually, I didn't even know this, but it's a good timing to make this podcast.

Sjors Provoost Yes.

Aaron van Wirdum That's right, because we planned it a while ago.

Sjors Provoost Yes.

Aaron van Wirdum But we, Bitcoin Core comes out every like six months is, so we could at least make a good estimate. Yeah, so the way we make these episodes is we go through the release notes of Bitcoin Core, we write down the... four or five maybe six topics that sort of are more interesting because there's always a bunch of bug fixes and there's always a bunch of stuff that you can't really talk much about although in this episode we will talk about some bug fixes

Sjors Provoost Yes.

Aaron van Wirdum because they're particularly relevant but we'll start out So we, again, we picked

Sjors Provoost Yes.

Aaron van Wirdum out like a handful of topics that are interesting enough, at least for those that are interested in this kind of stuff, to talk about a bit. And then mostly Sjors will do the explaining and I'll do the asking questions.

Sjors Provoost Sounds good.

Aaron van Wirdum Good? You're ready?

Sjors Provoost Go for it.

Aaron van Wirdum Okay. So the first one we wrote down is proof of work. I can't even read my own handwriting.

Sjors Provoost Proof work with Tor.

Aaron van Wirdum Proof of work defenses. Yeah, so there's, okay, go on.

Sjors Provoost Yeah. So.

Aaron van Wirdum I'll hand you the floor.

Sjors Provoost So everybody presumably knows the Tor network. They've been under attack quite a lot in recent years by, well, they have some suspicions of who these people are, but it doesn't really matter. Their network is constantly under attack.

Aaron van Wirdum What is the attack?

Sjors Provoost Basically a DOS, denial of service, so your Tor node, especially your hidden service sites, becomes unreachable or you can't connect to something, and that's kind of annoying. And remember that...

Aaron van Wirdum Yeah, so you're running a Tor node or a Bitcoin node, I guess, over Tor and you just get tons of connection requests.

Sjors Provoost Yeah, so on Tor you are you know as a consumer you are browsing stuff on Tor network or outside and There are websites on Tor called the darknet But a Bitcoin node can actually be like a darknet website as in you can be reachable on Tor as a hidden service And yeah, so the Tor project itself came up with a solution, or at least a partial solution, called Proof of Work. We're familiar with this idea. It's not one difficulty, I believe. It's just every node for themselves decides how much they want the clients to work. And that's sort of a way for them to rate limit the attack. Makes the attack just more expensive, because the attacker has to do a bunch of work. Whereas the legitimate users can do it.

Aaron van Wirdum And the work like in Bitcoin, it just means you have to make a bunch of hashes until you come up with like a valid hash as determined by your node, like how many

Sjors Provoost Yeah, I don't know if it's a hash.

Aaron van Wirdum zeros it needs.

Sjors Provoost Yeah, I don't know what exactly the proof of work they use. Probably not SHA-256 or something else. But yeah, that's the idea. And now Bitcoin Core, because it can also be a listening Tor node, if you have that installed, it will turn that feature on. So there's nothing that Bitcoin Core is doing other than telling the Tor software, like, use this.

Aaron van Wirdum Right, okay, so I'm using Bitcoin Core over Tor and you are using Bitcoin Core over Tor, let's say I want to connect to your node and now I have to do some, my node has to do some work in order for that connection to be accepted by you.

Sjors Provoost Yeah, your Tor software has to do some work for my Tor software to

Aaron van Wirdum Yeah, right.

Sjors Provoost let the nodes talk to each other. So the nodes don't really bother with any of this. It's the underlying Tor software that's doing it.

Aaron van Wirdum Why do you think because I wrote a book right on the prehistory of Bitcoin so also about the invention of proof-of-work which Adam Back at least a Bitcoin version of it was invented by Adam Back and for him at the time he saw proof-of-work kind of as a stopgap until digital cash was invented as a way to make a payment basically right but you have to prove work so wouldn't it be much cooler if Tor actually starts to use Bitcoin now that Bitcoin exists and then you send a couple of sats to a node in order to be connected so we can skip that proof of work aspect.

Sjors Provoost If I take your question...

Aaron van Wirdum Why don't you lobby the Tor project to implement that?

Sjors Provoost Well, your question was, would it be cooler?

Aaron van Wirdum Yes, agreed.

Sjors Provoost And the answer is yes. The question is, would it be easier? And the answer is absolutely not. So the Bitcoin proof of work itself is ridiculously simple, right? You just count the number of zeros before the hash or some equivalent problem. And like I said in the introduction here, we don't need... global coordination of the proof of work. So in Bitcoin, the complexity is in coordinating all the nodes to agree on what the proof of work is, what the difficulty is. As far as I know, for Tor, it's a local thing. Your node is getting dosed. It just increases the proof of work. until the DOS becomes manageable. That's not a coordination problem, so it's just much easier to do that way. I could imagine one day perhaps them using something like ecash because, yes, they could use Lightning, but the problem with Lightning is now you need to have your Tor network, have the Lightning channels routing across the Tor network, but you have this chicken-egg problem. If your Tor network goes down, you can't pay somebody to bypass the DOS, but ecash would bypass it, but ecash, as in Cashu, is... custodial. But it could help here. And I think there's also a philosophical difference, which could be a legal difference. Right now, the Tor network is entirely volunteer run. As soon as you start paying people, governments have opinions about that that are different than when you're not paying people. Although, you know, we've seen that not always work out.

Aaron van Wirdum Yeah, okay. TLDR, it's just much easier to use proof of work and that's new.

Sjors Provoost In theory, your nodes on Tor should now be better reachable when there's a DOS attack. But I haven't looked at any measurements here.

Aaron van Wirdum Right. Okay, next point then. I believe this isn't even a new thing, but we decided to include it anyways. So it has to do with CJDNS.

Sjors Provoost That's right.

Aaron van Wirdum Okay, explain.

Sjors Provoost I have no idea what it is.

Aaron van Wirdum Yeah.

Sjors Provoost Apparently it's something like Tor and I2P, like a privacy network, using a different mechanism. The main issue, and Bitcoin Core added support for that years ago, and apparently very few people use it, and it does add complexity to the code base. So the call to action to Bitcoin Core users is, if you like this feature, speak out, because otherwise we might disappear it.

Aaron van Wirdum Right, okay. So it's another potential privacy layer for Bitcoin Core that you can use. It is available now, but because it's so little use, there's now an open question like, should we even keep this?

Sjors Provoost Yeah, so if you have like three peers on the network, it's not...

Aaron van Wirdum And so what if one person says, I love cjdns. Is it then going to stay or what's the... How is this decision going to be made?

Sjors Provoost My guess is then that person would have to fix all the bugs related to it. And if they do that, then it might stay.

Aaron van Wirdum All right.

Sjors Provoost I haven't looked into details, but that's typically how it goes.

Aaron van Wirdum So it's part...

Sjors Provoost If you want a feature to stay, then you should take some responsibility. If you want a feature to stay and you're not willing to do the work, then it will definitely go away. If you are willing to do the work, then it might not go away.

Aaron van Wirdum Right. Yeah.

Sjors Provoost This is also how Windows survives and the wallet survives and the GUI survives and all these things that there's always people that want to get rid of it because it's extra stuff But then there's people say no don't take it away from me and I'll maintain it and then that's how it stays in Then

Aaron van Wirdum Yeah, I told you this in the preparation, but I think I randomly know this guy from years ago. I was at a conference and I just slept at his place. He offered me... I had no idea what CJDNS was. I had no idea who the guy was. But it's like now 10 years later, I come across this. I know that guy. I think his name is Caleb. Maybe I should reach out to him and ask about this.

Sjors Provoost you should maybe talk to him. Yeah, exactly.

Aaron van Wirdum Should I? Is that something you would want?

Sjors Provoost It sounds like...

Aaron van Wirdum I think I know the guy.

Sjors Provoost He should pitch it.

Aaron van Wirdum I think I have his number randomly again.

Sjors Provoost Yeah.

Aaron van Wirdum Okay, let's move on to the next topic.

Sjors Provoost Okay. Oh,

Aaron van Wirdum Increase... Jeez, my handwriting and the light also does help. Increase inbound activity.

Sjors Provoost you're going slightly different.

Aaron van Wirdum Is that the next one?

Sjors Provoost You're not doing lib event or is that later on your list?

Aaron van Wirdum What's that?

Sjors Provoost There should be one about libevent still, HTTP.

Aaron van Wirdum Am I skipping one?

Sjors Provoost Yes, there's another.

Aaron van Wirdum Sorry, what am I skipping?

Sjors Provoost Removing libevent in favor of HTTP ourselves.

Aaron van Wirdum That's the next one on my list.

Sjors Provoost Well, it's in the theme of removing things. Okay.

Aaron van Wirdum Okay, yeah, go. Okay, sure. We can skip.

Sjors Provoost So we have a bunch of dependencies in Bitcoin Core. One of them is called libevent. It's a very old library. It helps us... to HTTP stuff. You might ask yourself, what is Bitcoin Core doing that has anything to do with HTTP? Well, we have something called the RPC, which is the way your node talks to any kind of other software, typically.

Aaron van Wirdum Yeah. Which could be wallets, which could be applications, which could be...

Sjors Provoost Which could be external wallets, can be your lightning node, probably.

Aaron van Wirdum Yeah, just if you want your node to talk with anything else and that's what you would use

Sjors Provoost Anything else on your computer?

Aaron van Wirdum Yeah,

Sjors Provoost Yes. So your computer, so your Bitcoin node spins up a little web server that speaks JSON and it gets a command like get blockchain info and it replies, you know, this is the height, this is etc. So it's a very simple web server and that web server is powered by LibEvent. I think LibEvent was also doing some other things. But now we've taken the essence. that we need, the HTTP part. We implemented that and threw away lib event. So fewer dependencies. Now we have to maintain some HTTP server code. But apparently, that trade-off is worth it.

Aaron van Wirdum yeah, explain why is few I mean for those that don't know why is fewer dependencies better? Why are dependencies a problem?

Sjors Provoost We did a whole episode about dependencies in general, with multiple reasons. Dependencies are software maintained by other people, and other people could put backdoors in them, so that's one reason. But also, our software, Bitcoin Core, keeps modernizing in terms of the code, so using newer versions of C++, etc. And if your dependencies are not also modernizing, then you kind of have a problem, because you have to maintain compatibility with some older ways of doing things. That can be a reason, or it can just cause all sorts of bugs that are difficult to debug. So a reason to get rid of dependencies can be security, can be easier maintenance, and just reducing complexity, those kind of arguments. I'm not sure about the exact argument here, but yeah.

Aaron van Wirdum Yes Yeah, so as much as possible we would want Bitcoin Core to be standalone software that doesn't require Right,

Sjors Provoost Yeah, but there is a trade-off because if you implement everything yourself, that is more work, first of all, but also you might not be the best at it, right? You have specialization of labor in the world. Some people are very good at writing HTTP servers and other people are not. And the HTTP server in Bitcoin Core that we have written now is not the best one in the world. In fact, it is the worst one in the world because it is trying to do only what it needs to talk to these other pieces of software. It is not trying to power, I don't know, NSA.gov or some super popular website. So it's not designed to be fast or amazing. It's just designed to be simple.

Aaron van Wirdum okay in any case Right,

Sjors Provoost And to run on a somewhat trusted environment because RPC is usually not something you expose to the internet. You just expose it on your own computer to your own software or your own Docker container or something like that.

Aaron van Wirdum okay in any case so this this dependency is now removed in the new version of

Sjors Provoost Gone.

Aaron van Wirdum Bitcoin it's gone yes all right and so then let's so then I

Sjors Provoost It's gone.

Aaron van Wirdum have the increase inbound activity point

Sjors Provoost Yes. Okay, so Bitcoin Core connects to nodes when you start it up. It tries to connect to about eight peers in general for everything, and then two peers just for blocks, not for mempool stuff.

Aaron van Wirdum mm-hmm yeah so eight peers for

Sjors Provoost And

Aaron van Wirdum blocks and transactions and two for only blocks

Sjors Provoost depending on if you're on a data center or at home and your router allows it, it also listens to peers. We've done some episodes about that. It will listen to at most 125. peers that can connect to you. Again, if they can get through your router and to you, it is up to 125. That number has now gone up to 200. So number go up, even if the market doesn't go up, number go up. There is a catch. These extra nodes are only used for block relay, not for mempool relay.

Aaron van Wirdum the 75 extra block peers are only used for blocks that you will only share blocks

Sjors Provoost Yeah, exactly.

Aaron van Wirdum with them no transactions right and why was this change made or why wasn't it already the case or why now yeah

Sjors Provoost So relaying blocks takes a lot less bandwidth, because most of the time nothing happens, and then suddenly something happens. And these block announcements are the most important things. You really need to know what the next block is. If you miss a few mempool transactions, this is not a big deal, or if you get them a bit later. But the reason why we're doing this now is because later on we want to start connecting to more of these nodes. So remember we have now eight, I believe it's eight regular outbounds and then two ones that are blocks only. We want to increase that to maybe 10 blocks only so we get a better view of the network. But you don't want to do that until the listening side is actually able to because right now a lot of nodes are just full. You connect to them and they just say like, sorry, or they hang up on somebody else.

Aaron van Wirdum it still sounds very out of out of whack almost so you have 10 outgoing connections and 200 incoming now

Sjors Provoost Yeah, but keep in mind that the number of nodes that are listening is much smaller than the number of nodes that are connecting outwards because a lot of nodes are running on home networks that are protected by a router. So there's an asymmetry there. That's why the number has to be asymmetric too.

Aaron van Wirdum right okay and but as a [unclear: boiler] right

Sjors Provoost And there's a big trust difference. So you might say, well, if you just have all these incoming nodes that give you blocks, you're happy. But in general, we trust nodes that we connect to more than nodes that connect to us. If you go to a strange city and some guy approaches you and says, hey, I'm your tour guide, you don't trust that guy. But if you approach somebody randomly and say, where is the New York Stock Exchange? You will trust the answer from that person because you selected them and not the other way around.

Aaron van Wirdum right

Sjors Provoost But they might trick you into thinking you selected them. And that gets to the whole issue of eclipse attacks that we talked about many, many episodes ago.

Aaron van Wirdum Yeah, well, without going into that in detail, if we boil down the benefit here, that is the benefit, right? If you connect to more nodes, there's a smaller chance that all nodes are lying and therefore that you're being censored and somehow that you're not seeing all the new

Sjors Provoost Yes.

Aaron van Wirdum blocks or all the new transactions or something like that.

Sjors Provoost Yes.

Aaron van Wirdum So in that sense, number go up is in fact actually better.

Sjors Provoost Yes.

Aaron van Wirdum Correct?

Sjors Provoost Yeah. But we have to make the listening number go up before we can increase the number of

Aaron van Wirdum That's ultimately the benefit.

Sjors Provoost nodes that we connect to. So that'll be in some later release.

Aaron van Wirdum right right okay yeah we're moving past our topics swiftly which I like the next point I have is oh yeah this one so fetch inputs in parallel this one gets pretty technical but it's a meaningful thing I

Sjors Provoost Yes.

Aaron van Wirdum think yeah

Sjors Provoost So when a new block comes in, we have to check that whole block. And we basically check every transaction in a row or in sequence.

Aaron van Wirdum do the signatures check out

Sjors Provoost And for every transaction, we check the inputs first and then the outputs, roughly, anyway. How do you check an input, is then the question. An input refers to a previous transaction, which was from a previous block. And it has a script pub key, and that is stored in our UTXO set, at least in Bitcoin Core. So there's a file on our disk that is like 10 gigabytes these days, that has all the still unspent outputs, basically, and the scripts.

Aaron van Wirdum It basically says who owns what without names of course, but just what is this

Sjors Provoost Yeah, it just

Aaron van Wirdum current state of balances on Bitcoin.

Sjors Provoost says...

Aaron van Wirdum I'm putting that in quote marks.

Sjors Provoost Yeah, the balance and the conditions that you can spend it with. And those, we have to load them. And right now, or before this change, what we would do is we would look at the first transaction, the first input, we would fetch it, we would check it, then we would load the second, and we would check it, and we would load the third, and we'd check it. And this change basically says we're going to optimistically, we're just going to fetch all of them in a row, and while we're fetching them, we start checking them. So it's like a zipper, basically. And that just makes it much faster to validate a block. I haven't looked at the numbers, but I think it's like tens of percents faster. Because we're being less serial about things. We're doing things more in parallel.

Aaron van Wirdum By doing more things at the same time, it's faster to check if a block is valid,

Sjors Provoost Yeah,

Aaron van Wirdum right?

Sjors Provoost exactly.

Aaron van Wirdum And the main benefit here... I assume if you're syncing a node in particular. So if you're starting out a new Bitcoin node, it can take hours or days depending on your computer.

Sjors Provoost Exactly.

Aaron van Wirdum And this would speed that up by 10%.

Sjors Provoost Exactly.

Aaron van Wirdum Or roughly, you made a 10% estimate.

Sjors Provoost I don't know the number, but yeah, pretty dramatically. The ultimate... parallel software there, I think, is libbitcoin by Eric Voskuil.

Aaron van Wirdum Right.

Sjors Provoost He's always been sort of nagging Bitcoin Core about you guys are not doing things in parallel enough. And one way you can see that is if you have your node sync from the Genesis block to now, and you look at it, and you look at the bandwidth it's using compared to your internet speed. So maybe you have a gigabit fiber connection, and maybe it's only downloading at 50 megabytes per second. So you're like, well, it's definitely not limited by the bandwidth. And you look at your CPUs, and you have 16 cores, and it's only using three at a time, so it's not using all the cores. And you're looking at your hard disk, and it's like a super fast SSD drive, but it's not spinning at its maximum speed either. So it's not using all its powers combined. You'd want one of those three. to be completely full. So if you have a fast internet connection, then, you know, you would want to see

Aaron van Wirdum Ideally.

Sjors Provoost that internet connection just downloading blocks as fast as it can. Or if, you know, your internet is very fast, but your CPU is not the fastest, then you want your CPUs to be 100% with one of those three, and it's not. And changes like this are making things more parallel, so that hopefully we can max out at least one of the three.

Aaron van Wirdum Yeah. I want to point out and emphasize that over the years, this is one of Bitcoin's biggest challenges. You have to... Download the entire blockchain, validate every block, every transaction. It's, you know, it's a hard problem. And I think over the years, the work that Bitcoin Core developers do have made it, have kept that a possibility, even for regular users.

Sjors Provoost Yeah, so

Aaron van Wirdum I think over the years, syncing has not become much slower just because constant optimizations that I'm going to use this opportunity.

Sjors Provoost in 2017 I made a chart, okay?

Aaron van Wirdum Sure. I'm going to use this opportunity as a live show to get the audience involved because I do think this is the type of boring work that Bitcoin Core developers do that keep Bitcoin absolutely usable for everyone. And I think that deserves appreciation and really an applause for Sjors,

Sjors Provoost the [unclear: core devs?],

Aaron van Wirdum for guys like Sjors, for Bitcoin Core developers that are constantly working on improving this type of work.

Sjors Provoost the excellent boring stuff.

Aaron van Wirdum Yeah, the boring stuff that no one else pays attention to, but it's so

Sjors Provoost Yeah,

Aaron van Wirdum vital that it happens.

Sjors Provoost and it's been going on for a long time. So in 2017, I made a chart where what I did is I rented a bunch of these Amazon machines, and I installed the oldest possible Bitcoin Core versions I could find, or that I could still build, like 0.5, 0.6, 0.7, et cetera. And I would just have it sync the whole blockchain. And this was 10 years ago, so the blockchain was tiny relative to now, but it was already considered big. And, you know, the oldest node would not even sync ever, like, it would just not make it. But then eventually it might take, like, a whole day to sync the blockchain, and the fastest nodes could do it in six hours. So even then you could see the improvements. But we kind of thought we'd maxed out on that, but we still haven't. I think we can still sync two or three times faster. But then, you know, at some point it's done. At some point you have to deal with the fact that the blockchain keeps growing. But back then, yeah, every time the blockchain kept getting bigger, but the node kept getting faster, and it was sort of in balance there.

Aaron van Wirdum Yeah,

Sjors Provoost Like a Moore's law for Bitcoin core code.

Aaron van Wirdum right.

Sjors Provoost And then, of course, the critics would say, oh, we've got to increase the block size. It's like, well, we can barely keep up with this thing.

Aaron van Wirdum There was one more thing I want to say about this,

Sjors Provoost So maybe hold off a bit.

Aaron van Wirdum which I'm now forgetting.

Sjors Provoost Okay.

Aaron van Wirdum No, I think I'm blanking now. Anyways, next topic then. Okay, SEND is no longer experimental.

Sjors Provoost This is such a shameless plug of my own pull request.

Aaron van Wirdum Oh no, the thing I want to say is, so yeah, you mentioned that we do still have a

Sjors Provoost Oh,

Aaron van Wirdum lot of room for growth, right? This is still a major, you know, part of the Bitcoin Core codebase, even now that

Sjors Provoost yes.

Aaron van Wirdum can, you know, be improved significantly, you think?

Sjors Provoost Yes, yeah, I think so.

Aaron van Wirdum Yeah. Yeah.

Sjors Provoost And we know because libbitcoin is still faster.

Aaron van Wirdum Right. Yeah.

Sjors Provoost But we'll catch up.

Aaron van Wirdum Eventually. Okay. SEND is no longer experimental.

Sjors Provoost Yeah, this is just a shameless plug for a simple pull request that I wrote.

Aaron van Wirdum Yeah.

Sjors Provoost Many, many years ago, I introduced the send RPC, which was just a more modern way of doing what sendmany and sendtoaddress was doing, and also was sort of a stepping stone towards working with hardware wallets. And I marked it as experimental because we may want to break it, so people that are relying on it should keep an eye on breaking changes. But we haven't been making breaking changes in a long time, so I decided we should not make it experimental, and people agree. So nothing changed, it's just not experimental.

Aaron van Wirdum Right. So that means it's staying,

Sjors Provoost It might be deleted entirely, but it's... yeah.

Aaron van Wirdum not frivolously or what?

Sjors Provoost Things can still be deleted, but typically, when something is no longer experimental, we're a little bit more like, okay, in this release we're deprecating it, in the next release we're deleting it. When something is experimental, things can happen more suddenly. I don't think this is going away because it's useful. Sending coins is something people like to do, even though they prefer to receive them.

Aaron van Wirdum Then the last point of improvements when it comes to Bitcoin Core and then after that we'll discuss some of the bugs that were found which is also interesting. The last point of improvements which surprised me, it surprised me that this was not a thing yet but there's now a watch-only function in Bitcoin Core,

Sjors Provoost Well, an export watch-only.

Aaron van Wirdum it has been added. Yeah, yeah, right.

Sjors Provoost So, Bitcoin Cores had watch-only wallets for a very long time.

Aaron van Wirdum Watch-only wallet is basically a wallet that shows your balance and how many coins you have, but you just can't spend from it. Which also means that if you're running that on your computer and someone hacks into your computer, they can't spend from it either. It's really a window in your wallet without being a wallet, right?

Sjors Provoost So a setup you could imagine is you take a laptop, you install Bitcoin Core on it, You take it to the mountains, you generate your entropy correctly with whatever you use and you leave it in the mountain and the laptop never goes online again. But you want to watch your balance, receive transactions, etc. And for that you want a watch-only wallet. And so the new thing is that in the GUI there is a button that says export watch -only wallet and it produces for you, this watch-only wallet that you put on your USB stick and take home and then use it to watch.

Aaron van Wirdum Yeah.

Sjors Provoost And you could already do this.

Aaron van Wirdum Well, you can also receive coins through it, right?

Sjors Provoost You could export the script. Yeah, yeah, for sure.

Aaron van Wirdum You still have like a list of your addresses and yeah, so you can do that, you just

Sjors Provoost You can generate addresses. Exactly.

Aaron van Wirdum can't spend.

Sjors Provoost Yeah.

Aaron van Wirdum So it's a way to, yeah, like I said, keep track of what you own, have a window and

Sjors Provoost But you can make, for example, a PSBT and then take the PSBT to the mountains, sign it, and then take the PSBT back and send it to the internet.

Aaron van Wirdum then... Right, right, yeah, okay. So yeah, and the exact thing that's new is that you can now easily export it from your GUI. Yeah, cool. Okay, yeah, so these were the main improvements of Bitcoin Core 32 that we decided to highlight. And then there's the whole, it's been a hot topic, right? Recently in Bitcoin in general, the AI revolution is upon us. Everything and everyone is getting hacks left and right. And that has also, apart from Bitcoin Core, I should add, so far,

Sjors Provoost So far.

Aaron van Wirdum well, I mean, not just Bitcoin Core, by the way, but at least Bitcoin Core seems mostly fine. Although there were some bugs, which we'll discuss. But before we get to the actual bugs, there's also a new disclosure policy.

Sjors Provoost Yes.

Aaron van Wirdum So Bitcoin Core developers have... Well, maybe start by explaining what even is a disclosure policy for those that don't know. How would it usually have worked if a bug is found?

Sjors Provoost Yeah, well, you can go back a little bit further. Bitcoin cores often have bugs in the past. that were fixed. And often those fixes were done a little bit covertly so that people would have updated by the time they would learn about it. But then we often just forgot to actually disclose them. So a couple of years ago, we systematized the disclosure process where there's actually people that keep lists and analyze these disclosures at the right time. What is the right time? There was a policy for that, the disclosure policy. If it's a high severity issue, It's disclosed after the release is end of life. So that's typically two years after the release that contains a fix.

Aaron van Wirdum Yeah, so the bug is fixed, but it's only made public really.

Sjors Provoost Years

Aaron van Wirdum Like by the way, that was this pretty bad bad bug in older Bitcoin Core and

Sjors Provoost later.

Aaron van Wirdum this is what it was.

Sjors Provoost Yeah.

Aaron van Wirdum So the software, the patch was out, but the actual explanation was not yet,

Sjors Provoost Exactly.

Aaron van Wirdum right? Yeah, and that's and that's for severe bugs.

Sjors Provoost Yeah. And there's different severity levels. So for lower severity bugs, they might be announced like two weeks after the release itself. Anyway, that was systematized. But this was in the age before LLMs. And the problem that we started seeing is that, and we'll have an example of that soon, is that three people in a row reporting the same vulnerability, all found with different LLMs.

Aaron van Wirdum Yeah, so there was a bug and somehow it got reported multiple

Sjors Provoost So basically, the LLMs are...

Aaron van Wirdum times, which is very unusual.

Sjors Provoost Which is an indication that this bug is way too easy to find because the number of

Aaron van Wirdum Right.

Sjors Provoost people that report it is not all the people that could see it. Another way to judge that is if you know about a bug, somebody reports a bug, you ask a subtle question to your LLM and it finds it. You're like, okay, this is too easy to find. So that is now a new class or a new category where we say

Aaron van Wirdum You could clearly see after the release of some new AI, Kimi I believe.

Sjors Provoost Kimi K3, the sort of Belgian-Korean K-pop band.

Aaron van Wirdum And then all of a sudden there were like several bug reports. So that posed a challenge.

Sjors Provoost Yeah, so there the strategy of disclosing after two years is actually wrong because all the bad guys will know about it and all the users won't know about it.

Aaron van Wirdum So what I just explained that the bug was fixed but it was not explained yet. That doesn't really work anymore because now if there's some sort of fix someone it will just... people will pay attention or it's just easy to see something changed let's throw my LLM at it and then all of a sudden everyone finds it so sort of keeping it a secret

Sjors Provoost Yeah, this comes back to...

Aaron van Wirdum doesn't really do anything anymore is that yeah

Sjors Provoost Roughly. So one of the things is that when we fix something, usually the fixes are just done on master and then every six months we make a new major release like we just discussed version 32. And so you can just, you know, if the bug is unknown, then it's fine to wait several months and have the major release. And everybody gradually upgrades in the coming year. And then two years later, you announce it. That timeline works. But if you want people to upgrade faster, you're going to have to backport it, which means that we ship a release like 31.2 or 30.3. But those backports are smaller changes. So when you see a .release, yeah, there's only a few commits you need to look at as an attacker to realize where the possible exploit was. There was already a problem, but with LLMs, it's even worse. So anyway, the trade-off now is, for some bugs, it's better to just immediately announce it, have a fix, and encourage people to deploy the fix.

Aaron van Wirdum but so and that's only so the the qualification for that is

Sjors Provoost Easily discoverable.

Aaron van Wirdum Yeah, so there's basically a new category. There's a new category where Bitcoin Core developers will say, okay, this bug is easily discoverable by an LLM, by AI in particular, and therefore there's no point keeping it secret. We're just immediately going to tell you what it is and everyone basically just now

Sjors Provoost It

Aaron van Wirdum has to upgrade, right? That's what it comes down to. And this new policy really was installed because there were easily discoverable bugs found.

Sjors Provoost seems to me rather coincident that a week after announcing this new policy, there were three such bugs.

Aaron van Wirdum yeah yeah

Sjors Provoost So we could go into those.

Aaron van Wirdum yeah so the first bug okay let's uh so the first bug was undo undo file overflow under extreme conditions undo file overflow under extreme conditions

Sjors Provoost Yes. And before I explain the actual bug, this bug has a timeline, and the timeline

Aaron van Wirdum so what yeah

Sjors Provoost basically says in August, sorry, in July 8th this year, Project Loupe reported it. Project Loupe is a project by Spiral, I believe, that scans a bunch of these open source projects. They've been doing that for a while. Then L0rinc, one of the contributors... reported the same issue on August 21, independently. And then Andrew Dorman on October 23, disclosed the same issue. So we clearly have something that three people found independently.

Aaron van Wirdum yeah yeah

Sjors Provoost So this is a very classic example. Now what is the bug has to do with something called the undo file. So we talked about earlier today about how we synchronize a block. We check what's in the block, and then we have to check that every previous input is correct, et cetera. And then once the block is correct, we update our UTXO set. So our database on disk is now changed.

Aaron van Wirdum right

Sjors Provoost The question is, what happens if there's a reorg? How do we go back? Because we don't know what it was before. And the block itself does not tell you what it was before, because the block has inputs that are referring to something that's in the UTXO set. So we don't know what was in the UTXO set because we just changed the UTXO set.

Aaron van Wirdum Yeah, you basically you basically have to apply in case of a reorg in order to

Sjors Provoost We deleted these coins.

Aaron van Wirdum switch to the new chain you have to basically do the last block in reverse but because of the how the UTXO set works that's not actually possible. So then instead Bitcoin Core creates a different file an undo file in case of such a scenario then it knows exactly what to do and how to jump back a little bit in

Sjors Provoost So for every block we have the undo file

Aaron van Wirdum order to rebuild the chain.

Sjors Provoost and they're grouped. So I think it's 128 or so. So for every undo file contains 128 blocks, I believe. And now the problem is that the size of the undo file can be surprisingly large. You'd think they would be the same size or smaller than blocks, but they're actually, they can be much bigger because we're spending inputs.

Aaron van Wirdum Yeah, so even if a block is at most 4 megabytes the undo file for that block can be way bigger.

Sjors Provoost Yes.

Aaron van Wirdum Yeah,

Sjors Provoost And since there's 128 blocks in a single undo file, you might get them too big. A simple example is you have inputs that are 10 kilobytes scriptPubKeys. Then every input is 10 kilobytes. And you have like 1,000 inputs per transaction times. Anyway, it adds up. And it could add up to over 2 gigabytes or over 4 megabytes. Sorry, 4 gigabytes.

Aaron van Wirdum the undo file can be over 4 gigabytes potentially.

Sjors Provoost Yes. And then you say, why is that a problem? Well, because in the undo file, we track where things are. And we track that with a 32-bit number. And a 32-bit number only goes up to 4 gigabytes. And then it circles around. So what you end up doing is... You're saying, oh, this information is in the undo file at position 5 billion, but that's actually position 1 billion, and now you've either overwritten it or you're pointing to the wrong place. And that's not a problem immediately. At that point, your disk is corrupt, right? You've overwritten part of your undo information. But that's fine, as long as there's no reorg. So the attack would be somebody creating these blocks. that fill up your undo file, and then they cause a reorg, your node tries to unwind, it hits this data that doesn't exist, and it just crashes.

Aaron van Wirdum Yeah, so just in order to successfully pull off this attack

Sjors Provoost And then it...

Aaron van Wirdum you first need to feed a node essentially with transactions that... make the undo file that big right which is not organic use it would have to be

Sjors Provoost Yes.

Aaron van Wirdum someone that's actually trying to achieve that for for yeah maybe like a hundred

Sjors Provoost No, you'd have to fill out quite a few blocks.

Aaron van Wirdum blocks time you just fill it with like specific types of transactions that stuff

Sjors Provoost Yes, and not every node has the same undo files, because it depends

Aaron van Wirdum the undo file right that yeah

Sjors Provoost on some coincidences on historically how your node grew. So it's not easy to get an attack that would hit every node. And then you have to create this one block that spends them all, and then you have to reorg either that block or a similar block. And if any of that fails, you just have to start from scratch. So it's not the most practical attack.

Aaron van Wirdum yeah Yeah, but just to run this one more time, if I'm an attacker and I want to do this, I first have to stuff 100 blocks full with these transactions and then I basically myself have to cause a reorg, so mine a block myself that's in conflict with another block so that miners or maybe even two blocks in order to get longer and only

Sjors Provoost Yeah, I guess...

Aaron van Wirdum then the attack will be triggered and then what happens? So what happens to the nodes?

Sjors Provoost At that point, your node tries to reorg, but it can't because the undo file is corrupt.

Aaron van Wirdum Right.

Sjors Provoost And so it just crashes.

Aaron van Wirdum It crashes or it stays on the old chain?

Sjors Provoost No, I think it crashes.

Aaron van Wirdum It crashes, okay.

Sjors Provoost And by extension, you know, it's never...

Aaron van Wirdum So what's

Sjors Provoost Yeah, and then by extension, I guess you'd have to somehow bring it back and ignore

Aaron van Wirdum that?

Sjors Provoost the new block or something, but it'd be annoying.

Aaron van Wirdum Right, right. Okay, so that's a bug. It's unlikely to be exploited and that's because it's also classified as medium or

Sjors Provoost Yeah, it was classified as low severity, but easily discoverable.

Aaron van Wirdum something, right? Low severity even, yeah, okay. Right. But and this bug was now fixed, yes?

Sjors Provoost Yes, and this block was probably not worth doing a new release for. But there were three bugs at the same time, so they were kind of all fixed.

Aaron van Wirdum Right.

Sjors Provoost The other one,

Aaron van Wirdum and it was immediately as pointed out was immediately disclosed for the reasons because it was easy to find yeah

Sjors Provoost immediately once we had the new policy, because remember, it's been reported three

Aaron van Wirdum right right right yeah it does not cover that bug

Sjors Provoost times in three months. Yep.

Aaron van Wirdum should we get into how it was fixed or does it get very like technical and it come

Sjors Provoost It gets fixed by using a 64-bit number, but that still leaves Windows

Aaron van Wirdum okay right

Sjors Provoost vulnerable. And FAT32 system. So if you have an ancient FAT32 file system, that's a 32-bit file system. So files cannot be bigger than, I think, 2 gigabytes. And so once your undo file is bigger than 2 gigabytes, your node just crashes. But then the reasoning is nobody runs an important node on Windows. And so the attack is just really not worth doing. And it will be fixed, hopefully,

Aaron van Wirdum so it takes a ton of effort and then even then the only node you'll affect yeah

Sjors Provoost to crash a bunch of Windows nodes. Yeah.

Aaron van Wirdum okay then there's another bug was this higher severity

Sjors Provoost Yes, the second one we're discovering is medium severity.

Aaron van Wirdum Can you just read what it's called? Because I can't read my own.

Sjors Provoost You can't read your handwriting. The bug is called unbounded memory growth through TCP urgent data on non-Linux systems. Many words.

Aaron van Wirdum Yes, too many for me. That's too complicated already. Okay, what is it?

Sjors Provoost Okay, so the first words are unbounded memory growth.

Aaron van Wirdum What's this bug?

Sjors Provoost That is bad. So we're talking about RAM here. If you start using up more RAM than you have, if a program tries to use more RAM than you have, the program crashes. And if the node crashes, then you can do all sorts of shenanigans, like, you know, plunder the lightning or whatever, if they don't detect your... Because you just keep crashing it, and they never get a chance to see the new blocks, or their lightning node just doesn't react. Things like that. So we don't want that. So that's the first three words. Then something called TCP urgent data. We should talk about that, I guess. So TCP is one of the lower layers of the internet, just the messages between different IP addresses. The Bitcoin P2P protocol kind of lives on top of that. So when you ask for a block, it's like a four megabyte message. That message is chunked into little bits, and that's what TCP does that. And then there is a... At this TCP protocol, there is something called an urgent flag or some exception protocol. That's like a meta protocol. I think it was introduced in the very early days of the Internet. I think it was something like if you're downloading a big file and you decide halfway you don't want to do it, you set this flag and that's how you tell the other side to stop. And so you're distinguishing this from the content because the other node is just seeing bytes. But there's a special byte that says stop what you're doing and then you continue. That has no purpose in Bitcoin. And so... We didn't really deal with it specifically. But apparently it allowed an attack to happen. And then we get to why that happens. We get to the sort of lower level traffic control of the Bitcoin node. Let's say our nodes are connected. I am sending you stuff. You are sending me stuff. And if you are sending me stuff, I get these bytes and I have to park them somewhere and then process them. For example, I get four megabytes of bytes from you, and then I see the first, maybe first megabyte of that is a block. So I take that block, I do something with the block, and then I look at the next bytes you've been sending me. But that parking lot takes up memory, and if that parking lot gets bigger than my own memory, then I crash. So there's a rate limiting in Bitcoin Core that says, if there's too much stuff on the parking lot, I will not accept new bytes from you. I will just simply ignore them. that TCP protocol makes sure that you're not sending bytes. That was a good safety mechanism. But unfortunately, when somebody sets this urgent byte, this urgent flag in the protocol level, we would accidentally also start reading bytes, even though the traffic light was on red. So it's kind of a way to bypass the red traffic light.

Aaron van Wirdum Right. Yeah.

Sjors Provoost And so a very easy way to crash nodes.

Aaron van Wirdum Yeah. And so as I understand it, this is really a bug in kind of the really lower level code of Bitcoin Core.

Sjors Provoost Yeah, this is like 15 year old code that two people understand and they're all

Aaron van Wirdum That's the type two

Sjors Provoost not that active Yeah, no two Bitcoin Core developers plus the attackers Probably

Aaron van Wirdum Bitcoin Core developers or just two people? Okay.

Sjors Provoost a few more but not a lot It's one of those areas of the code that you you know kind of don't want to look at unless unless you have to and we have to

Aaron van Wirdum Yeah. Yeah. It's kind of, it's almost not, well it's kind of not Bitcoin related. It's just deeper in there and that's where the bug...

Sjors Provoost Well, yeah, this is one of those cases where we could have used a library, but we didn't use a library. We wrote our own networking code, but then you are responsible for doing it correctly, and it's a hard problem.

Aaron van Wirdum Right. Maybe you said this, but if the attack is exploited, what's the worst thing? It brings down your node.

Sjors Provoost Yeah, it crashes your node.

Aaron van Wirdum Yeah, it crashes your node. You did mention that. Right. And this is also, I think, a very typical AI, LLM type of thing to find, right? Because you guys, as mentioned, it's kind of not really what you're working on or thinking about even.

Sjors Provoost Yeah, so the AI looks at the [unclear: agent/ancient] code and is like, oh, I think

Aaron van Wirdum Right.

Sjors Provoost this could be crashed in this and this way because it remembers the details of the TCP protocol or all of that stuff. Yeah.

Aaron van Wirdum Okay, and so this was fixed. Again, is it worth going into what the fix is or do you even know?

Sjors Provoost The fix boils down to... ignoring this emergency signal.

Aaron van Wirdum Right.

Sjors Provoost So there's like three, kind of whenever the node is working, it's doing three things in a row. It's sending what it needs to send, it's receiving what it needs to receive, and it's checking this urgent flag. And we just ignore the urgent flag. So then when you set it, nothing happens, roughly. But then on Windows, there's a deep rabbit hole.

Aaron van Wirdum Yeah.

Sjors Provoost But for now, we've fixed it. But yeah, it's kind of annoying.

Aaron van Wirdum Yeah.

Sjors Provoost And I think the only reason why this is...

Aaron van Wirdum And it's kind of the type of code you don't like touching because it's...

Sjors Provoost Yeah.

Aaron van Wirdum Yeah.

Sjors Provoost So the good news is there's only a problem on Mac OS and Windows and not on Linux. And most of the critical infrastructure runs on Linux. Because otherwise I think this would have been a high severity one. Because if you can crash every node in the Bitcoin network, that's pretty bad.

Aaron van Wirdum Do you know why Linux is unaffected?

Sjors Provoost Because there are two, I would say, network traffic control systems. One is called Select and the other is called Poll. And on Linux we use one and on the other systems we use the other.

Aaron van Wirdum Right.

Sjors Provoost And I think that one of them is more modern, and we kind of want to switch over to the modern version everywhere, but that's a whole separate big refactor that somebody needs to do, which means somebody needs to be motivated to look at this horrible low-level network code.

Aaron van Wirdum Yeah. You did mention three bugs, but we only wrote down two or at least I only wrote...

Sjors Provoost Yeah, there was a third one that was a bit similar to this one, but we're not going to go into that.

Aaron van Wirdum We're skipping the third one. Was that high, low, that was low severity?

Sjors Provoost That was low. I believe it was low.

Aaron van Wirdum Right, but still easy to find, yeah?

Sjors Provoost It was also a way to make somebody run out of memory, but it would take much longer.

Aaron van Wirdum Right, okay. Awesome, how long have we been recording Sjors?

Sjors Provoost About 48 minutes. So I think in terms of the normal episode we're good. But unless you have any other questions.

Aaron van Wirdum That's roughly what an episode is, so...

Sjors Provoost Then I say thank you for listening to Bitcoin.

Aaron van Wirdum Explained. Thanks for being here. Now that we're here, do we... We're not recording anymore, but... You already did a Q&A session, right?

Sjors Provoost I'll leave it on but. Yes.

Aaron van Wirdum Could be fun. Does anyone have a question for Sjors? This is like a good opportunity, maybe?

Sjors Provoost Or for Aaron.

Aaron van Wirdum Yeah.

Audience member You guys were talking about the watch-only wallet. What is the difference with just a regular xpub

Sjors Provoost Yes. The wallet comes with all the stuff. So it comes with your transaction, history, the labels you've added, the descriptors. An xpub is just one thing. In general, we prefer descriptors over xpub because descriptors tell you what to do with the xpub. But yeah, in principle, you could already do these things by just ferrying an XPUB around. Now you just ferry all the extra metadata.

Audience member Yeah, because there are some things you could do with a watch-only wallet that you mentioned, like creating transactions and seeing your history, but you can do this with like [unclear] wallet and xpub. You can do the same thing.

Sjors Provoost You can because there are conventions on what you can do with an XPUB. But these conventions are also foot guns. Because if you take an XPUB that was supposed to be used for the legacy one blah blah blah address, and then you decide to use it to create a SegWit address, Then you import your mnemonic, and your mnemonic is actually looking in the wrong place, and you think your coins are gone. They're not actually gone, but they're like, you will spend a bunch of time figuring out where they are. So descriptors are there. The descriptors is just a bunch of text around the XPUB saying, this is a legacy address. This is a taproot address. This is a derivation from, if you [unclear: start at the seed?], do this derivation, do this derivation, do this derivation. Then it says, this is the XPUB. And then everything else. So that's what descriptors do. But you could already export descriptors, too, in read-only format. And then manually import them again. But this is quite tedious. So now you just copy the whole wallet file, just that one file or directory.

Off-mic speaker Sounds like a great feature.

Sjors Provoost I think so, too.

Aaron van Wirdum Alright, should we wrap it or we can do like one more question if anyone has one? But I don't think so. Okay, then let's just wrap it.

Sjors Provoost Cool. Thank you.

Aaron van Wirdum Sjors, thanks. Everyone, thanks for being here.