Episode 51: Compact Blocks

Aaron van Wirdum Live from Utrecht, this is Bitcoin Explained.

Sjors Provoost Hello.

Aaron van Wirdum Hey Sjors, welcome back.

Sjors Provoost Thank you.

Aaron van Wirdum Happy New Year.

Sjors Provoost Not yet.

Aaron van Wirdum I don't think our editors will be fast enough to have this ready before the end of this year. So let's pretend it's a new year for all our listeners.

Sjors Provoost Oh my god, oh my god, it's 2022. The world has not actually ended with a meteorite strike.

Aaron van Wirdum In case anyone noticed, we did take a short break. Sjors and I, we were both traveling for a while.

Sjors Provoost Seeking warmer weather.

Aaron van Wirdum Yeah, and now we're picking it up again. And we've got a banger episode today. I was surprised we didn't cover this one yet. We're going to discuss compact blocks.

Sjors Provoost Yes.

Aaron van Wirdum Which apparently we hadn't done yet. So, shall we just get into it, Sjors?

Sjors Provoost Sounds good.

Aaron van Wirdum All right, so compact blocks. First of all, what problem are we solving here?

Sjors Provoost Well, I guess what we're solving is that, as we know, blocks are created every 10 minutes.

Aaron van Wirdum Maybe I should briefly mention this is not a new thing. This has been in Bitcoin for a while. It was implemented in Bitcoin Core 0.13. Yeah, which is like four years ago I guess.

Sjors Provoost I think it's 2016 or something. Yeah, it's pretty long ago.

Aaron van Wirdum Oh, even more than four years. Yeah, it's been a while, uh, so it's not a new thing, but it is something that's today I guess pretty fundamental to how Bitcoin works and specifically the Bitcoin peer-to-peer network so the way nodes connect to other nodes.

Sjors Provoost Yep.

Aaron van Wirdum Okay, so there's a problem that needed to be solved five years ago what was the problem?

Sjors Provoost Well, so as we know, there's, you know, one block on average every 10 minutes or so. And sometimes there's two blocks at the same time. And that's kind of a waste because one of the blocks will not be built upon. So, and that's called sort of the orphan rate or the stale block rate. And, well, we'd like to prevent that. And the reason you want to prevent that is that miners will waste a lot of money.

Aaron van Wirdum Sjors, are we talking about the same thing?

Sjors Provoost I don't know.

Aaron van Wirdum The redundancy is this. I'm going to explain what the redundancy is, which I think is the correct answer, but maybe there's some truth to your thing as well. The redundancy is that nodes are constantly sending transactions over the peer-to-peer network.

Sjors Provoost Oh, I thought, okay, I was going at a much higher level than that.

Aaron van Wirdum Yeah, yeah. Start at the beginning. So nodes are sending transactions to each other on the peer-to-peer network, and they're all collecting their transactions in their mempool. So they all have all the transactions, or at least most of the transactions. And then once every 10 minutes, a new block is found. Now, originally, this block was then sent over the Bitcoin network, which meant that all of the transactions in that block were also sent as part of that block over the network again. That was redundant because Bitcoin nodes really already had these transactions in their mempool. So they were basically receiving the same transactions twice. And there's two problems with this. There's kind of the same problem from different perspectives. One problem is that it requires more inbound bandwidth than strictly necessary. So it's a bit of a waste in that sense. And then the other problem is actually a bit bigger. That's the outbound bandwidth that you need. Well, first of all, it's a waste again that you need more. Wait, is that even a waste? Yeah, that's the same type of waste.

Sjors Provoost It's a bit of a waste.

Aaron van Wirdum But then the bigger problem is that the size of the blocks with all of the transactions in it, they can be a bit slow to transmit all this redundant data that you're transmitting kind of slow it's slower than necessary if you could only send the bare essentials so essentially not the transactions for a second time you could only send the bare essentials then you're sending a compact block which is also the name of this protocol and by sending by making this more efficient it's you make the network more healthy and specifically, that's good for mining centralization, or I should say it's good for mining decentralization.

Sjors Provoost Exactly, which is why I brought up the orphan rate, because one indication of... Basically, when you're seeing multiple blocks produced at the same time, usually by different mining, that generally means it's different mining pools that are not up-to-date about each other's block. And one reason they're not up-to-date about each other's block is because it's so slow to move the blocks from one side of the network to the other.

Aaron van Wirdum Right, now I understand why you started there. Yeah, so... Yeah, you're right. If you see two blocks, it's usually because different parts of the network haven't seen the newest block yet. Some part of the network, some miners haven't seen the newest block, so they're still mining on an old block, and then they find a new block as well, and now you have competing blocks, and therefore you get an orphan. Yeah. All right, go on.

Sjors Provoost And so miners can deal with that in two ways. Well, one way miners can deal with that problem that is not what we're going to discuss is they would all join the same pool or they would build a bunch of private infrastructure to relay blocks to each other.

Aaron van Wirdum Which they do.

Sjors Provoost That is the centralization pressure, basically.

Aaron van Wirdum Yeah. Well, let's zoom into that a little bit. So miners can and do construct... private networks essentially they connect with each other and they send blocks over that network so it's fast as possible they do that today right yeah yeah, so why is that exactly a problem? Why is that a centralization issue?

Sjors Provoost It's a problem if you know you're another miner and you're not in that clique because then what could happen is say one third or two thirds of all the miners is in one such a network and you're not then your blocks are going to be the ones that get staled all the time because they're all together and building on each other's blocks.

Aaron van Wirdum Yeah, ideally the Bitcoin network... Yeah, ideally the Bitcoin network is completely open and permissionless, including for new miners. And by having a group of miners connect to each other through private channels, that's essentially a form of centralization. And in a way, you kind of have gatekeepers. I mean, they're kind of soft gatekeepers.

Sjors Provoost It could create centralization if the competitive advantage of that is big enough. But if the rest of the network relays it fast enough, then such private networks are less beneficial. And I guess you can try to estimate how big of a problem this is. So 10 minutes is 600 seconds. So if, say, you have to wait, I don't know, one minute to get the block as a miner, well, that's 10% of your revenue, basically, in a way.

Aaron van Wirdum Yeah, but even if it's six seconds, that's 1%, which is, you know, mining is a marginal business. They're all competing on very strict levels I guess.

Sjors Provoost Yeah, I guess it depends on the times right? Right now I guess because there's a chip shortage and and other problems with supplies and warehouses there's it's probably quite profitable to mine but at some point it's designed to be extremely competitive where every penny matters so you know getting the the propagation to below I don't know, a second or so is is very nice.

Aaron van Wirdum Anyway, so yeah the point being that this private network, these private networks that do exist these relay networks, they offer a benefit to the in-group of miners. And to counter that, we'd ideally make the peer-to-peer network as fast as possible. That's the goal here. Or at least that's other than the bandwidth waste, which we mentioned, which is kind of the minor issue. The bigger goal is to make the peer-to-peer network as fast as possible and compact blocks help with that.

Sjors Provoost So there was a small practical issue. People run nodes partially altruistically, right? And if your network breaks down every time you get a block because it's trying to pump one megabyte through the small little home router, maybe you get annoyed by your node and you turn it off. So there is a smaller incentive like that. But I think the bigger issue is the miner centralization risk.

Aaron van Wirdum Yeah, what you're referring to is the fact that every time you get a block and you transmit it, you transmit it immediately to like eight peers or whatever and that sort of bandwidth bump could actually slow down your video streaming or whatever you were doing. So that was also kind of a minor problem but something worth noting. But yeah, the biggest benefit is trying to speed up the peer-to-peer network. And that's… So that's how we get to compact blocks.

Sjors Provoost One other point is that the total bandwidth, like the total amount of bandwidth for this redundancy is not too bad. I think it's only 10% or so. 10% of the bandwidth of a typical node is used for relaying blocks, even in this inefficient scenario. So we're talking maybe about a 10% reduction in bandwidth, but it's more about the other things we talked about. Okay.

Aaron van Wirdum Yes. So compact blocks. Did I mention we're discussing compact blocks?

Sjors Provoost I don't know, but that's what we're doing.

Aaron van Wirdum I think I've now mentioned it 40 times. So compact blocks. What are compact blocks?

Sjors Provoost Well, it's kind of what the word suggests. So typically a block starts with the header and then has the coinbase transaction and then it has all the other transactions. And that's what takes up a lot of space. But the little trick that we use now is that... We know that the other side probably already has all the transactions in the block because they have it in the mempool. So instead, you just send the header of the block and the coinbase transaction of the block. And maybe if you want to, a couple of very recent transactions that you think the other side might not have. And that's the entire message. So that message might be 30 kilobytes instead of one or two megabytes.

Aaron van Wirdum Wait, wait, wait. Your peer still needs to know which transactions out of his mempool actually went into the block, right?

Sjors Provoost Yeah, sorry. So actually the compact block has the header and then it has a list of, sort of a compressed list of transactions. Each transaction is represented by a short ID. Maybe, I forgot how long it is, but much shorter than a real transaction. And then it's followed by some full transactions that you think the other side doesn't have. In particular, that's the coinbase transaction because you create that coinbase transaction. So by definition, nobody else will know about that.

Aaron van Wirdum Yeah, so let's zoom into these short IDs because I think that's sort of the most interesting or important part of compact blocks.

Sjors Provoost Yeah.

Aaron van Wirdum So it doesn't include all the transactions. Instead, it includes short IDs per transaction. Now, these short IDs are not the same thing as transaction IDs. There are these things called transaction IDs, which I think are just hashes of the transaction content, right?

Sjors Provoost Yes.

Aaron van Wirdum But that's not what these are. These transaction IDs using compact blocks are even shorter than that, right?

Sjors Provoost That's right.

Aaron van Wirdum So they're not a full hash.

Sjors Provoost Yeah, so normally a transaction hash is 256 bytes— bits, so 64 bytes, the SHA-256 hash of the contents of the transaction. These ones are much shorter, I think a third of that. And the downside of that is that it's possible for an evil person to...

Aaron van Wirdum We'll get into that in a second.

Sjors Provoost Yep.

Aaron van Wirdum Actually, just finish your sentence.

Sjors Provoost So the downside of that is that an evil person could create these short IDs, like could create the same short ID for multiple transactions, which is impossible with a SHA-256 hash. So that's the downside.

Aaron van Wirdum That could be a collision. Yeah, that's what I wanted to get at next. First, just to reiterate... So the way it works is you get this complex compact sorry, compact block with all these short transaction IDs. I receive that as a peer. I use the short transaction IDs to pick the matching transactions from my mempool. And out of that, I can reconstruct the block myself. And that's how I know the block, right?

Sjors Provoost That's right.

Aaron van Wirdum Yes. Right. And now you were getting into one of the problems of these super short version transaction IDs which is that there is it's technically possible that there are collisions so several valid transactions can return the same identical short ID so different transactions and they're both valid have the same ID. So this is a problem if you have two of these transactions in your mempool.

Sjors Provoost Exactly.

Aaron van Wirdum And you get one of these IDs in a compact block, then you don't know which one to pick.

Sjors Provoost Right, so when you receive a compact block you go into your mempool and you calculate the short ID of everything in your mempool, and then you you you basically compare that to what's in the short block, and if there's two matches, then you have a problem. However, the short IDs are picked in such a way that this is quite unlikely to happen. I think it's once every 250,000 blocks or so.

Aaron van Wirdum How many blocks are there now? Like 600,000 something?

Sjors Provoost Yes.

Aaron van Wirdum 700,000?

Sjors Provoost So in other words, it's very unlikely.

Aaron van Wirdum It would have happened like twice or three times in all of Bitcoin's history.

Sjors Provoost Well, it'll happen somewhere all the time. But for any individual node, it's very unlikely. Right? So there are lots of nodes. So it might, if it only happens for one node, it might only happen twice.

Aaron van Wirdum Would that differ per node? Why?

Sjors Provoost Well, no, for every node, it might only happen twice, but it's going to be different for each node.

Aaron van Wirdum Why?

Sjors Provoost Yes. So that's maybe interesting to go into. So the way you hash the transaction into a short ID is unique depending on the block. So the hash of the block is included in this transaction hash, and it's unique per peer, or it might be unique per peer. So in other words, if you're sending me this compact block, you're going to be using different hashes than when somebody else is sending me those blocks.

Aaron van Wirdum Is there any benefit to that? Why would it be that way?

Sjors Provoost So the idea there is mainly so that if there is such a collision, then only some nodes will experience this collision rather than all nodes. So that means that the compact block will just go around the node that has a problem with it.

Aaron van Wirdum It doesn't slow down the entire network.

Sjors Provoost Exactly. Because it's uncorrelated.

Aaron van Wirdum Right, got it. That's pretty smart.

Sjors Provoost Yeah. It also makes the attack more difficult.

Aaron van Wirdum Yeah, you were mentioning a specific attack. I just mentioned the problem. So can you repeat what this is?

Sjors Provoost So mainly we talked about accidental collisions, but you probably don't want somebody to send you a bunch of gibberish that you don't have any matches for. They can spam you. So there's a couple of protections in this protocol that I don't pretend to understand completely, but that will make it more difficult to abuse the system. And I guess the biggest one is that you're still sending the header of a block. So that means you're basically, when you receive this compact block message, it starts with the header of the actual block, and that header has proof of work in it. So if somebody wants to bother you, they'll actually have to find a block with sufficient proof of work that you don't know about yet. And that's going to be pretty rare. Does that make sense?

Aaron van Wirdum It does.

Sjors Provoost Because... Usually when you receive the message, you see the header first. If you already have the block, then you'd stop processing because you don't care. If you don't have the block, then the header contains something else, namely the previous block. So every Bitcoin block header will tell you, this is the block that I'm building on top of. So if you don't have that block either, then you also ignore the message. So whatever is coming to you has to be based on a new block that you haven't heard about. And even if there is something wrong with that message, I guess some other peer will send the real one. And then you start ignoring the fake message. So there's a very short window of opportunity to bother you in any kind of way. Because you might be connected to multiple peers that are all sending you this compact block message. So if one of them is lying or sending some garbage, you're processing that and wasting a little bit of resources. But a fraction of a second later, you get the same block from an honest peer. And now you've already got it, so you ignore the fake messages.

Aaron van Wirdum Right.

Sjors Provoost And it's more difficult because the hashes depend on the hash of the block, et cetera, et cetera.

Aaron van Wirdum All right. So as mentioned, this was included in Bitcoin Core a while ago, four or five years ago. Does that mean it's now done or is this still something that's being improved or what's the status?

Sjors Provoost If you look at the Bitcoin Core repository, there are a couple of smaller changes that go in every now and then, even as recent as this year to fix mistakes in how to handle it or to make it a little bit more robust against attacks. I think something that was introduced a few years ago was if you're still syncing the node for the very first time from the Genesis block, then somebody might spam you with those compact blocks that are very old or fake compact blocks that are very old because the proof of work was very low in the beginning. And so there's little checks like that that make sure like, okay, I'm going to ignore these messages until I'm almost up to date.

Aaron van Wirdum Right. But the general protocol's there and works and Bitcoin Core nodes benefit from it, right?

Sjors Provoost Yeah, pretty much. There's nothing you need to do.

Aaron van Wirdum All right. Well, [unclear]…

Sjors Provoost And I'm sure there's still improvements that could be made. But for that, you'd have to dive a lot deeper into it.

Aaron van Wirdum All right, so does that conclude our episode on compact blocks?

Sjors Provoost I guess it does, the short one.

Aaron van Wirdum Yeah, I will mention, a quick shout out to Matt Corallo, I guess, who sort of originally designed and built this.

Sjors Provoost Yeah, I think along with Greg Maxwell.

Aaron van Wirdum Greg Maxwell, he's involved with everything.

Sjors Provoost Yes.

Aaron van Wirdum And I think that's it then, Sjors.

Sjors Provoost Okay, well, thank you for listening to Bitcoin Explained.