WEBVTT - Automatic transcript; recognition may contain errors. Reviewed for technical wording and speaker labels. Times match the published episode.

00:00:00.660 --> 00:00:04.460
<v Aaron van Wirdum>Live from Utrecht, this is Bitcoin Explained.

00:00:04.720 --> 00:00:05.020
<v Sjors Provoost>Hello.

00:00:05.820 --> 00:00:06.920
<v Aaron van Wirdum>Hey Sjors, welcome back.

00:00:07.200 --> 00:00:07.620
<v Sjors Provoost>Thank you.

00:00:07.840 --> 00:00:08.440
<v Aaron van Wirdum>Happy New Year.

00:00:09.040 --> 00:00:09.420
<v Sjors Provoost>Not yet.

00:00:10.230 --> 00:00:15.440
<v Aaron van Wirdum>I don't think our editors will be fast enough to have this ready before the end

00:00:15.440 --> 00:00:18.520
<v Aaron van Wirdum>of this year. So let's pretend it's a new year for all our listeners.

00:00:18.980 --> 00:00:23.100
<v Sjors Provoost>Oh my god, oh my god, it's 2022. The world has not actually ended with a

00:00:23.100 --> 00:00:24.480
<v Sjors Provoost>meteorite strike.

00:00:24.940 --> 00:00:29.220
<v Aaron van Wirdum>In case anyone noticed, we did take a short break. Sjors and I, we were both

00:00:29.220 --> 00:00:30.340
<v Aaron van Wirdum>traveling for a while.

00:00:31.740 --> 00:00:32.860
<v Sjors Provoost>Seeking warmer weather.

00:00:33.260 --> 00:00:39.140
<v Aaron van Wirdum>Yeah, and now we're picking it up again. And we've got a banger episode today. I

00:00:39.140 --> 00:00:43.590
<v Aaron van Wirdum>was surprised we didn't cover this one yet. We're going to discuss compact

00:00:43.590 --> 00:00:44.130
<v Aaron van Wirdum>blocks.

00:00:44.470 --> 00:00:44.910
<v Sjors Provoost>Yes.

00:00:45.270 --> 00:00:50.500
<v Aaron van Wirdum>Which apparently we hadn't done yet. So, shall we just get into it, Sjors?

00:00:50.640 --> 00:00:51.020
<v Sjors Provoost>Sounds good.

00:00:51.320 --> 00:00:56.990
<v Aaron van Wirdum>All right, so compact blocks. First of all, what problem are we solving here?

00:00:57.950 --> 00:01:03.390
<v Sjors Provoost>Well, I guess what we're solving is that, as we know, blocks are created every 10 minutes.

00:01:03.850 --> 00:01:08.310
<v Aaron van Wirdum>Maybe I should briefly mention this is not a new thing. This has been in Bitcoin

00:01:08.310 --> 00:01:13.890
<v Aaron van Wirdum>for a while. It was implemented in Bitcoin Core 0.13.

00:01:14.540 --> 00:01:17.160
<v Aaron van Wirdum>Yeah, which is like four years ago I guess.

00:01:17.160 --> 00:01:20.460
<v Sjors Provoost>I think it's 2016 or something. Yeah, it's pretty long ago.

00:01:20.460 --> 00:01:24.540
<v Aaron van Wirdum>Oh, even more than four years. Yeah, it's been a while, uh, so it's not a new

00:01:24.540 --> 00:01:29.240
<v Aaron van Wirdum>thing, but it is something that's today I guess pretty fundamental to how Bitcoin

00:01:29.240 --> 00:01:34.280
<v Aaron van Wirdum>works and specifically the Bitcoin peer-to-peer network so the way nodes connect

00:01:34.280 --> 00:01:34.900
<v Aaron van Wirdum>to other nodes.

00:01:34.900 --> 00:01:35.300
<v Sjors Provoost>Yep.

00:01:35.300 --> 00:01:40.900
<v Aaron van Wirdum>Okay, so there's a problem that needed to be solved five years ago what was the problem?

00:01:41.370 --> 00:01:45.750
<v Sjors Provoost>Well, so as we know, there's, you know, one block on average every 10 minutes or

00:01:45.750 --> 00:01:50.770
<v Sjors Provoost>so. And sometimes there's two blocks at the same time. And that's kind of a waste

00:01:50.770 --> 00:01:55.320
<v Sjors Provoost>because one of the blocks will not be built upon. So, and that's called sort of

00:01:55.320 --> 00:02:01.280
<v Sjors Provoost>the orphan rate or the stale block rate. And, well, we'd like to prevent that.

00:02:01.400 --> 00:02:05.540
<v Sjors Provoost>And the reason you want to prevent that is that miners will waste a lot of money.

00:02:07.110 --> 00:02:08.670
<v Aaron van Wirdum>Sjors, are we talking about the same thing?

00:02:08.870 --> 00:02:09.450
<v Sjors Provoost>I don't know.

00:02:09.730 --> 00:02:13.370
<v Aaron van Wirdum>The redundancy is this. I'm going to explain what the redundancy is, which I

00:02:13.370 --> 00:02:17.050
<v Aaron van Wirdum>think is the correct answer, but maybe there's some truth to your thing as well.

00:02:18.060 --> 00:02:22.740
<v Aaron van Wirdum>The redundancy is that nodes are constantly sending transactions over the

00:02:22.740 --> 00:02:23.500
<v Aaron van Wirdum>peer-to-peer network.

00:02:23.740 --> 00:02:26.220
<v Sjors Provoost>Oh, I thought, okay, I was going at a much higher level than that.

00:02:26.380 --> 00:02:30.400
<v Aaron van Wirdum>Yeah, yeah. Start at the beginning. So nodes are sending transactions to each

00:02:30.400 --> 00:02:33.240
<v Aaron van Wirdum>other on the peer-to-peer network, and they're all collecting their transactions

00:02:33.240 --> 00:02:37.140
<v Aaron van Wirdum>in their mempool. So they all have all the transactions, or at least most of the

00:02:37.140 --> 00:02:42.860
<v Aaron van Wirdum>transactions. And then once every 10 minutes, a new block is found.

00:02:43.380 --> 00:02:48.660
<v Aaron van Wirdum>Now, originally, this block was then sent over the Bitcoin network, which meant

00:02:48.660 --> 00:02:54.320
<v Aaron van Wirdum>that all of the transactions in that block were also sent as part of that block

00:02:54.320 --> 00:03:00.020
<v Aaron van Wirdum>over the network again. That was redundant because Bitcoin nodes really

00:03:00.020 --> 00:03:03.580
<v Aaron van Wirdum>already had these transactions in their mempool. So they were basically receiving

00:03:03.580 --> 00:03:08.680
<v Aaron van Wirdum>the same transactions twice. And there's two problems with this.

00:03:09.220 --> 00:03:13.560
<v Aaron van Wirdum>There's kind of the same problem from different perspectives. One problem is that

00:03:13.560 --> 00:03:19.380
<v Aaron van Wirdum>it requires more inbound bandwidth than

00:03:19.380 --> 00:03:23.770
<v Aaron van Wirdum>strictly necessary. So it's a bit of a waste in that sense. And then the other

00:03:23.770 --> 00:03:29.470
<v Aaron van Wirdum>problem is actually a bit bigger. That's the outbound bandwidth that you need.

00:03:29.610 --> 00:03:33.350
<v Aaron van Wirdum>Well, first of all, it's a waste again that you need more. Wait, is that even a

00:03:33.350 --> 00:03:35.770
<v Aaron van Wirdum>waste? Yeah, that's the same type of waste.

00:03:35.810 --> 00:03:36.450
<v Sjors Provoost>It's a bit of a waste.

00:03:36.550 --> 00:03:42.410
<v Aaron van Wirdum>But then the bigger problem is that the size of the blocks with all of the

00:03:42.410 --> 00:03:43.370
<v Aaron van Wirdum>transactions in it,

00:03:44.910 --> 00:03:50.750
<v Aaron van Wirdum>they can be a bit slow to transmit all this redundant data that you're

00:03:50.750 --> 00:03:55.310
<v Aaron van Wirdum>transmitting kind of slow it's slower than necessary if you could only send the

00:03:55.310 --> 00:03:59.690
<v Aaron van Wirdum>bare essentials so essentially not the transactions for a second time you could

00:03:59.690 --> 00:04:04.110
<v Aaron van Wirdum>only send the bare essentials then you're sending a compact block which is also

00:04:04.110 --> 00:04:09.550
<v Aaron van Wirdum>the name of this protocol and by sending by making this more efficient it's

00:04:09.550 --> 00:04:12.550
<v Aaron van Wirdum>you make the network more healthy and specifically,

00:04:13.840 --> 00:04:18.040
<v Aaron van Wirdum>that's good for mining centralization, or I should say it's good for mining

00:04:18.040 --> 00:04:18.860
<v Aaron van Wirdum>decentralization.

00:04:18.920 --> 00:04:24.120
<v Sjors Provoost>Exactly, which is why I brought up the orphan rate, because one indication of...

00:04:24.120 --> 00:04:27.680
<v Sjors Provoost>Basically, when you're seeing multiple blocks produced at the same time, usually

00:04:27.680 --> 00:04:31.420
<v Sjors Provoost>by different mining, that generally means it's different mining pools that are

00:04:31.420 --> 00:04:34.820
<v Sjors Provoost>not up-to-date about each other's block. And one reason they're not up-to-date

00:04:34.820 --> 00:04:38.100
<v Sjors Provoost>about each other's block is because it's so slow to move the blocks from one side

00:04:38.100 --> 00:04:38.940
<v Sjors Provoost>of the network to the other.

00:04:39.160 --> 00:04:45.040
<v Aaron van Wirdum>Right, now I understand why you started there. Yeah, so... Yeah, you're right. If

00:04:45.040 --> 00:04:49.280
<v Aaron van Wirdum>you see two blocks, it's usually because different parts of the network haven't

00:04:49.280 --> 00:04:53.060
<v Aaron van Wirdum>seen the newest block yet. Some part of the network, some miners haven't seen the

00:04:53.060 --> 00:04:56.680
<v Aaron van Wirdum>newest block, so they're still mining on an old block, and then they find a new

00:04:56.680 --> 00:04:59.160
<v Aaron van Wirdum>block as well, and now you have competing blocks, and therefore you get an

00:04:59.160 --> 00:05:00.700
<v Aaron van Wirdum>orphan. Yeah. All right, go on.

00:05:00.860 --> 00:05:05.280
<v Sjors Provoost>And so miners can deal with that in two ways. Well, one way miners can deal with

00:05:05.280 --> 00:05:10.420
<v Sjors Provoost>that problem that is not what we're going to discuss is they would all join the

00:05:10.420 --> 00:05:13.840
<v Sjors Provoost>same pool or they would build a bunch of private infrastructure to relay blocks

00:05:13.840 --> 00:05:14.320
<v Sjors Provoost>to each other.

00:05:14.740 --> 00:05:15.520
<v Aaron van Wirdum>Which they do.

00:05:15.520 --> 00:05:17.460
<v Sjors Provoost>That is the centralization pressure, basically.

00:05:17.660 --> 00:05:17.800
<v Aaron van Wirdum>Yeah.

00:05:19.840 --> 00:05:24.060
<v Aaron van Wirdum>Well, let's zoom into that a little bit. So miners can and do

00:05:24.060 --> 00:05:29.060
<v Aaron van Wirdum>construct... private networks essentially they connect with each other and they

00:05:29.060 --> 00:05:32.960
<v Aaron van Wirdum>send blocks over that network so it's fast as possible they do that today right

00:05:32.960 --> 00:05:37.060
<v Aaron van Wirdum>yeah yeah, so why is that exactly a problem? Why is that a centralization issue?

00:05:37.060 --> 00:05:41.620
<v Sjors Provoost>It's a problem if you know you're another miner and you're not in that clique

00:05:41.620 --> 00:05:46.200
<v Sjors Provoost>because then what could happen is say one third or two thirds of all the miners

00:05:46.200 --> 00:05:51.710
<v Sjors Provoost>is in one such a network and you're not then your blocks are going to be the ones

00:05:51.710 --> 00:05:55.810
<v Sjors Provoost>that get staled all the time because they're all together and building on each

00:05:55.810 --> 00:05:56.350
<v Sjors Provoost>other's blocks.

00:05:57.350 --> 00:06:01.390
<v Aaron van Wirdum>Yeah, ideally the Bitcoin network... Yeah, ideally the Bitcoin network is

00:06:01.390 --> 00:06:07.290
<v Aaron van Wirdum>completely open and permissionless, including for new miners. And by having a

00:06:07.290 --> 00:06:10.530
<v Aaron van Wirdum>group of miners connect to each other through private channels, that's

00:06:10.530 --> 00:06:14.770
<v Aaron van Wirdum>essentially a form of centralization. And in a way, you kind of have gatekeepers.

00:06:14.990 --> 00:06:16.590
<v Aaron van Wirdum>I mean, they're kind of soft gatekeepers.

00:06:16.770 --> 00:06:20.690
<v Sjors Provoost>It could create centralization if the competitive advantage of that is big

00:06:20.690 --> 00:06:24.290
<v Sjors Provoost>enough. But if the rest of the network relays it fast enough, then such private

00:06:24.290 --> 00:06:29.310
<v Sjors Provoost>networks are less beneficial. And I guess you can try to estimate how big of a

00:06:29.310 --> 00:06:35.020
<v Sjors Provoost>problem this is. So 10 minutes is 600 seconds. So if, say, you have to wait,

00:06:35.080 --> 00:06:40.840
<v Sjors Provoost>I don't know, one minute to get the block as a miner, well,

00:06:40.940 --> 00:06:45.600
<v Sjors Provoost>that's 10% of your revenue, basically, in a way.

00:06:45.700 --> 00:06:51.060
<v Aaron van Wirdum>Yeah, but even if it's six seconds, that's 1%, which is, you know, mining is a

00:06:51.060 --> 00:06:56.130
<v Aaron van Wirdum>marginal business. They're all competing on very strict levels I guess.

00:06:56.130 --> 00:06:59.770
<v Sjors Provoost>Yeah, I guess it depends on the times right? Right now I guess because there's a

00:06:59.770 --> 00:07:04.470
<v Sjors Provoost>chip shortage and and other problems with supplies and warehouses there's it's

00:07:04.470 --> 00:07:08.750
<v Sjors Provoost>probably quite profitable to mine but at some point it's designed to be extremely

00:07:08.750 --> 00:07:13.670
<v Sjors Provoost>competitive where every penny matters so you know getting the the propagation to

00:07:13.670 --> 00:07:16.550
<v Sjors Provoost>below I don't know, a second or so is is very nice.

00:07:16.550 --> 00:07:20.910
<v Aaron van Wirdum>Anyway, so yeah the point being that this private network, these private networks

00:07:20.910 --> 00:07:25.790
<v Aaron van Wirdum>that do exist these relay networks, they offer a benefit to the in-group of

00:07:25.790 --> 00:07:30.810
<v Aaron van Wirdum>miners. And to counter that, we'd ideally make the peer-to-peer network as fast

00:07:30.810 --> 00:07:36.410
<v Aaron van Wirdum>as possible. That's the goal here. Or at least that's other than the bandwidth

00:07:36.410 --> 00:07:41.270
<v Aaron van Wirdum>waste, which we mentioned, which is kind of the minor issue. The bigger goal is

00:07:41.270 --> 00:07:44.420
<v Aaron van Wirdum>to make the peer-to-peer network as fast as possible and compact blocks

00:07:44.420 --> 00:07:46.120
<v Aaron van Wirdum>help with that.

00:07:45.810 --> 00:07:51.070
<v Sjors Provoost>So there was a small practical issue. People run nodes partially altruistically,

00:07:51.750 --> 00:07:55.570
<v Sjors Provoost>right? And if your network breaks down every time you get a block because it's

00:07:55.570 --> 00:08:00.530
<v Sjors Provoost>trying to pump one megabyte through the small little home router, maybe you get

00:08:00.530 --> 00:08:05.510
<v Sjors Provoost>annoyed by your node and you turn it off. So there is a smaller incentive like

00:08:05.510 --> 00:08:07.950
<v Sjors Provoost>that. But I think the bigger issue is the miner centralization risk.

00:08:08.250 --> 00:08:12.530
<v Aaron van Wirdum>Yeah, what you're referring to is the fact that every time you get a block and

00:08:12.530 --> 00:08:17.580
<v Aaron van Wirdum>you transmit it, you transmit it immediately to like eight peers or whatever and

00:08:17.580 --> 00:08:23.400
<v Aaron van Wirdum>that sort of bandwidth bump could actually slow down your video streaming or

00:08:23.400 --> 00:08:28.240
<v Aaron van Wirdum>whatever you were doing. So that was also kind of a minor problem but something

00:08:28.240 --> 00:08:32.980
<v Aaron van Wirdum>worth noting. But yeah, the biggest benefit is trying to speed up the

00:08:32.980 --> 00:08:38.480
<v Aaron van Wirdum>peer-to-peer network. And that's… So that's how we get to compact blocks.

00:08:38.800 --> 00:08:42.580
<v Sjors Provoost>One other point is that the total bandwidth, like the total amount of bandwidth

00:08:42.580 --> 00:08:47.940
<v Sjors Provoost>for this redundancy is not too bad. I think it's only 10% or so. 10% of the

00:08:47.940 --> 00:08:51.360
<v Sjors Provoost>bandwidth of a typical node is used for relaying blocks, even in this inefficient

00:08:51.360 --> 00:08:55.480
<v Sjors Provoost>scenario. So we're talking maybe about a 10% reduction in bandwidth, but it's

00:08:55.480 --> 00:08:57.460
<v Sjors Provoost>more about the other things we talked about. Okay.

00:08:58.040 --> 00:09:02.890
<v Aaron van Wirdum>Yes. So compact blocks. Did I mention we're discussing compact blocks?

00:09:03.030 --> 00:09:04.950
<v Sjors Provoost>I don't know, but that's what we're doing.

00:09:05.390 --> 00:09:10.330
<v Aaron van Wirdum>I think I've now mentioned it 40 times. So compact blocks. What are compact blocks?

00:09:10.830 --> 00:09:15.630
<v Sjors Provoost>Well, it's kind of what the word suggests. So typically a block starts with the

00:09:15.630 --> 00:09:18.810
<v Sjors Provoost>header and then has the coinbase transaction and then it has all the other

00:09:18.810 --> 00:09:24.630
<v Sjors Provoost>transactions. And that's what takes up a lot of space. But the little trick that

00:09:24.630 --> 00:09:29.110
<v Sjors Provoost>we use now is that... We know that the other side probably already has all the

00:09:29.110 --> 00:09:32.430
<v Sjors Provoost>transactions in the block because they have it in the mempool. So instead, you

00:09:32.430 --> 00:09:37.110
<v Sjors Provoost>just send the header of the block and the coinbase transaction of the block. And

00:09:37.110 --> 00:09:40.450
<v Sjors Provoost>maybe if you want to, a couple of very recent transactions that you think the

00:09:40.450 --> 00:09:44.650
<v Sjors Provoost>other side might not have. And that's the entire message. So that message might

00:09:44.650 --> 00:09:48.770
<v Sjors Provoost>be 30 kilobytes instead of one or two megabytes.

00:09:49.360 --> 00:09:54.200
<v Aaron van Wirdum>Wait, wait, wait. Your peer still needs to know which transactions out of his

00:09:54.200 --> 00:09:56.520
<v Aaron van Wirdum>mempool actually went into the block, right?

00:09:56.800 --> 00:10:02.780
<v Sjors Provoost>Yeah, sorry. So actually the compact block has the header and then it has a

00:10:02.810 --> 00:10:06.610
<v Sjors Provoost>list of, sort of a compressed list of transactions. Each transaction is

00:10:06.610 --> 00:10:12.130
<v Sjors Provoost>represented by a short ID. Maybe, I forgot how long it is, but much shorter than

00:10:12.130 --> 00:10:16.860
<v Sjors Provoost>a real transaction. And then it's followed by some full transactions that you

00:10:16.860 --> 00:10:20.060
<v Sjors Provoost>think the other side doesn't have. In particular, that's the coinbase transaction

00:10:20.060 --> 00:10:24.800
<v Sjors Provoost>because you create that coinbase transaction. So by definition, nobody else will

00:10:24.800 --> 00:10:25.380
<v Sjors Provoost>know about that.

00:10:25.790 --> 00:10:30.250
<v Aaron van Wirdum>Yeah, so let's zoom into these short IDs because I think that's sort of the most

00:10:30.250 --> 00:10:33.150
<v Aaron van Wirdum>interesting or important part of compact blocks.

00:10:33.410 --> 00:10:33.550
<v Sjors Provoost>Yeah.

00:10:33.770 --> 00:10:39.170
<v Aaron van Wirdum>So it doesn't include all the transactions. Instead, it includes short IDs per

00:10:39.170 --> 00:10:43.790
<v Aaron van Wirdum>transaction. Now, these short IDs are not the same thing as transaction IDs.

00:10:43.950 --> 00:10:47.650
<v Aaron van Wirdum>There are these things called transaction IDs, which I think are just hashes of

00:10:47.650 --> 00:10:48.890
<v Aaron van Wirdum>the transaction content, right?

00:10:49.230 --> 00:10:49.470
<v Sjors Provoost>Yes.

00:10:50.760 --> 00:10:55.180
<v Aaron van Wirdum>But that's not what these are. These transaction IDs using compact blocks are

00:10:55.180 --> 00:10:56.760
<v Aaron van Wirdum>even shorter than that, right?

00:10:57.000 --> 00:10:57.440
<v Sjors Provoost>That's right.

00:10:57.620 --> 00:10:58.860
<v Aaron van Wirdum>So they're not a full hash.

00:11:00.330 --> 00:11:06.090
<v Sjors Provoost>Yeah, so normally a transaction hash is 256 bytes— bits, so 64

00:11:06.090 --> 00:11:10.930
<v Sjors Provoost>bytes, the SHA-256 hash of the contents of the transaction.

00:11:11.970 --> 00:11:17.460
<v Sjors Provoost>These ones are much shorter, I think a third of that. And the downside of that is

00:11:17.460 --> 00:11:20.480
<v Sjors Provoost>that it's possible for an evil person to...

00:11:20.480 --> 00:11:21.780
<v Aaron van Wirdum>We'll get into that in a second.

00:11:22.180 --> 00:11:22.340
<v Sjors Provoost>Yep.

00:11:22.900 --> 00:11:24.100
<v Aaron van Wirdum>Actually, just finish your sentence.

00:11:24.240 --> 00:11:29.900
<v Sjors Provoost>So the downside of that is that an evil person could create these short IDs, like

00:11:29.900 --> 00:11:33.720
<v Sjors Provoost>could create the same short ID for multiple transactions, which is impossible

00:11:33.720 --> 00:11:38.090
<v Sjors Provoost>with a SHA-256 hash. So that's the downside.

00:11:38.550 --> 00:11:42.150
<v Aaron van Wirdum>That could be a collision. Yeah, that's what I wanted to get at next. First, just

00:11:42.150 --> 00:11:47.780
<v Aaron van Wirdum>to reiterate... So the way it works is you get this complex compact

00:11:47.780 --> 00:11:52.780
<v Aaron van Wirdum>sorry, compact block with all these short transaction IDs. I receive that as a

00:11:52.780 --> 00:11:58.740
<v Aaron van Wirdum>peer. I use the short transaction IDs to pick the matching transactions from

00:11:58.740 --> 00:12:04.720
<v Aaron van Wirdum>my mempool. And out of that, I can reconstruct the block myself. And that's

00:12:04.720 --> 00:12:05.660
<v Aaron van Wirdum>how I know the block, right?

00:12:05.800 --> 00:12:06.420
<v Sjors Provoost>That's right.

00:12:06.720 --> 00:12:10.460
<v Aaron van Wirdum>Yes. Right. And now you were getting into one of the problems of these super

00:12:10.460 --> 00:12:16.100
<v Aaron van Wirdum>short version transaction IDs which is that there is it's technically possible

00:12:16.100 --> 00:12:22.060
<v Aaron van Wirdum>that there are collisions so several valid transactions can

00:12:22.060 --> 00:12:27.830
<v Aaron van Wirdum>return the same identical short ID so different

00:12:27.830 --> 00:12:32.030
<v Aaron van Wirdum>transactions and they're both valid have the same ID. So this is a problem if you

00:12:32.030 --> 00:12:34.370
<v Aaron van Wirdum>have two of these transactions in your mempool.

00:12:34.370 --> 00:12:34.830
<v Sjors Provoost>Exactly.

00:12:34.830 --> 00:12:38.830
<v Aaron van Wirdum>And you get one of these IDs in a compact block, then you don't know which one to pick.

00:12:38.830 --> 00:12:42.410
<v Sjors Provoost>Right, so when you receive a compact block you go into your mempool and you

00:12:42.410 --> 00:12:47.450
<v Sjors Provoost>calculate the short ID of everything in your mempool, and then you you you

00:12:47.450 --> 00:12:49.770
<v Sjors Provoost>basically compare that to what's in the short block, and if there's two matches,

00:12:50.540 --> 00:12:55.040
<v Sjors Provoost>then you have a problem. However, the short IDs are picked in such a way that

00:12:55.040 --> 00:13:00.360
<v Sjors Provoost>this is quite unlikely to happen. I think it's once every 250,000 blocks or so.

00:13:01.040 --> 00:13:04.300
<v Aaron van Wirdum>How many blocks are there now? Like 600,000 something?

00:13:04.560 --> 00:13:04.700
<v Sjors Provoost>Yes.

00:13:04.700 --> 00:13:05.320
<v Aaron van Wirdum>700,000?

00:13:05.460 --> 00:13:07.180
<v Sjors Provoost>So in other words, it's very unlikely.

00:13:07.180 --> 00:13:10.840
<v Aaron van Wirdum>It would have happened like twice or three times in all of Bitcoin's history.

00:13:10.900 --> 00:13:15.420
<v Sjors Provoost>Well, it'll happen somewhere all the time. But for any individual node, it's very

00:13:15.420 --> 00:13:21.360
<v Sjors Provoost>unlikely. Right? So there are lots of nodes. So it might, if it only happens for

00:13:21.360 --> 00:13:22.900
<v Sjors Provoost>one node, it might only happen twice.

00:13:24.430 --> 00:13:25.970
<v Aaron van Wirdum>Would that differ per node? Why?

00:13:27.120 --> 00:13:29.800
<v Sjors Provoost>Well, no, for every node, it might only happen twice, but it's going to be

00:13:29.800 --> 00:13:30.760
<v Sjors Provoost>different for each node.

00:13:30.900 --> 00:13:31.060
<v Aaron van Wirdum>Why?

00:13:31.240 --> 00:13:35.660
<v Sjors Provoost>Yes. So that's maybe interesting to go into. So the way you hash the transaction

00:13:35.660 --> 00:13:41.530
<v Sjors Provoost>into a short ID is unique depending on the block. So the hash of the block

00:13:41.530 --> 00:13:46.190
<v Sjors Provoost>is included in this transaction hash, and it's unique per peer, or it might be

00:13:46.190 --> 00:13:46.870
<v Sjors Provoost>unique per peer.

00:13:48.250 --> 00:13:52.950
<v Sjors Provoost>So in other words, if you're sending me this compact block, you're going to be

00:13:52.950 --> 00:13:58.150
<v Sjors Provoost>using different hashes than when somebody else is sending me those blocks.

00:13:58.310 --> 00:14:00.670
<v Aaron van Wirdum>Is there any benefit to that? Why would it be that way?

00:14:00.770 --> 00:14:05.750
<v Sjors Provoost>So the idea there is mainly so that if there is such a collision,

00:14:06.700 --> 00:14:10.860
<v Sjors Provoost>then only some nodes will experience this collision rather than all nodes. So

00:14:10.860 --> 00:14:15.060
<v Sjors Provoost>that means that the compact block will just go around the node that has a problem with it.

00:14:15.060 --> 00:14:17.120
<v Aaron van Wirdum>It doesn't slow down the entire network.

00:14:17.480 --> 00:14:18.980
<v Sjors Provoost>Exactly. Because it's uncorrelated.

00:14:19.380 --> 00:14:20.960
<v Aaron van Wirdum>Right, got it. That's pretty smart.

00:14:21.160 --> 00:14:21.360
<v Sjors Provoost>Yeah.

00:14:22.670 --> 00:14:24.470
<v Sjors Provoost>It also makes the attack more difficult.

00:14:25.970 --> 00:14:31.710
<v Aaron van Wirdum>Yeah, you were mentioning a specific attack. I just mentioned the problem.

00:14:32.070 --> 00:14:33.410
<v Aaron van Wirdum>So can you repeat what this is?

00:14:33.710 --> 00:14:39.130
<v Sjors Provoost>So mainly we talked about accidental collisions, but you probably don't want

00:14:39.130 --> 00:14:43.810
<v Sjors Provoost>somebody to send you a bunch of gibberish that you don't have any matches for.

00:14:43.910 --> 00:14:47.610
<v Sjors Provoost>They can spam you. So there's a couple of protections in this protocol that I

00:14:47.610 --> 00:14:50.770
<v Sjors Provoost>don't pretend to understand completely, but that will make it more difficult to

00:14:50.770 --> 00:14:54.850
<v Sjors Provoost>abuse the system. And I guess the biggest one is that you're still sending the

00:14:54.850 --> 00:14:59.770
<v Sjors Provoost>header of a block. So that means you're

00:14:59.770 --> 00:15:04.190
<v Sjors Provoost>basically, when you receive this compact block message, it starts with the header

00:15:04.190 --> 00:15:09.480
<v Sjors Provoost>of the actual block, and that header has proof of work in it. So if somebody

00:15:09.480 --> 00:15:12.400
<v Sjors Provoost>wants to bother you, they'll actually have to find a block with sufficient proof

00:15:12.400 --> 00:15:16.680
<v Sjors Provoost>of work that you don't know about yet. And that's going to be pretty rare.

00:15:18.000 --> 00:15:18.860
<v Sjors Provoost>Does that make sense?

00:15:19.420 --> 00:15:20.180
<v Aaron van Wirdum>It does.

00:15:21.260 --> 00:15:21.700
<v Sjors Provoost>Because...

00:15:23.430 --> 00:15:26.870
<v Sjors Provoost>Usually when you receive the message, you see the header first. If you already

00:15:26.870 --> 00:15:31.830
<v Sjors Provoost>have the block, then you'd stop processing because you don't care. If you don't

00:15:31.830 --> 00:15:35.050
<v Sjors Provoost>have the block, then the header contains something else, namely the previous

00:15:35.050 --> 00:15:39.470
<v Sjors Provoost>block. So every Bitcoin block header will tell you, this is the block that I'm

00:15:39.470 --> 00:15:44.160
<v Sjors Provoost>building on top of. So if you don't have that block either, then you also ignore

00:15:44.160 --> 00:15:49.830
<v Sjors Provoost>the message. So whatever is coming to you has to be based on a new block that you

00:15:49.830 --> 00:15:53.200
<v Sjors Provoost>haven't heard about. And even if there is something wrong with that message, I

00:15:53.200 --> 00:15:57.660
<v Sjors Provoost>guess some other peer will send the real one. And then you start ignoring the

00:15:57.660 --> 00:15:58.140
<v Sjors Provoost>fake message.

00:15:59.990 --> 00:16:04.050
<v Sjors Provoost>So there's a very short window of opportunity to bother you in any kind of way.

00:16:05.960 --> 00:16:08.860
<v Sjors Provoost>Because you might be connected to multiple peers that are all sending you this

00:16:08.860 --> 00:16:12.720
<v Sjors Provoost>compact block message. So if one of them is lying or sending some garbage, you're

00:16:12.720 --> 00:16:16.680
<v Sjors Provoost>processing that and wasting a little bit of resources. But a fraction of a second

00:16:16.680 --> 00:16:21.210
<v Sjors Provoost>later, you get the same block from an honest peer. And now you've already got it,

00:16:21.330 --> 00:16:23.930
<v Sjors Provoost>so you ignore the fake messages.

00:16:24.270 --> 00:16:24.670
<v Aaron van Wirdum>Right.

00:16:24.850 --> 00:16:28.210
<v Sjors Provoost>And it's more difficult because the hashes depend on the hash of the block, et

00:16:28.210 --> 00:16:28.670
<v Sjors Provoost>cetera, et cetera.

00:16:29.210 --> 00:16:34.590
<v Aaron van Wirdum>All right. So as mentioned, this was included in Bitcoin Core a while ago, four

00:16:34.590 --> 00:16:39.810
<v Aaron van Wirdum>or five years ago. Does that mean it's now done or is this still something that's

00:16:39.810 --> 00:16:41.790
<v Aaron van Wirdum>being improved or what's the status?

00:16:42.070 --> 00:16:47.070
<v Sjors Provoost>If you look at the Bitcoin Core repository, there are a couple of smaller changes

00:16:47.070 --> 00:16:52.410
<v Sjors Provoost>that go in every now and then, even as recent as this year to fix mistakes in how

00:16:52.410 --> 00:16:56.720
<v Sjors Provoost>to handle it or to make it a little bit more robust against attacks. I think

00:16:56.720 --> 00:17:02.480
<v Sjors Provoost>something that was introduced a few years ago was if you're still syncing the

00:17:02.480 --> 00:17:07.220
<v Sjors Provoost>node for the very first time from the Genesis block, then somebody might spam you

00:17:07.220 --> 00:17:11.920
<v Sjors Provoost>with those compact blocks that are very old or fake compact blocks that are very

00:17:11.920 --> 00:17:13.820
<v Sjors Provoost>old because the proof of work was very low in the beginning.

00:17:15.050 --> 00:17:19.130
<v Sjors Provoost>And so there's little checks like that that make sure like, okay, I'm going to

00:17:19.130 --> 00:17:21.590
<v Sjors Provoost>ignore these messages until I'm almost up to date.

00:17:21.960 --> 00:17:27.620
<v Aaron van Wirdum>Right. But the general protocol's there and works and Bitcoin Core nodes benefit

00:17:27.620 --> 00:17:28.620
<v Aaron van Wirdum>from it, right?

00:17:28.980 --> 00:17:30.760
<v Sjors Provoost>Yeah, pretty much. There's nothing you need to do.

00:17:31.120 --> 00:17:32.300
<v Aaron van Wirdum>All right. Well, [unclear]…

00:17:31.620 --> 00:17:37.270
<v Sjors Provoost>And I'm sure there's still improvements that could be made. But for that, you'd

00:17:37.270 --> 00:17:38.890
<v Sjors Provoost>have to dive a lot deeper into it.

00:17:39.230 --> 00:17:43.130
<v Aaron van Wirdum>All right, so does that conclude our episode on compact blocks?

00:17:43.310 --> 00:17:44.750
<v Sjors Provoost>I guess it does, the short one.

00:17:45.050 --> 00:17:49.670
<v Aaron van Wirdum>Yeah, I will mention, a quick shout out to Matt Corallo, I guess, who sort of

00:17:49.670 --> 00:17:51.410
<v Aaron van Wirdum>originally designed and built this.

00:17:51.590 --> 00:17:53.330
<v Sjors Provoost>Yeah, I think along with Greg Maxwell.

00:17:53.330 --> 00:17:55.670
<v Aaron van Wirdum>Greg Maxwell, he's involved with everything.

00:17:56.170 --> 00:17:56.450
<v Sjors Provoost>Yes.

00:17:58.110 --> 00:18:00.960
<v Aaron van Wirdum>And I think that's it then, Sjors.

00:18:01.300 --> 00:18:04.420
<v Sjors Provoost>Okay, well, thank you for listening to Bitcoin Explained.

