Episode 100: Bitcoin Core 31

Aaron van Wirdum Live from Utrecht, this is Bitcoin.

Sjors Provoost Explained.

Aaron van Wirdum Hey, Sjors.

Sjors Provoost Hey, Aaron. It's been a while.

Aaron van Wirdum Yes. Welcome back on our... Hey, this is our 100th episode, Sjors.

Sjors Provoost That is correct. Episode 100.

Aaron van Wirdum I did not think we would make it with our biannual podcast. But we're back.

Sjors Provoost We've exponentially approached it. Exponentially slowing down approached it. But we made it. Well, at least we made it to the start.

Aaron van Wirdum Yeah, we're sticking to our semi-official rule to only record in person, so that's why it took a while. I was not in the Netherlands, but now I'm back in beautiful sunny Utrecht for a beautiful, fresh Bitcoin Explained episode.

Sjors Provoost Exactly, and we're going to pick a very fresh topic for a node that was released only three months ago.

Aaron van Wirdum Yes, we do make an episode. I think we've made an episode for every new Bitcoin Core release since we started the podcast. So we're at least keeping that up. But yeah, so this is Bitcoin Core 31, which came out when?

Sjors Provoost In April 19th.

Aaron van Wirdum Fantastic.

Sjors Provoost And in fact, there is now a 31.1 release with some bug fixes.

Aaron van Wirdum Right, we're not even... Well, I guess these are just bug fixes. So when did this come out?

Sjors Provoost But the bug fix is related to one of the features we're going to cover. So it's good that we had to wait for this bug fix.

Aaron van Wirdum Okay. Well, oh, is CoinKite still sponsoring us, Sjors?

Sjors Provoost I think so.

Aaron van Wirdum Should I?

Sjors Provoost We're optimistically going to assume that they still are.

Aaron van Wirdum Should I endorse or advocate for their projects just in case?

Sjors Provoost We should at least thank them.

Aaron van Wirdum Dear listener, check out CoinKite with your favorite hardware wallet vendor with the COLDCARD, the OPENDIME, the SATSCARD, the TAPSIGNER. What else do they have, Sjors?

Sjors Provoost I have no idea.

Aaron van Wirdum Definitely these things. CoinKite.com. Check it out I assume that's their website.

Sjors Provoost Yes.

Aaron van Wirdum All right. Or just Google CoinKite and see where it takes you.

Sjors Provoost I'm pretty sure they have every conceivable domain.

Aaron van Wirdum I would think so. Okay, Sjors. So Bitcoin Core 31 released a couple of months ago, and you made a special selection for our dear listeners of some of the highlights in the new Bitcoin Core release. Is that right?

Sjors Provoost A highly curated list.

Aaron van Wirdum Yeah. Okay, so the first one you picked, which we'll discuss, is there's been... We wrote down Cluster mempool. Is cluster mempool... First question, is this a completely new thing? So the term cluster mempool, because I feel like I heard it before, is that just because we were discussing that it was coming? So my question is... Has cluster mempool been updated, or is cluster mempool a new thing in Bitcoin Core 31?

Sjors Provoost I believe that 31 is the first version that has cluster mempool in it. The concept has been around for a few years, so it's been mentioned on different podcasts, maybe even ours, but I'm not sure.

Aaron van Wirdum Okay, so this was a long time coming, and now, three months ago, in Bitcoin Core 31, there's now cluster mempool.

Sjors Provoost Yes, and you will probably not notice a difference.

Aaron van Wirdum As a regular user?

Sjors Provoost As a user, yeah.

Aaron van Wirdum Okay, so what is cluster mempool? Just take it from the top.

Sjors Provoost Well, I don't completely understand it, so maybe I'm not the right person to ask.

Aaron van Wirdum That's a great start.

Sjors Provoost But it is just a better or sane...

Aaron van Wirdum That's why people listen to our podcast, to get sort of half-baked, they think this is how it works type of explanations.

Sjors Provoost Well, it depends on the subject, but I mean, the bottom line is that it is a sane version of the mempool. The original mempool code was kind of awkward, difficult to reason about. One of the weirdest aspects of it was that... So the mempool is there to decide, or it should reflect what miners are going to mine basically first. And so ideally, if your mempool is full and you need to kick something out, then you're going to kick the most worthless transaction out and not the most valuable one.

Aaron van Wirdum And by that you mean the transaction that pays the lowest fee?

Sjors Provoost Well, not necessarily, because it's more complicated than that, right? Because it's not just about the fee that your transaction pays, but it's also how much your parents pay or how much your uncles... and nephews and et cetera pay.

Aaron van Wirdum And I guess that's the point.

Sjors Provoost And that was very complicated to reason about. So there were some strange scenarios where if the mempool was full, you would actually kick out the transaction that if you were a miner would be the first one you'd want to mine because of this intricate relationships between transactions.

Aaron van Wirdum Okay, but the simple example of this is child pays for parents, right? So there's a transaction that pays a very low fee, but from that transaction are coins spent again in a new transaction that pays a high fee. So as a miner... You want to mine both of those, including the very low fee transaction, because then you also get the high fee from the next transaction.

Sjors Provoost Yeah, and as far as I know, that worked. But once you make the scenarios more complicated, things do not behave very well.

Aaron van Wirdum Yeah, but so that's the point I'm making. That already worked, right? Bitcoin Core already had at least the basic child pays for parent logic. Also in the mempool, I think, right?

Sjors Provoost Yeah, as far as I, my limited understanding is that it was able to reason basically from the ancestors down to the children or from the children up to the ancestors. Like you can go down the family graph up or down.

Aaron van Wirdum Like one step.

Sjors Provoost But not sideways. And I think now it can take into account like, oh, you know, this parent and child are worthless, but they have a rich cousin that's paying a lot of fees. And so we should include that.

Aaron van Wirdum So they're connected through, I don't know, like the grandfather in this analogy. You could just draw more.

Sjors Provoost Yeah, so you have a parent, a child.

Aaron van Wirdum The term cluster kind of says it. You can take a bigger cluster of transactions into account. when it comes to which transactions you want to keep in the mempool or kick out of the mempool.

Sjors Provoost Yeah and where this becomes important is we have talked about this previously I think in equally vague terms things like pinning attacks where you're trying to escape from a Lightning channel so you broadcast a transaction and then somebody else makes a child of that transaction that makes it so that you will never get confirmed because the total fee is too low and so you want to bump that fee but you can't because you hit some sort of limit and there's a lot of these annoying complicated cases.

Aaron van Wirdum Real quick, I don't know if you know the answer to this question, but is it the case that the more sophisticated miners and mining pools were practically already doing this, but Bitcoin Core was just not keeping up and therefore mempools weren't really reflecting what was actually being mined? Or is this also really an innovation that miners are going to use?

Sjors Provoost There's a couple of different things happening here. As far as people that have looked at what blocks look like, the quirks in the blocks suggest that they use Bitcoin Core to decide what goes in the block. However, they might have tweaked things like the minimum fee rate. So last summer we saw that... suddenly miners started including 0.1 sats per vbyte transactions. And that is a case of the miners are ahead of what Bitcoin Core was doing by default. And then Bitcoin Core tends to follow what the miners are doing because of this rule that we want to basically anticipate what's going to be mined. So I wouldn't say that they use some sort of sophisticated algorithm as far as I know. In fact, it's kind of the opposite. If the mempool is not as sophisticated as it can be, There is an incentive for somebody to say, hey, I can make better block templates than Bitcoin Core does. Why don't you send the transactions to me as sort of an accelerator? And then you get centralization pressure. So Bitcoin Core kind of needs to be as good as possible while not wasting too many resources of the node. Because there's a computational time there, right? Every time a transaction comes in, you need to do some calculation to see if it's good or bad.

Aaron van Wirdum Yeah, as good as possible from the... perspective of a miner that wants to maximize fee revenue, right?

Sjors Provoost Yes.

Aaron van Wirdum Was this just something that was kind of always on Bitcoin Core's to -do list, but it's hard and it's complicated and that's why it took a while? Or is there a more specific reason it was implemented? I think you told me that it has something to do with Ark maybe also.

Sjors Provoost No, it predates it. So the idea of cluster mempool predates Ark, I think by several years.

Aaron van Wirdum Okay.

Sjors Provoost I mean, over the past 10 years or so, I guess since Lightning especially, there's been a lot of discussion about pinning attacks that I just described before. And then you have things like RBF, which has a bunch of different rules. Like it has some rules in the BIP and it has some rules, actual rules that are not in the BIP. And, you know, it got quite convoluted. It wasn't very elegant. It was kind of a hot, sort of a hodgepodge of, well, if we make this rule and this exception, and then, you know, we have a carve out here. It was kind of a messy set of rules that was just good enough, but then people kept finding ways to still pin you as a Lightning operator. So there were stakes. And this was kind of a redesign of it, which people came up with several years ago, thinking of how can we design the mempool logic from scratch. the sorting logic from scratch in a way that gets rid of all these weird exceptions too. So I think a couple of rules have been dropped in how RBFs are processed and exceptions have been removed. So it's more elegant and it seems to be more robust.

Aaron van Wirdum Okay. And this is a very, I mean, we're talking about the mempool as a kind of a very individual thing in the sense of you have your own mempool and, you know, every node has that. But this, I assume automatically, or I mean, you can answer it automatically. This also applies to what you share with your peers.

Sjors Provoost As far as I know, you share with your peers the moment you add it into your own mempool.

Aaron van Wirdum Right, exactly.

Sjors Provoost So that doesn't really change. I mean, if for some reason a transaction now goes into your mempool because of these different heuristics, give a slightly different result, then yeah, you would be sharing it with your peers.

Aaron van Wirdum Yeah, but that was like a whole project, right? That Gloria in particular was working on, I think, with the package relay. So is this related to that? Is this kind of the same project in that sense?

Sjors Provoost So my understanding is that this will make package relay... So what we discussed in a previous episode, the one parent, one child, special case of package relay. So it's a package of two. And this is quite useful because if you need to get out of a Lightning channel, that's usually just two transactions. One is the penalty transaction that you want to apply or the escape transaction. And the other is the fees that you want to add because when you make the channel, you agreed on either very low fees or in the case of ephemeral dust, no fees at all. And so the special case of two was possible even without cluster mempool. But the hope is that with cluster mempool, you can reason about a much larger set of transactions. And then my optimistic take, and could be wrong, is that things like Ark would now become possible. Because with Ark, you have, say, 25 transactions to get out. And don't ask me how Arks work. You might have a package of 25 because you have a tree of transactions that are needed to escape. But right now, the nodes tend to process transactions one at a time. Whereas in such a scenario, maybe all the transactions don't pay enough fees except the very last one, which pays all the fees. And the node needs to understand that it needs to consider all 25 of them and that the last one is paying all the fees. And I think that's possible with package relay. And then hopefully cluster mempool makes it so that the node can reason about these big packages. It doesn't right now. It just has the better architecture under it.

Aaron van Wirdum Sorry, that last sentence caught me off guard. It doesn't have it right now?

Sjors Provoost No. We have cluster mempool, so it's a better architecture that might allow packages larger than two, but we don't do that yet.

Aaron van Wirdum Oh, okay, but that gets back to how do you even get the transactions in the first place and if other pairs don't get the whole cluster to you?

Sjors Provoost Well, okay, so the cluster is a bunch of transactions that might have been given to you sequentially. So it builds up over time.

Aaron van Wirdum Right. But you need to keep them in order to know if the child or the parents are also valid, right? You need to not eject them, which is the point.

Sjors Provoost Yeah. When you start out, you will accept anything that's over one sat per V-byte because your mempool is empty. You've got 300 megabytes.

Aaron van Wirdum Yeah, or 0.1 sat per V-byte, I think.

Sjors Provoost Yes, exactly. So you build up this mempool, including the clusters that are in it of related transactions. And then at some point, your mempool starts overflowing. And so you now have to decide which one are you going to throw out. And that is when you look at these clusters and you find the worst cluster and then each cluster is divided into chunks. So you take any given set of transactions that are related as a cluster and you kind of squash it in a line in a way that the best part of the cluster is at the top and the worst part of the cluster is at the bottom. And so you take the worst part of the cluster and you throw that out. And that's a very fast calculation now.

Aaron van Wirdum Okay, but you throw that out in a way that the rest of the cluster is still valid.

Sjors Provoost Yes, so the way you squash the cluster, like you flatten it, you flatten it in a way that one side of it doesn't need the other side of it. There's a certain order to it. So you want to order transactions by the most fees first, whenever possible. But if that's not possible because there's a child for parents, you might have the first transaction is a cheap parent, the second transaction is a... cheap child, and then the third transaction is a grandchild that pays a lot. And that's the first chunk. And then the next chunk might be some other uncle transaction that pays a little bit less than the super rich child, etc. And that means that if you cut the tail off from the cluster, you're throwing away the least valuable part of it, but you're keeping it. And you can cut it at multiple places. But not at all places, right? Because like you said, some transactions depend on each other. And so it's sorted in a way that you can cut it in a number of places without breaking those dependencies.

Aaron van Wirdum Right.

Sjors Provoost And then you cut the worst part. And for mining, you do the opposite. So for mining, you look at the best cluster and the best part of the cluster.

Aaron van Wirdum Maybe I'm going a bit too much into the weeds right now, but in this hypothetical scenario that you describe, you're starting with an empty mempool, basically. But in this scenario that you describe, you're starting with an empty mempool, basically.

Sjors Provoost Yes.

Aaron van Wirdum But in reality, most of the time, you don't start with an empty mempool. Your mempool is kind of always full by default, right?

Sjors Provoost Well, not today, but let's say it is full, and a new transaction comes in. So then what you have to do, as far as I understand it, is you look at your mempool and see which is this relevant to any of the clusters I have, or is it a completely new cluster because it's unrelated to anything I've seen. If it is a new cluster, then you just see, okay, is this new cluster better than any of my existing clusters?

Aaron van Wirdum But then you need the other transactions in the cluster as well, right? Otherwise, how can you tell?

Sjors Provoost So a new transaction can be a cluster of one. It's not related to anything else in the mempool. It's basically just spending from something that's confirmed.

Aaron van Wirdum Yes, but what if that is a very low fee-paying transaction and you don't see the rest of the cluster yet, then you would normally throw it out, even though maybe the rest of it...

Sjors Provoost Yeah, so if you receive a new transaction...

Aaron van Wirdum So for that, you need package relay, ultimately, right? Or some form of it that is so compatible with...

Sjors Provoost Yeah, so if you see a new transaction that is not related to any of the clusters you're aware of, but it is part of a cluster, apparently... then yeah, you might just reject it. And you'd need something like package relay, especially big package relay to say, okay, here's somebody escaping from their Ark, for example, 25 transactions. I see the first one of that 25 transactions. I'm like, well, this is a new cluster, but it pays nothing. So I'm going to ignore it. And the second one, I'm also going to ignore it because it's paying from a parent that I, you know, spending from a parent that I don't know about. So that's where this comes in. But in the case where it's just a single transaction and you need to evaluate it on its own, either it's better than any of your clusters or parts of the clusters, and then you know where to sort it and you put it in the mempool, or it's not and you ignore it. And that's a fast calculation, so you can do that when you get lots of new transactions every second.

Aaron van Wirdum Okay, so then I think this will be my last question on this topic. Ideally, we'd want package relay for clusters, right?

Sjors Provoost I think so, but some people might say no. I'm not sure.

Aaron van Wirdum But in any case, that's not here yet. That's not here yet.

Sjors Provoost No, the only package relay for clusters is a cluster of two.

Aaron van Wirdum Okay, right. Okay, well, that answers my question. That was my question. So right now, cluster mempool works for your mempool, and you make more complicated, complex, advanced. calculations for that, but it's not necessarily all related as a package yet.

Sjors Provoost Exactly.

Aaron van Wirdum And maybe you say there's arguments against doing that also.

Sjors Provoost Maybe. I mean, if you look at the one child, one parent example in Lightning, the TRUC transactions or the ephemeral dust transactions, the question there is what is the maximum cost that an attacker can impose on you? So right now I think that's capped at like 10,000. sats in total, so that's like $5 now, the worst that the attacker can do to you. But if you say, okay, we're allowing a cluster package of size million transactions, then, I don't know, maybe the attacker can do more damage to you. But maybe that's not too bad. I don't know.

Aaron van Wirdum Okay, in any case, short summary, cluster mempool, your node, your mempool, will make more advanced calculations to determine what to keep and what to throw out. which benefits miners that would use this to create blocks and individual nodes because then their mempools better reflect the blocks that are actually mined, which benefits the estimation and stuff like that and relay. Correct?

Sjors Provoost Yeah, I think some of this will or is being used for fee estimation too.

Aaron van Wirdum Right. Does that conclude our topic?

Sjors Provoost Yeah, I think so. We can move on.

Aaron van Wirdum Okay. So the next upgrade in Bitcoin Core 31, there's now a function called private broadcast.

Sjors Provoost A startup option called private broadcast.

Aaron van Wirdum A startup option called private broadcast. Okay.

Sjors Provoost And it's been fixed in version 31.1. So if you want to use it, you should use 31 .1 to be on the safe side. What it does is, so right now, if you make a transaction and you send it, it just goes to a bunch of your peers. And that's not great if one of your peers happens to be Chainalysis and they can sort of figure out, oh, well, I saw it from you first and not from others, so maybe it's yours, right? So you're kind of leaking, potentially leaking the origin of your own transaction. In the past, there were proposals like Dandelion. I don't know if we've ever covered that.

Aaron van Wirdum Oh, that's years ago.

Sjors Provoost Yes.

Aaron van Wirdum That's like a 10-year-old proposal. I don't think we ever covered that.

Sjors Provoost Okay. Yeah, it's quite an old proposal. It's quite cool, but it basically said, okay, we're going to send a transaction, one hop, another hop, another hop, like just as a stem basically, and then it explodes at some random node in the distance.

Aaron van Wirdum Right.

Sjors Provoost That turns out to be extremely complicated to implement for all sorts of reasons. But there's a sort of a poor man's dandelion, I guess, where it's this. So instead of connecting to your peers, when you have this new transaction, you open a new Tor connection.

Aaron van Wirdum Just good old Tor?

Sjors Provoost Yes. Completely new Tor circuit, I think. And then you find a peer randomly on the internet. You connect to it. You say, hi, here's the transaction. Bye. And then you wait. And you see if any of your peers, normal peers, announced this transaction.

Aaron van Wirdum To you, basically.

Sjors Provoost To you, because then you know it kind of went through the network. If not, then you just keep trying another Tor peer, etc.

Aaron van Wirdum Right.

Sjors Provoost So this is still a bit cumbersome to use, but it's there.

Aaron van Wirdum Right, right, yeah. Okay, so yeah, the problem, I mean, yeah, one of the ways to de-anonymize Bitcoin users is basically for, you know, the companies like Chainalysis. Am I pronouncing it?

Sjors Provoost There are different people who pronounce it in different ways for different reasons, but it's Chainalysis.

Aaron van Wirdum But there's different companies, right? They actually monitor the Bitcoin network itself to see where transactions originate, which is not foolproof, but at least it's supposedly used, right?

Sjors Provoost Yeah, so the first peer that you, if you're a spy and you're connecting to lots of different nodes, and the first node you hear about for a given transaction is, you know, more likely to be the original one than the second node you hear it from.

Aaron van Wirdum Yeah, exactly. Okay, so the, and then the solution now is to use, to just use, sounds simple enough. Why is this new? I would say, I mean, I didn't propose it 10 years, so it's easy to say now, but you would think someone could have just come up with this 10 years ago. Is there something difficult about this?

Sjors Provoost No, I suspect the idea itself actually might already be at least five years old, if not more. It's possible that somebody proposed it, then came up with Dandelion. That sounds really cool. And then you try that and it doesn't work. It took another five years to get this one-shot Tor thing through. I think the idea is not super new. The downside, I think, is relying on Tor. Because it's kind of like this black box trust assumption. Like, okay, we hope Tor actually works and doesn't. Because if this doesn't work, let's say Tor was fake. Like, it doesn't actually protect your privacy. Then you are creating a connection to the spy agency and telling them, this is my transaction.

Aaron van Wirdum And I want to keep it private.

Sjors Provoost Because you're only using this connection to send your transactions. There's no decoys.

Aaron van Wirdum Yeah, it's a dependency, right? Ideally, you would not want to...

Sjors Provoost And there's some practical issues. How do you configure your node to use Tor, etc.? Now, a lot of these node-in-the-box solutions will do that for you.

Aaron van Wirdum Okay, so this is new in Bitcoin Core 31, because it's not by default, right? You have to actually opt in somehow?

Sjors Provoost Yeah, you have to turn on a setting, and then you have to do something special when you make a transaction to use this, I think.

Aaron van Wirdum Okay, and if I understand correctly, this is not implemented in the wallet right now. As a Bitcoin Core wallet user, you can't use this. You would have to... What do you actually have to do? You have to construct your own transaction and then...

Sjors Provoost Yeah, I think you create the transaction as normal, but you don't send it, you don't broadcast it, you just copy it and then send it over, copy it to an RPC command.

Aaron van Wirdum Okay.

Sjors Provoost So it's a little tedious, but...

Aaron van Wirdum It's a little bit manual labor.

Sjors Provoost Yeah, but I can imagine that you could add this to the wallet. It's not rocket science, it's just one step at a time.

Aaron van Wirdum Yeah. Okay, so that's a new option, that's a new feature in Bitcoin Core 31. I think that basically covers it, right? It's kind of straightforward, or is there more to say about this?

Sjors Provoost Yeah, no, that covers it.

Aaron van Wirdum Okay, great. More privacy is great. I like it. Okay. ASMAP.

Sjors Provoost Or ASMAP, depending on your mood. We had an episode about this, so just listen to it. Back then, we...

Aaron van Wirdum Did we do a whole episode on only ASMAP?

Sjors Provoost I think so. We had a whole bunch of episodes about eclipse attacks, and one of them was about ASMAP.

Aaron van Wirdum Okay, but to recoup that a little bit.

Sjors Provoost The quick summary is that we don't like eclipse attacks.

Aaron van Wirdum What is an eclipse attack again?

Sjors Provoost It's where you are connected to only one.

Aaron van Wirdum Pretending to be several.

Sjors Provoost Pretending to be several. And then they can feed you maybe an alternative blockchain or pretend there's no update or all sorts of shenanigans you don't want.

Aaron van Wirdum If every pair you're connecting to is actually the same attacker, then they can give you blocks that are... double spending or whatever. It's not necessarily the largest chain you're seeing then.

Sjors Provoost And if you are a miner, then they can make you produce a block that's not actually on the most work chain, and then they can abuse you to double spend somebody else. So it's not just you as a normal node, but it's also you as a miner node where you do not want to be eclipsed.

Aaron van Wirdum Yeah. Bottom line, you want to be connected to the actual network. You want to see actual blocks that the rest of the network sees. So an eclipse attack stops that. So this is a solution, or this is a measure against eclipse attacks.

Sjors Provoost Yeah, so over the years, lots of different measures have been taken to prevent eclipse attacks. One of the measures that's been in there for quite long, I think from the beginning, is that when we decide who to connect to, we use this thing called buckets. We have a whole list of peers, or a whole list of... nodes in our database that we think are nodes, so just a bunch of IP addresses, essentially. And we say, okay, let's take all the IP addresses that start with a 1, and we make only one connection to those, and then we look at all the IP addresses that start with a 5, and we only make one connection to those, and that's how we diversify the people that we connect to. And that's just based on what the IP addresses look like, which is not great because there's companies like Amazon that have lots of different IP addresses that start with different first digits, basically. So ASMAP is a project to actually look at these owners and group those and try to diversify across actual owners of the network. So there's a bunch of university research projects out there that are monitoring. I think it's basically the gossip between different internet providers. So AS is an autonomous system. It's a group of IP addresses. It could be your ISP. It could be Amazon. It could be Google. Like sort of one entity that controls a group of IP addresses. And this stuff is gossiped around the internet because it's needed in order for you to find a route from... A to B goes through all these autonomous systems. And so there are these giant databases of exactly which IP address belongs to which autonomous system. And what happens is every month or so or every two months, a group of volunteers, Bitcoin Core-related volunteers, come together and at the same time try to download these giant maps from those research institutions, a couple of different ones in the world. And then they do that at exactly the same time because this map changes every second or more often. And then if they all have the exact same view, some math is performed on it and it becomes like a one megabyte file that's added to Bitcoin Core now. And this one megabyte file just means that for given any IP address, tell me that it's like, what group does it belong to? It doesn't really give you the name. It just says, okay, this IP address is a different group than this other IP address. Because the only thing you care about is that you don't connect to the same group twice, the same autonomous system twice.

Aaron van Wirdum Right, okay, the very, I think it's a pretty hypothetical attack, right? But the potential attack here would be that, say, Amazon, whoever controls Amazon or whoever hacked into Amazon. they control a whole bunch of IP addresses now, and these IP addresses don't even look the same because Amazon is so big. They have so many IP addresses under their control. So to prevent an Eclipse attack from such an attacker, there's a more accurate way to map which IP addresses belong to which entity or behind which ISP or something like that.

Sjors Provoost Yeah, exactly.

Aaron van Wirdum Okay, and from there, these buckets are then used to... for a node to say, okay, connect to a diverse range of it.

Sjors Provoost Yeah, exactly.

Aaron van Wirdum But we're not just connected to Amazon, we're connected to all kinds of different ISPs.

Sjors Provoost Yes, and so in practice, this probably means that some obscure internet provider in Zambia will now get more connections because they're sort of picked as the not standard option and Amazon might get fewer connections.

Aaron van Wirdum Right, okay. And these maps, S-maps, these are actually... These are files. You actually download these on your Bitcoin Core node. These are now on your hard drive.

Sjors Provoost Yeah, so what's already been available is, basically since we did that episode in 2022, is this ability to download this ASMAP file, put it in your node, and then enter it in your config, and it would use it. What's new now is that, by default, Bitcoin Core comes with one of those maps.

Aaron van Wirdum Okay, and uses them. Or is that actually?

Sjors Provoost No, you still have to turn that on.

Aaron van Wirdum Okay.

Sjors Provoost But now you just set asmap=1. And it works. Rather than downloading a file, putting it somewhere, etc. And then every time Bitcoin Core updates, the new version will have a new map. If you've turned this feature on, it will use the new map.

Aaron van Wirdum Right, yeah. It's not automatically updated or gossiped around. It's just whenever there's a new Bitcoin Core release, you get the newest ASMAPs.

Sjors Provoost Yes. And you can still download your own or make your own, but that's kind of difficult. And the reason it took so long is because... We kind of don't like shipping a blob that is hard to reproduce, is hard to verify. So that's why there's this ritual of like multiple people in different places of the world try to make the same AS map file at the exact same time. And then if like three of them or so have the same result, then there's at least some auditability that this process was fair. And then you can look at the code and see how it works. It's not ideal because the map of the internet, there's no official map of the internet. these basically maps that people are downloading exist for one second and then they change. Just like Google Maps, I guess, you know, every time there's a different street somewhere in the world, the whole map changes.

Aaron van Wirdum Yeah, there's no official map, but this is publicly available data. And the Bitcoin Core developers themselves actually use that to turn it into maps. This is a Bitcoin Core-specific project.

Sjors Provoost Yeah, I mean, I'm sure other projects could use it if they have the same need to avoid eclipse attacks.

Aaron van Wirdum Right. Okay. So that's also new in Bitcoin Core 31?

Sjors Provoost Mm-hmm.

Aaron van Wirdum Did that cover it?

Sjors Provoost Yeah, I think so.

Aaron van Wirdum Is there anything else to say about that?

Sjors Provoost No.

Aaron van Wirdum Okay. Smaller risk of eclipse attacks. Also sounds good.

Sjors Provoost Yeah.

Aaron van Wirdum I like privacy. I like smaller risks of eclipse attacks as well. All good so far.

Sjors Provoost Great.

Aaron van Wirdum What was the first thing again? Oh, cluster mempool. I like more fees and more accurate mempools as well, Sjors. All good so far.

Sjors Provoost I'm glad you're happy.

Aaron van Wirdum Very. Last one. Oh, no, I was to ask. Wait, was this this topic or the last one? No, this was the last topic on the private broadcast. I do want to ask. So you mentioned there was a bug and it's fixed. What was the bug, actually?

Sjors Provoost Yeah, if I remember, it's related to another episode we probably have done, which is about the V2 transport, the encrypted peer-to-peer network. What happens is if you try to connect to a node that has the encrypted... So if you try to connect to another node and you want to use a new encrypted communication protocol, if that doesn't work because they don't actually speak the encrypted protocol, you kind of disconnect and reconnect and then try the unencrypted protocol. Now, for Tor, it doesn't matter whether you use encrypted or unencrypted, if both sides are on Tor anyway, because Tor itself encrypts too, so it's going to double, but maybe you're connecting via Tor to somebody who's not on Tor through the exit nodes, right? So you need the encryption, but the... The problem was, I think that if the V2 connection didn't work, it would disconnect and reconnect, but the reconnection would not go over Tor.

Aaron van Wirdum Right.

Sjors Provoost So that's very bad.

Aaron van Wirdum Right, so you're trying to get privacy, but then when it fails, you just give up on the privacy.

Sjors Provoost Yeah, and this is exactly what I said. One of the reasons why these features like this are scary is because you are trusting the black box Tor, and in this case, if you make a mistake, because you're only using this connection, for this thing that you say it's yours, it's extra dangerous when something goes wrong, like this.

Aaron van Wirdum Right, and so it did go wrong at the initial release.

Sjors Provoost Yeah, I'm sure it was somewhat of an edge case, but it still makes it clear that you want to be careful with features like this, because they can be a foot gun if they go wrong.

Aaron van Wirdum Right, and in any case, it was fixed in 31.1.

Sjors Provoost Yeah, and that's why we waited with this episode, right?

Aaron van Wirdum Exactly. Okay, last topic. I think that's the last topic for us, right?

Sjors Provoost Already?

Aaron van Wirdum Yeah. I wrote down IPC interface, and I already forgot what it... IPC interface, Sjors? Tell me.

Sjors Provoost Oh, yeah, that's the mining-related stuff.

Aaron van Wirdum Oh, right, yeah.

Sjors Provoost Stratum V2-related.

Aaron van Wirdum Right. So this is... Okay, just take it. Go for it.

Sjors Provoost Yeah, I don't know how much we've covered that Stratum V2 in earlier discussions, but...

Aaron van Wirdum I think we had a whole episode on that also at some point. But that's maybe, like... three years ago by now. I think that's been a while.

Sjors Provoost In any case, the Stratum V2 system allows you to make your own block templates, among other things.

Aaron van Wirdum If you're a miner, yeah.

Sjors Provoost Exactly. Well, you can make them even if you're not mining.

Aaron van Wirdum It's kind of useless.

Sjors Provoost So the initial idea was to have Bitcoin Core actually speak the Stratum V2 protocol and provide block templates. but it was a bit controversial to add even more complexity to Bitcoin Core because it's already a big project. So the idea then became to say, okay, we're not going to implement Stratum V2 itself, but we're going to just offer all the information that Stratum V2 needs. So you install Bitcoin Core, and then you install a separate piece of software called a template provider. That template provider speaks to your own node on one side, and it speaks to Stratum v2, the rest of the Stratum V2 ecosystem on the other side.

Aaron van Wirdum It's kind of like a translator.

Sjors Provoost The mining pool or yet another piece of software, because Stratum V2 can be multiple pieces of software.

Aaron van Wirdum Okay.

Sjors Provoost So in the scenario where you are mining your own block templates, you have Bitcoin Core that talks to the template provider. The template provider then talks to the job declarator server, sorry, client. And the job declarator client talks to the mining pool. to declare the job so it doesn't really matter like in practice nowadays I think there's wizards and so you just say do you want to solo mine or do you want to use a pool do you want to make your own templates yes no and it just installs whatever you need so it hopefully gets to a simplicity level that's comparable to what DATUM is doing where you just download like one thing but basically Bitcoin Core does not speak does not understand any of that all it does is just create new block templates for you And the way the Stratum software talks to the node is using this thing called IPC, inter-process communication. And it's just a more efficient way than RPC, what we had before. And that was all part of Bitcoin Core v30, so we might have even covered it two or three episodes ago.

Aaron van Wirdum What's new? Because I imagine you could already solo mine, for example, with Bitcoin Core. You could already construct a block, right? You could already... say I want to mine or if I were to mine.

Sjors Provoost Yeah, Stratum V1 or just Stratum has been around forever and that was using the RPC and you would call a method called getblocktemplate and it would create a very large JSON blob with all the transactions you need, etc. The problem with that was that it was kind of poll-based. So you as the mining software or the miner, you need to constantly tell your node, okay, can you make a new block for me? Can you make a new block for me? Etc. And there was a way to say, okay, make me a block and let me know when there's a better block available. But the only thing it did was check if anything changed in the mempool. And maybe there was a long time ago that that was actually interesting, but nowadays the mempool changes all the time, so you would just get a new block every fraction of a second, which is not useful.

Aaron van Wirdum That's not useful?

Sjors Provoost No, you don't want to have... You don't want to have a new block if it's not any better. So something changed in the mempool could be like the bottom one gigabyte of your mempool changed and you get a new template. That's not useful.

Aaron van Wirdum It sounds like this maybe ties into the first thing with the cluster mempool.

Sjors Provoost So what the Stratum V2 stuff lets you do, the IPC interface lets you do, is... First, you ask for a block template. That's kind of similar to how it worked before.

Aaron van Wirdum It's what goes into a block.

Sjors Provoost Yeah, a little bit more efficient in terms of the encoding. But for this discussion, it doesn't matter. And then you can subscribe to updates. And in particular, you can say, give me a new block if the fees have gone up by at least one sat per vbyte. So not any change in the mempool, but the actual block itself is better.

Aaron van Wirdum Yeah, okay.

Sjors Provoost And you can make it a thousand or whatever number there is because... you have some mining infrastructure, some ASICs that maybe you don't want to give them a new template like 100 times a second because it takes them time to update.

Aaron van Wirdum Right.

Sjors Provoost And updating is... When it's updating, it's not mining, so you're losing some revenue, so maybe you only want a new template if it's 100 sats or whatever number makes sense.

Aaron van Wirdum Right. Okay, so if I understand correctly, this is not... technically a Stratum V2 upgrade but it is very compatible with Stratum V2 so Bitcoin Core 31 released something that's particularly compatible with Stratum V2 and and specifically when it comes to creating your own blocks you know as a miner deciding what transactions you want to include in a block

Sjors Provoost yeah this was in V30

Aaron van Wirdum you still need some other pieces of software that's the actual Stratum V2 software that takes that data and mines or connects to a pool and or whatever else is needed. We don't have to get into that.

Sjors Provoost Yeah, exactly. So you still need to install some extra software, but Bitcoin Core has methods that are tailor-made for that software, but could be used by some other mining protocol too.

Aaron van Wirdum Right.

Sjors Provoost It could even be used for Stratum V1-style mining, right? If you run some pool software, it could use the IPC too.

Aaron van Wirdum Right, and you mentioned DATUM, so is this also compatible with DATUM?

Sjors Provoost If they want to. They're not currently using it, but they could. As far as I know, there's nothing magic that they're doing that doesn't work with this, and otherwise they should open a GitHub issue.

Aaron van Wirdum Do you know why they don't use it?

Sjors Provoost I think partly because they were building this before the IPC interface was even finished. So it's a matter of just sequencing.

Aaron van Wirdum Because Bitcoin core is woke source. That's what I heard.

Sjors Provoost That's possible. Maybe there's some evil feature in there. I remember making a GitHub issue on the Bitcoin.

Aaron van Wirdum We were just discussing ASMAP. That's at the very least a bit queer, don't you think?

Sjors Provoost So I was making a GitHub issue. on the Bitcoin Core repo, asking the DATUM people, like, hey, is there something you need in the IPC interface? And I think Luke probably replied with something like, well, you guys are fascists or dictators or something like that. So, like, okay, I guess not. But they can still ask for things if they're missing something.

Aaron van Wirdum Right. Okay. In any case, this is very compatible with Stratum V2.

Sjors Provoost And so this was already in V30, but V31 just made it better. It fixed some bugs, added some whistles, some bells.

Aaron van Wirdum Right.

Sjors Provoost So it's great.

Aaron van Wirdum Okay.

Sjors Provoost I think that's all I've got. So anything else?

Aaron van Wirdum Yeah, I think so as well. I mean, no, I think this was our 100th episode then, Sjors.

Sjors Provoost Excellent. Well, thank you again then for listening to Bitcoin.

Aaron van Wirdum Explained.