Episode 49: The Fake Peers Attack!

Aaron van Wirdum Live from Utrecht, this is Bitcoin Explained.

Sjors Provoost Hello, and welcome back to Utrecht.

Aaron van Wirdum Thank you, Sjors. This is the first time in a long time that we're recording live in Utrecht, I think.

Sjors Provoost Yes. Something about El Salvador and other places.

Aaron van Wirdum Zurich.

Sjors Provoost Zurich, yep.

Aaron van Wirdum Yeah, so we're back. We're back in cloudy, rainy the Netherlands.

Sjors Provoost Actually, today it's nice. Half clouded.

Aaron van Wirdum Not compared to El Salvador, though, Sjors.

Sjors Provoost I don't know. I haven't been there.

Aaron van Wirdum So today we're going to talk about... You really scraped the bottom of the barrel for this one.

Sjors Provoost I did?

Aaron van Wirdum We're running out of the main topics and now you're looking in the [unclear] of Bitcointalk for remaining stuff that we can discuss in this podcast. Can't you?

Sjors Provoost That's one way to bring it. The other way would be to say, did you know that Bitcoin was attacked?

Aaron van Wirdum Wow, that's a much better angle.

Sjors Provoost And that the media is hiding this from us because they haven't been talking about it. It's a cover-up.

Aaron van Wirdum Yeah, we have a...

Sjors Provoost We're going to bring the truth.

Aaron van Wirdum We're going to bring you the truth on this episode on the attack that was happening on Bitcoin. It was the attack of the fake peers.

Sjors Provoost The attack of the fake peers, yes.

Aaron van Wirdum It almost sounds like a movie title even.

Sjors Provoost Yeah, I mean, there's David Gerard's book, you know, Attack of the 50-foot Blockchain. But this is the attack of the fake peers, yep.

Aaron van Wirdum All right, so Sjors. So I think I'm just going to… give you the stage on this one. What was the attack? What is the attack of the fake peers? What happened, Sjors?

Sjors Provoost Well, it was a cold day in July. So somewhere in July...

Aaron van Wirdum So it must have been in the Netherlands it was cold.

Sjors Provoost Exactly. Somewhere in July, what people started noticing, people who are running nodes, is that sort of random... people were connecting to them. Now, of course, this happens all the time. But these random people that were connecting to them would connect to them and then send, I think it was 500 messages. And each of those 500 messages would contain 10 addresses that were supposed to represent other nodes in the network.

Aaron van Wirdum So when you say addresses, you mean IP— IP addresses in this context?

Sjors Provoost Yes, that's right. So they would connect you and say, hey, did you know about these 10 IP addresses? And did you know about these 10 IP addresses? And they would just do that 500 times, and then they would disconnect. And that's a bit weird.

Aaron van Wirdum Yeah, so maybe before we move on, we should take a small step back, which is we discussed this in previous episodes: You're our library guy. So maybe you know which episodes these were, but we discussed in previous episodes, how nodes essentially bootstrap to the network. That is, how do Bitcoin nodes find other Bitcoin nodes, right?

Sjors Provoost Yeah, we discussed that in episode 13. And we talked about, yeah, so the node starts up and it usually asks, looks at a DNS seed, say, hey, tell me something. There's like five or six DNS seeds of people who are... It's not really trusted, but it is definitely not untrusted either. They get a list of nodes to connect to, and then your node will just randomly try a couple and others and others.

Aaron van Wirdum Yeah, and now you're using the word node, but these are also IP addresses, right?

Sjors Provoost Yes.

Aaron van Wirdum Yeah. So you start with a small group of IP addresses. And then from there, the nodes you connect to, they will share more IP addresses of other nodes with you. And that's how you connect to more nodes. And that's sort of how the network forms, right? So there are messages between nodes sharing IP addresses about other nodes.

Sjors Provoost That's right.

Aaron van Wirdum Right. And it's these messages that you just explained were being... Spoofed? Is that the right way to put it?

Sjors Provoost Well, the messages are real, but the contents was nonsense. So, indeed, a node would connect to you, and they would send you a bunch of addresses, but it turns out those addresses were just random numbers. So an IP address is just a number 1 to 255, and then another number 1 to 255, etc. Four numbers, usually with IPv4. And yeah, those numbers were just randomly generated. So if you were to map them out, you would see they were all over the spectrum. And that's not actually what the internet looks like because a lot of IP addresses are not used at all. And so it included IP addresses that just cannot exist. So clearly they were artificial IP addresses. And the problem with that is that then the odds of there actually being a node there is not that good. Because if you're just making up a random IP address, then there might not be a node there. The whole point of gossiping the nodes is that you get actual nodes. Because if you could just try random IP addresses yourself if you wanted to.

Aaron van Wirdum Right. So there were nodes on the network that were sharing random IP addresses with other nodes on the network. And that's what we're defining as an attack here because the IP addresses that they were sharing were just random numbers and they didn't actually point you to a real Bitcoin node, right?

Sjors Provoost Yeah. And apparently, according to the people who did some research on it, it happened on a fairly large scale. So these people were connecting to lots and lots and lots of nodes in the network, at least to the nodes that are listening. Because one of the things is if you're starting a node, you might not be listening to the outside world. It depends, especially if you're behind a router. It's not always the case. But if your node is listening to the outside world, then what it does is it actually tells its peers its own IP address. So the first time you connect to another node, you're saying, hey, by the way, this is my IP address. Please spread the word.

Aaron van Wirdum Do you say it's yours or do you just give an IP address?

Sjors Provoost I think you just give one.

Aaron van Wirdum Right.

Sjors Provoost And, you know, the other node doesn't really know. But you can give up to 10. And that's kind of the mechanism that this attack is exploiting is kind of saying, hello, here's 10 IP addresses. And the usual assumption is if that's a new node, then, you know, the first one is probably that node. So they're kind of abusing the way that nodes used to introduce themselves to the network. Because once your address has been gossiped around, then people can connect to you and give you good stuff.

Aaron van Wirdum All right, so I'm running a listening node. I don't think I actually am, but for the... As a matter of examples, Sjors, I'm running a listening node.

Sjors Provoost Yeah.

Aaron van Wirdum Now, there is this other node that we just explained that is sharing random IP addresses. So it's sharing that with my node, and we just define this as an attack. Why are we defining this as an attack? What's the problem for me or for my node? How am I... you know, how am I a victim of what's going on?

Sjors Provoost Well, it's not a big problem for your node in reality, so it's kind of a mild attack, you could say, at least in terms of it's not an attack that would kill your node in any way. It would just get these IP addresses, and then, you know, we talked about it in earlier episodes, I think. Anyway, when a node gets a list of IP addresses, it puts them in a bunch of buckets and shuffles them around and makes sure that it doesn't listen too much to, like, the same source. And so, you know, you already have lots of IP addresses from earlier, from honest nodes. So most likely it won't bother you too much. You might connect to a few nodes that don't exist. You know, you're just wasting some of your time, but not a huge amount.

Aaron van Wirdum But I would connect to these IP addresses or would I just notice, would my node just notice that these aren't nodes and just disconnect immediately?

Sjors Provoost Well, that's, that's, but that process takes time, right?

Aaron van Wirdum Okay, yeah, that's what I'm asking.

Sjors Provoost Your node is keeping a list of nodes, of IP addresses that it could potentially connect to if it needs more connections.

Aaron van Wirdum Got it.

Sjors Provoost And then it'll go through that list that is a useless list, basically.

Aaron van Wirdum Right, so it's wasting some of my resources. It's wasting some of my resources because my node is storing IP addresses that aren't real Bitcoin node IP addresses. So I guess that's a little bit of waste there. And then once in a while, I'll try to connect with the node and there's a little bit of bandwidth waste. Yeah. And that's sort of it.

Sjors Provoost Yeah. So at the individual level, this is not really, you know, you could call it an attack. It's like a kid throwing a little pebble at you. You could tell the kid, hey, you're attacking me and you could shoot the kid. But usually we just, you know, keep walking.

Aaron van Wirdum Got it.

Sjors Provoost And nobody noticed it, right? Most people who are not actively looking at their node would have not even know this attack happened. So that's good.

Aaron van Wirdum So then why would an attacker, I think that's the next question, why would an attacker even bother then? Because clearly someone was bothering to do that. So what is the potential benefit of this type of attack?

Sjors Provoost So there's two people who wrote a paper, Matthias Grundmann and Max Baumstark, and we'll link to it in the show notes from, I believe, the Karlsruhe Institute of Technology. And the department is the Institute of Information Security and Dependability. And I guess that might give a little hint. So what they're guessing is that this attacker was not so much trying to destroy the network as they were trying to map the network to get a sense of how well nodes are connected to each other. And the reason they can do that is because when you receive these 10 IP addresses, you will forward some of them to some of your peers, but not all of them. So you get some exponential decay where you send them to your neighbors and their neighbors send some of them to their neighbors. And so if you're the attacker and you're also just running regular nodes, then eventually you'll hear some of the echo of your own attack, basically, because your peers will eventually relay it back to you. And by looking at this sort of this echo, you can determine a little bit of what the network looks like, the shape of it, how well connected it is, how robust it is.

Aaron van Wirdum Right. So in the same way that individual transactions, for example, make their way through the network by nodes forwarding the transaction to other nodes, they are also forwarding these IP addresses to other nodes. So if you keep track of how the IP addresses are shared over the network, you learn something about how nodes connect to each other, which nodes connect to each other, in what order they connect to each other, that kind of stuff.

Sjors Provoost Roughly, though not precisely.

Aaron van Wirdum Yeah, and you can potentially use that to analyze the actual transactions as well. So if you want to learn something about where transactions originate, for example.

Sjors Provoost I don't think you can with this type of attack. That's at least what I've been told. But you can get a general sense. At least you can get a general sense of the shape or maybe the robustness of the network. So that's useful information then if you want to attack in the future, perhaps. Or more likely, my guess would be maybe it was just people doing research, some academic research, trying to find out what the network looks like.

Aaron van Wirdum Right.

Sjors Provoost And maybe they'll publish a paper next year.

Aaron van Wirdum Right. So what it does is it helps you map the network. And for whatever reason you want to do that, we don't know that, but it helps you map the network. It helps you figure out who's connecting to who.

Sjors Provoost Yeah, but there are some defense mechanisms in nodes already to make it not too easy to use this information. So for example, one thing is if you're telling a node a bunch of IP addresses, it's not immediately going to connect to all of them because that would be kind of obvious. And it doesn't relay all of them. And there's some time delay in when it relays some of them. So it makes it very difficult to say specifically which node connects to which node connects to which node.

Aaron van Wirdum And these are these defense… mechanisms specifically for this type of attack then or…

Sjors Provoost That's my guess but I'm not sure.

Aaron van Wirdum And have they been in Bitcoin Core for a long time or are they new?

Sjors Provoost I think they get incrementally added, I guess, when people do these types of attacks then, then people like Greg Maxwell and Pieter Wuille will look at it and like maybe we can add this little defense.

Aaron van Wirdum I see. So these are attacks that have been happening more often over the past couple of years?

Sjors Provoost I think so. Not the specific type of attack.

Aaron van Wirdum But these general types of IP sharing attacks?

Sjors Provoost I think just probing the network in weird ways.

Aaron van Wirdum Right.

Sjors Provoost But it's not really my area of expertise, so I can't really tell much more about that than this vague answer that I just gave you that it does seem that people are probing the Bitcoin network.

Aaron van Wirdum Fair enough.

Sjors Provoost Often probably just for research, but who knows? There might be some evil army out there.

Aaron van Wirdum So is there a solution? Is there a definitive solution? Is this even a concern that developers feel like deserves a solution?

Sjors Provoost So there is indeed a countermeasure that's been added. Like we said, it wasn't a... Yeah, so that's kind of the cliffhanger we can end on. But first, let me explain what the countermeasure is. The countermeasure basically says that… And normally, when people are acting nice, they will connect to you and they'll send you one IP address, namely their own. And then occasionally they'll send you some other IP addresses, but not very frequently. And people have measured the average speed. I think it's like 20 times— No, once every 20 seconds on average or something like that. And so this defense mechanism...

Aaron van Wirdum Once in every 20 seconds, an average node will share an IP address with a peer.

Sjors Provoost Yeah, it might be a different number, but anyway, there's some average number.

Aaron van Wirdum Something in that ballpark.

Sjors Provoost Yeah. And so you can use that by introducing a sort of a rate limiter. So a rate limiter basically says, okay, when a new node connects to me, I'll allow it to send me one address immediately. And then I'm going to allow it every 20 seconds up to one address. So it just tracks how many seconds have gone by, and if it's sending too many addresses, it'll just ignore the new ones that go over this rate limit. So you don't get punished for it. But this way, when somebody connects to you and sends a firehose of addresses to you, you just ignore all of them.

Aaron van Wirdum Right.

Sjors Provoost Except the first one. Now, there are cases when nodes actually want to receive addresses from their peer, and there's a special message from that. So if somebody connects to you and you say, please tell me addresses, like give me up to 1,000, then, of course, you will not rate limit the response. You'll make sure that they can actually give you those addresses. But if it's unsolicited, then you rate limit it.

Aaron van Wirdum Sure. So it's rate limited unless you overwrite that limit.

Sjors Provoost Yeah, and that pretty much gets rid of this attack. Now, the interesting part is that apparently...

Aaron van Wirdum Is this the cliffhanger?

Sjors Provoost This is the cliffhanger.

Aaron van Wirdum Nice.

Sjors Provoost This fix was added before the attack happened. A few weeks before. Or even just one week before.

Aaron van Wirdum So the fix was added in Bitcoin Core.

Sjors Provoost Yeah, so not released. It was just in the... No, it wasn't even merged. So it was an open pull request.

Aaron van Wirdum Right.

Sjors Provoost So that means it proposed change to Bitcoin Core. And it was open. And then I think a week later or so that attack happened. And it was then merged like a couple of weeks later, but it's been released now in version 22.0.

Aaron van Wirdum Right. So the problem was anticipated. A solution was developed. And before the solution was actually deployed, the problem was abused.

Sjors Provoost Yeah. So it almost sounds like somebody saw the solution and thought, huh, maybe I can do this attack. Or perhaps somebody was already planning this attack and then thought, oh shit, I'd better do it now because it won't be possible anymore soon.

Aaron van Wirdum Yeah, that's an interesting point about open source development in general. Now, this is a very innocent example, but we've seen other examples of this. I remember back in the Bitcoin Unlimited days, if you remember that, where they had this alternative implementation that had a bug in it; then it was basically the bug was fixed but before the fix was deployed, the bug was exploited and it brought down all the Bitcoin Unlimited nodes at that time.

Sjors Provoost Yeah, so around 2013 on Bitcoin, something similar didn't happen, but could have happened, where I think the OpenSSL library was made, so basically it was made stricter in a soft fork by saying, okay, the signatures have to be— I don't know, some positive number or not a negative number, some some constraints on the signatures— was made because otherwise OpenSSL would be unpredictable or something like that. And it was presented as just a nice cleanup soft fork, but it was actually also a patch for a security vulnerability where somebody could have posted a slightly different kind of signature and caused a fork because some nodes would accept it and other nodes would not accept it.

Aaron van Wirdum Yeah, this is a very interesting problem. I think it has happened a few times before and it seems like the overall solution from Bitcoin Core developers is to sort of pretend that it's not a big deal until people have actually downloaded and used the software and then later they'll reveal that it was actually a much bigger problem.

Sjors Provoost At least in those examples that that seems to have happened. Yeah, and that's of course not ideal because in open source development you want to be very transparent about things you're changing, because you know, if you're being not transparent about fixing a critical bug, then maybe you're also not transparent about adding inflation, right? So, so it's, it's delicate balance.

Aaron van Wirdum Well, that's another example, actually. That was also the inflation bug a couple of years ago. And that was another example where the fix was presented as something very unimportant.

Sjors Provoost Well, no, it wasn't presented as something. It was presented as important, but it was not the full truth. It was presented as this will crash your node, this bug. And we fixed a bug that can crash your node. And that was true. It could crash your node. But it could also create inflation, which is a bit more important. And that was, of course, not announced.

Aaron van Wirdum Yeah.

Sjors Provoost Because somebody could have done that in that window of opportunity.

Aaron van Wirdum It's an interesting problem. Anyways, back to the fake peers attack. The fake peers attack happened, as we discussed, after a fix was actually written but not deployed yet. But now the fix is deployed. It was deployed in Bitcoin Core 22.

Sjors Provoost Yeah, so anybody who's running Bitcoin Core 22 won't have this problem and anybody who doesn't still doesn't really have the problem.

Aaron van Wirdum Got it. I think that that covers it all. Sure, sure.

Sjors Provoost No, I think so too. So in that case, thank you for listening to Bitcoin Explained.

Aaron van Wirdum There you go.