Episode 32: Segregated Witness (SegWit)

Aaron van Wirdum Before the show, a quick word from our sponsor. Live from Utrecht, this is the Van Wirdum Sjorsnado.

Sjors Provoost Hello.

Aaron van Wirdum Sjors, we're going to talk about a classic today.

Sjors Provoost Yeah, we're going to party like it's 2015.

Aaron van Wirdum SegWit.

Sjors Provoost That's right.

Aaron van Wirdum Segregated Witness, which was the previous soft fork. Well, it was the last soft fork. We're working towards the Taproot soft fork now.

Sjors Provoost It's the last soft fork we know of.

Aaron van Wirdum Yes, I guess so. It activated in 2017. It was developed in... It started being developed in 2015.

Sjors Provoost I think it was late 2015, yeah. the final idea came about to turn it into a soft fork.

Aaron van Wirdum Yeah, and it's probably, would you say, it's probably the biggest protocol upgrade Bitcoin has seen so far, right?

Sjors Provoost Well, it's the biggest protocol upgrade we've seen since the days of completely reckless deployments of upgrades, right?

Aaron van Wirdum Wouldn't you say it's almost the biggest code change, for example?

Sjors Provoost Yeah, I think so. It's a bigger change than P2SH, but I don't know what was done in the very early Satoshi days, you know, when hundreds of opcodes were turned off and all that stuff.

Aaron van Wirdum So where do we start? Do we start...

Sjors Provoost We could start with what the problem was.

Aaron van Wirdum Okay, what was the problem? Why do we need SegWit, Sjors?

Sjors Provoost Yeah, why do witnesses need to be segregated?

Aaron van Wirdum Exactly.

Sjors Provoost So the problem was transaction malleability. And transaction malleability means that… If I'm sending you some coins and you're sending them to Ruben, who's not here, then that transaction that you're sending refers to the transaction that I just sent. And the problem there, and that's fine normally, but the problem is that somebody could take our transaction and manipulate it. It could take my transaction and manipulate it and then your transaction would no longer refer to my transaction but would refer to a void.

Aaron van Wirdum Yeah, and to be more specific, I think, what the part of the transaction that's being manipulated is actually the signature. So every transaction is signed with a cryptographic signature and the signature I don't understand the details, but I know that the signature can be tweaked somehow in a way that it looks different but it's still valid.

Sjors Provoost Yeah and there were lots of ways to do that. So one of the ways that was fixed without SegWit is that you could, I think, just multiply the signature with minus one or something, just put a minus in front of it, and it would still be valid. And so anybody could just put that minus in front of it. So you would broadcast a transaction and it would go from one node to the other. Somebody else could see that transaction and they could say, well, I'm just going to flip this bit and send it onwards and then we'll see which one wins. And this is just for… simple signatures but I think there were other… if you have more complicated scripts, there are also ways that somebody can mess with that script.

Aaron van Wirdum Yeah, so someone can mess with it in flight. Basically, you send the transaction to the network, and then it's forwarded from peer to peer and until it reaches miner and it's included in a block, but every peer on the network… but can basically take the transaction, tweak it a little bit, and forward it, or the miner can do that.

Sjors Provoost Yeah, I guess even the person making it can do it.

Aaron van Wirdum Sure.

Sjors Provoost Yeah, or the miner can do it. So you may ask yourself, why is this a problem, right? Because I sent you some coins and you sent them to Ruben. Okay, I ruined your transaction, so you just sent it again. So that's not a big deal.

Aaron van Wirdum You didn't really ruin the transaction, you just tweaked it a bit, but it's still valid.

Sjors Provoost Well, I ruined your transaction. So I sent coins to you and you sent coins to Ruben, but that last transaction no longer points to an existing transaction because somebody messed with my transaction.

Aaron van Wirdum Yeah, it's the second transaction that's getting in trouble.

Sjors Provoost Yeah. And this is not a problem in the scenario we just described, right? Because you can just make a new one. Except, what about if... You're not you, but what if I sent a transaction to a super secure vault in the Arctic, like thousands of meters underground? And then I went to the Arctic and I created a redeem transaction back to my hot wallet, but I didn't broadcast it. I just… I just signed it.

Aaron van Wirdum Why are you making such a complicated example with vaults and arctics?

Sjors Provoost Well, this one isn't that complicated. I just, you know, I went to my vault. I created the transaction out of my vault. And then… I basically buried the vault in like… like 100 meters of rock. So it's very difficult for me to go back to the vault and make a new transaction. And then I broadcast my original transaction that sends the money to the vault and somebody messes with it. Now I have to go back to Antarctica and there's you know there's COVID and it's very complicated. So that's a terribly difficult example. Another example would be Lightning.

Aaron van Wirdum Yeah, that's the more obvious one.

Sjors Provoost Yeah, so with Lightning, what happens, and we've explained Lightning in earlier episodes, but the idea is you sent money, two people sent money to a shared address, and then the only way to get money out of that address is with transactions that you've both signed before you sent money into that address. So you don't want somebody messing with the transaction that goes into the address. Because then you can't spend from it anymore. Or you can, but you'd both have to sign it again. And so one party could kind of cheat the other party out of the coins.

Aaron van Wirdum Yeah, the point with Lightning is that you're building unconfirmed transactions on each other. So if… then… if one of the underlying transactions is tweaked then the transactions that follow up on that one aren't valid anymore.

Sjors Provoost Yeah, and people spend lots of time trying to find ways around that problem, because people were thinking about Lightning-like solutions quite a while, and it was just really hard to solve.

Aaron van Wirdum Yeah, so to be clear, the concrete attack in this Lightning example is that one of the parties would tweak the transaction they shared between them, send this tweaked transaction to the network, and then I guess the other party probably wouldn't even recognize that transaction, or even if he did, he couldn't use his own transaction to get his funds back. Am I saying that right?

Sjors Provoost Yeah, all the transactions that get your funds back are no longer valid. So this could be a problem, you know, in all the cheating scenarios.

Aaron van Wirdum Yeah, exactly. There's also another well-known example, which was the Mt. Gox case. And that was a mistake on the part of Mt. Gox as well. If we assume that the story we've been told about how the hack happened is really true, but the story was...

Sjors Provoost The hack or you mean one of the many hacks?

Aaron van Wirdum The big one, basically. They claimed that the big one was due to transaction malleability. And the story was that they were basically doing their internal accounting based on transaction IDs. So a customer would withdraw funds, use malleability to change the withdrawal transaction a little bit, still get the money because the transaction is still valid, but then claim, guys, I made a withdrawal, but I never received the money. Mt. Gox would take the transaction ID, look if it was in the blockchain, so that there's no transaction ID like that in the blockchain. Our customer must be right. And then resend the coins.

Sjors Provoost Yeah, but there are so many other things you have to do wrong for that particular thing to happen.

Aaron van Wirdum Yeah.

Sjors Provoost But anyway, let's… let's blame it on malleability.

Aaron van Wirdum I'm just giving an example of something that could go wrong because of malleability if, yeah, in this case, you make other mistakes as well. So that was malleability.

Sjors Provoost Yeah.

Aaron van Wirdum So that's what we want to solve, right?

Sjors Provoost Yeah, and there have been you know partial solutions to this already because it's it's a much bigger problem than just the signature I think but in either way, it turns out that it seems like a whack-a-mole game that's just really hard to solve. And the fundamental problem there seems to be that because you're pointing to something that includes a signature, it just gets too complicated. One thing SegWit does is it no longer refers to the signature because it refers to, well, the signature is put somewhere else in a transaction, in some sort of extra data.

Aaron van Wirdum To make this very clear, in case some of our listeners aren't keeping up, the thing is a transaction consists of all of the transaction data plus the signature. Formally or usually, or still the case in some transactions, the transaction data and the signature is hashed together. This gives you a string of numbers and that's the transaction ID. Because the signature can be tweaked, that means the hash, meaning the transaction ID, can also be tweaked. And you end up with basically the same transaction with a different transaction ID. And that causes all the problems we just discussed. So that's the problem we needed to solve. Somehow we need to make sure that a transaction would always result in the same transaction ID.

Sjors Provoost Yeah, so the solution there is to put the hash… sorry, put the signature in a separate place inside the transaction that, as far as old nodes are concerned, doesn't even exist. And you still refer back to other transactions by the original data. So basically the original part of the transaction, that still creates the hash. And the signature is this new data, and you do not use it to create a hash.

Aaron van Wirdum Right, so the signature ID can't be tweaked anymore because the signature isn't in there anymore.

Sjors Provoost Right, so you can still tweak the signature if you wanted to, although there's some limitations on that too. But if you tweak the signature, that's not part of the hash. And this is nice, right? So that's one thing it does, SegWit. And the other thing it does is it just, because this data goes into a place that old nodes don't care about, well, suddenly you can bypass the one megabyte block size limit without a hard fork. Because old nodes will see a block with exactly one megabyte in it, but new nodes will see more… more megabytes.

Aaron van Wirdum Yeah, blocks have a one megabyte limit. And that was… that used to be transaction data plus all the signatures, plus a little bit of metadata. And now it's basically mostly the signature data and not the signatures. And that's where the block size increase comes from. The signatures is sort of the increase.

Sjors Provoost Yep, exactly. And that's theoretically up to four megabyte, but in practice it's more like two and a half, I guess, the total size that you get for blocks.

Aaron van Wirdum Yeah. Why is that? There is some new calculation for how data is counted when it comes to the signature.

Sjors Provoost Well, yeah. So I think what happens is you take the old data and you multiply it by three or something, and then you take the new data and you add it up. So the signature is kind of discounted in a way. And that's… that's kind of an arbitrary number but at least it creates an incentive to use SegWit.

Aaron van Wirdum Right. And that's also why it's a bit more flexible now, the block size limit. If there are many... Transactions with many signatures for example multi-sig transactions then the size of the blocks could be a little bit bigger because of how it's all calculated?

Sjors Provoost Right, because with the usual old-fashioned transactions there is not much going on in terms of signatures you know, there just aren't that many signatures but you could conceive of much more complicated transactions, that have much longer signatures like in a multi-sig situation, and those are nicely discounted in SegWit, yeah.

Aaron van Wirdum Right. So how is it possible that SegWit could be deployed as a soft fork?

Sjors Provoost Okay, yeah.

Aaron van Wirdum Which means backwards compatible upgrade. So old nodes still recognize the SegWit chain as long as it has majority hash power at least.

Sjors Provoost Yeah, and they do this because this new data that we've added is not... Like it's not communicated to all nodes. So every transaction has a little piece of witness that's not communicated to all nodes. And every block has a part that is the witness that's not communicated to all nodes. So basically new nodes...

Aaron van Wirdum First of all, where is this part?

Sjors Provoost I think it's at the end of the block.

Aaron van Wirdum It's in the coinbase transaction, right?

Sjors Provoost So I think it's appended at the end of the block, but it's also referred to in the coinbase transaction. Because what you do want to do is you want to make sure that the block hash just refers to the things that are in the block. But it only refers to the things that are in a block as far as legacy nodes are concerned. So… but you don't want to tell the legacy nodes about the SegWit stuff. So what happens is there is a OP_RETURN statement in the coinbase which refers to a hash of all the witness stuff.

Aaron van Wirdum Yeah, the Coinbase, in case listeners don't know this, isn't just the company. It's also the transaction that pays the miner his rewards. So basically the first transaction in any block.

Sjors Provoost Yeah, and the transaction can just spend the money however it wants, but it has to contain at least one output with OP_RETURN in it. And that OP_RETURN must refer to the witness blocks. So all nodes just see an OP_RETURN statement and they don't care.

Aaron van Wirdum And OP_RETURN is a little bit of text.

Sjors Provoost Yeah, OP_RETURN basically means, okay, you're done verifying, ignore this, but it can be followed by text, which is then ignored.

Aaron van Wirdum Right.

Sjors Provoost Except by new nodes, which will actually check this.

Aaron van Wirdum Right.

Sjors Provoost So this allows the nodes to communicate blocks and transactions to new nodes and to old nodes, and they all agree on what's there. And the other thing, the other reason why this can be in soft fork, and that's more important for the new nodes, is… Well, you're spending, where are you sending the coins to when you're using SegWit? So you're using a special address type now. And this address type, or like on the blockchain, what you have is a scriptPubKey. That is what an output says. So an output of a transaction tells you how to spend the new transaction. It puts a constraint on it. And so this scriptPubKey with SegWit starts with a zero, or at least does now, but with Taproot, it'll start with a one. And then it's followed by the hash of a public key or the hash of a script. And new nodes know what to do with this. They see this version zero. They know, okay, this is SegWit the way we know it. And they see a public key hash, and they know, okay, whoever wants to spend this needs to actually provide the public key and a signature. But old nodes, what they see is, okay, there is this condition, which is put zero on the stack and put this random garbage on the stack that I don't know what it is. And the end result is there's something on my stack and it's not zero, and I did not fail. And so, okay, whatever, this is fine, you can spend this. So old nodes think that anybody can spend that coin, but new nodes know exactly who can spend it and who can not spend it.

Aaron van Wirdum Yeah, it's actually [unclear] anyone can spend out…

Sjors Provoost Yes.

Aaron van Wirdum So in a hypothetical situation, where there would only be all the nodes on the network, then it would also literally mean that the coins in these addresses could be spent by anyone.

Sjors Provoost Yeah, this is why the activation of Taproot was of course, you know, always exciting because yes the miners signaled, but okay, what happens?

Aaron van Wirdum Yeah, we discussed that in the last episode, of… I think or the one before that.

Sjors Provoost Well, in general, we've talked about what can go wrong with soft fork activation, and this would be one of it. And so, well, it didn't go wrong. So that's good.

Aaron van Wirdum Yeah, so the reason it didn't go wrong is because if there's a mix of all the new nodes on the network, but most miners enforce the new rules, then most miners will ensure that these coins in the Anyone Can Spend outputs from the perspective of old nodes, won't actually get spent. They'll consider blocks that spend these coins invalid, and as long as they're the majority, they'll also create the longest chain. So now new nodes are happy because all the new rules are being followed, and old nodes are happy because no rules are being broken from their perspective, and they just follow the longest chain. So everyone's still in consensus.

Sjors Provoost Yeah. And this rule I just told you about, this scriptPubKey that puts things on the stack, and as long as it's not zero, everybody's happy. It's kind of a hack. It's kind of just leveraging some ugly aspect of, you know, ancient ways that Bitcoin scripts work. But with SegWit, the first thing will be the number zero or the number one, et cetera. And this actually introduces a cleaner variant of the same principle, which is that as far as a SegWit node is concerned, if it starts with the number zero, it's going to enforce the rules. If it starts with the number one or higher, it'll consider it a… it doesn't matter, anybody can spend this. And if we get Taproot, then the new nodes will see version zero, they'll enforce the rules. They'll see version one, they'll enforce the rules. But if they see version two or higher, they will just consider it valid. And that means that moving forward, it's much easier to introduce soft forks like Taproot without having to find another hack in the old scripting system to exploit.

Aaron van Wirdum Right. So SegWit was a little bit of a hack, but it was, in that sense, a one-time hack because now we can use versioning. And every time we want to introduce a new rule for spending coins, it's going to be pretty clean and easy moving forward.

Sjors Provoost Yeah, exactly. And within Taproot, I guess there's a little out of scope for this one, but within Taproot, we have these multiple branches that can have their own condition. Those scripts also have a versioning mechanism. So there's even more versioning that can be done.

Aaron van Wirdum Right. One more question, Sjors. We mentioned that the signatures, they're included in the end of the block you mentioned.

Sjors Provoost I think they're just appended.

Aaron van Wirdum But there's a reference in the coinbase. So how are all these transactions included in one little transaction?

Sjors Provoost Well, it's called the Merkle tree.

Aaron van Wirdum Merkle tree?

Sjors Provoost Yeah.

Aaron van Wirdum This sounds exciting.

Sjors Provoost We talked about Merkle trees in an earlier episode.

Aaron van Wirdum I think we did.

Sjors Provoost Quite at length. We tried to explain them and it was possibly quite terrible, but we've done it and we're not going to do it again. But basically, it's essentially just taking a hash, but a Merkle tree is a little bit more elegant than a hash because it allows you to… like point to specific elements inside the tree, where you know, a hash will just say yes or no for everything that's in it. Could be a whole megabyte. It's correct or not. But with a Merkle tree you can say okay, I can actually prove that this specific transaction exists inside that tree at that position without having to like reveal everything else in it, and that's kind of cool. Yeah,

Aaron van Wirdum And I think it's essentially sort of a mirror of the actual transactions, right? Which are also included in the Merkle tree in the block.

Sjors Provoost Yeah, it's the same idea. So it's not rocket science.

Aaron van Wirdum So we have one Merkle tree for transactions, the regular transaction data, and then sort of a mirroring Merkle tree for all the references to the signatures in the coinbase block, right?

Sjors Provoost Yeah exactly. And I think you could generalize that to something called extension blocks, where you could add something else to transactions in the future and just refer to that in a coinbase output. And so you could, you know, increase block size through soft forks to a degree. But you can't really go super far with that because as far as the old, old nodes is concerned, there still has to be a valid transaction out there. And a valid transaction probably has to have at least an input, you know, and at least an output, even if the output says do whatever you want with this. Can't make it smaller than that. And there is still the one megabyte limit as far as these old nodes are concerned. You can't use extension blocks just to add data to transactions. You can use it to add data to transactions, but you can't use it to create an infinite number of transactions because those transactions have a minimum size, probably about 60 bytes.

Aaron van Wirdum We're going off the rails.

Sjors Provoost That's fine. All right, bring us back to the rails. I think there were some other benefits of SegWit that we wanted to mention.

Aaron van Wirdum Well, so we mentioned transaction malleability is solved, which was necessary for something like the Lightning Network. So that's why we have Lightning Network now, because we had SegWit. The other benefit we mentioned is block size limit increase.

Sjors Provoost Yeah, and I guess we had four years almost of low fees. Now they're high again.

Aaron van Wirdum Yeah, they're stacking up now. Then we had the versioning, so easier to make new upgrades. Were there more benefits than that?

Sjors Provoost Yeah, there is. There is committing to the inputs. So this is fun for hardware wallets. If you're a hardware wallet and you want to sign something, we talked about that in one of the very first episodes where we explained that if you're a hardware wallet and you want to sign a transaction, you want to look at the output amounts, you can do that. But you want to make sure that the input amounts... actually sort of add up to the same as the output amounts so that money isn't just disappearing into fees. But the only way to do that is to actually have the input transactions and look at their output amounts. And so that meant that in the old days, you would have to send all the input transactions to the hardware wallet as well, and it would have to process them, and it's kind of a lot of work, or it could be a lot of work if they're big transactions.

Aaron van Wirdum Yeah, so to be clear, this is always the case for any wallet. You always, you know, you have inputs, that's the coins you own, and then you have the outputs, that's the coins you're sending, including a change output to yourself usually. And then the difference between them, that's the fee and that's for the miner to keep.

Sjors Provoost Yeah, the fee is not actually mentioned in the transaction.

Aaron van Wirdum Yeah, exactly. There's no… there's no fee amount or anything like that in transaction. You just have to calculate it yourself. That's fine for a regular wallet because the regular wallet just knows how much all of the inputs are worth and the outputs are obvious there in the transaction. And then the difference, it's easy to calculate. But a hardware wallet is basically just signing from private keys and it doesn't necessarily know how much all the inputs are worth. So now it's, am I saying this right? Yeah, that's right. Yeah, it's sending money away, but it's actually not sure how much money it's sending and therefore a hardware wallet has the risk that it's sending 10 million coins as a fee without realizing that?

Sjors Provoost Right. The main problem there is the fee could be arbitrary. And so if somebody colludes to the miner or just wants to take your coins hostage in some weird way that's not good so what SegWit does is it commits to those inputs. So normally, a transaction you know, in the old days a transaction would… the input would just be the ID of the transaction that we just talked about with all the malleability stuff, and… Yeah, the index, basically. So a transaction has multiple outputs, so you'd say this is spending output zero of this and this transaction. And with SegWit, what's basically added to that is the amount. Actually, not just the amount, I think the entire transaction. So take the transaction and hash it, and that's what you're committing to now. And that includes the output amounts of that transaction. So now, when you're signing it, you can check it. It could still be entirely fake, by the way. You could craft a fake transaction with fake inputs and any output amount you want. But then if the hardware wallet signs it and you put it on the blockchain, well, it's not going to be valid. So that's kind of a useless cheat.

Aaron van Wirdum Yeah, we talked about that in episode two, maybe?

Sjors Provoost No, I think in the first episode.

Aaron van Wirdum Episode one?

Sjors Provoost Yeah, with the actual tornado.

Aaron van Wirdum Right. Okay, so now we have four... benefits of SegWit one of them is malleability, which was sort of the main one. I think that was the reason… it was included… it was included in the Elements sidechain of Blockstream, I think, before it even made it to Bitcoin. And I think that was the reason they had it was solving malleability.

Sjors Provoost So, I mean, yeah, it enables things like Lightning. So that's a pretty big capacity increase.

Aaron van Wirdum Exactly. So that's one. And then we have the block size increase, which is two. Then we have the versioning bits, which makes it easier to deploy future upgrades, which is three. And then four, you just mentioned, is the hardware wallet fee issue is solved.

Sjors Provoost Yeah, or at least we thought it was solved. We explained in the first episode that there's some gotchas. But yeah, those are, I think, the four main benefits. And I think there's some minor tweaks as well in there. It was a pretty big change. Compared to that, Taproot is relatively simple.

Aaron van Wirdum Yeah, so I spent a lot of time on reddit.com slash r/btc, and all I read there is that SegWit is the awfulest thing ever. How come, Sjors?

Sjors Provoost The awfulest?

Aaron van Wirdum Yes, it's horrible.

Sjors Provoost Okay, well, sorry to hear that. I don't know. I mean, I've heard more reasonable objections from non-RBTC places saying, well, it would have been slightly simpler to do it as a hard fork. But the more I look at it, the less I'm convinced of that.

Aaron van Wirdum Yeah, I guess the argument there would be that the signature hash reference… the signature hash tree is included in the coinbase and there would have been a cleaner place to put it if…

Sjors Provoost Well, you wouldn't have had to put it anywhere. If you do a hard fork, you can just add the witness data to the blocks and in the main Merkle tree. So you don't need to do anything in the OP_RETURN.

Aaron van Wirdum Right.

Sjors Provoost But the downside is you need to actually do a hard fork. And just to think through what's involved to do that is that's where all the complexity then goes and all the precedent risk. So I think it's good that this was done as a soft fork.

Aaron van Wirdum Yeah, I was saying it in jest, but I think that is probably the only argument that I've heard that even makes slight sense, that that's also being exposed. [unclear] English word…

Sjors Provoost It's been a while since

Aaron van Wirdum I read all that stuff.

Sjors Provoost I think that's one of the more serious ones. But other than that, I think the main arguments were, you know, it was a Blockstream conspiracy.

Aaron van Wirdum Sure, yeah, yeah. Of course you have all that.

Sjors Provoost Extra complexity so that Bitcoin Core can get paid more and a whole bunch…

Aaron van Wirdum Are you a conspiracy denier? Are you asleep still, Sjors?

Sjors Provoost I'm a conspiracy denier.

Aaron van Wirdum Wake up, sheeple.

Sjors Provoost I'm sorry. I will keep sleeping. I think that's it, right?

Aaron van Wirdum I think so, Sjors.

Sjors Provoost All right, then. Thank you for listening to the Van Wirdum Sjorsnado.

Aaron van Wirdum There you go.