On the morning UTC of May 18, 2020, the Keep team decided to trigger the 10-day emergency pause of deposits allowed by the TBTCSystem contract after roughly 48 hours on Ethereum and Bitcoin mainnet performing ramp-up tests. The team triggered this pause after finding a significant issue in the redemption flow of deposit contracts that put signer bonds for open deposits at risk of liquidation when certain types of bitcoin addresses were used in redemption.

In practice, the bonds backing open deposits (and a portion of the outstanding TBTC supply) belonged entirely to a single operator who jumped in to the system early and was testing and communicating frequently with the team during our testing as well. Shortly after the pause, the team offered a 1.005-to-1 exchange of BTC for TBTC to recover TBTC supply, resulting in recovery of 99.83% of the supply to this address. The team will be triggering a controlled redemption of open deposits to free up the remaining bonds of the single bonder responsible for backing those deposits (currently ongoing). The team will also be coordinating the removal of any remaining unused ETH in the system, though they are not at risk.

What Happened

First, some setup for those less familiar with the tBTC system. In tBTC, someone with access to BTC, a depositor, can open a tBTC deposit on the Ethereum chain. This deposit is a smart contract that interacts with 3 signers in a separate network who jointly generate and control a Bitcoin wallet, in such a way that no single signer can access the wallet. When opening the deposit, the depositor selects one of several available lot sizes, and, once the deposit has generated the wallet, sends that amount of BTC to its Bitcoin address. The depositor then submits proof to the deposit smart contract that the BTC transfer has taken place, and is able to mint the equivalent amount of TBTC, an Ethereum ERC-20 token. This allows a holder of BTC to enter the tBTC system and then use their TBTC token balance to interact with smart contracts that support it.

To properly prove transactions on Bitcoin to the Ethereum chain, a relay is employed that communicates enough data about Bitcoin blocks to an Ethereum smart contract to confirm that a Bitcoin transaction has accumulated a certain number of confirmations. This is used to ensure that a transaction (a) exists on the Bitcoin chain and (b) is sufficiently confirmed that there is reasonable certainty that a fork of the Bitcoin chain will not be able to remove it.

To ensure that the signers who control the wallet cannot illicitly transfer the BTC they jointly hold in an unauthorized way, they are required to put up a bond equivalent to 150% of the BTC value in ETH. That bond is held by the deposit smart contract until the deposit is redeemed, a process described in the next paragraph.

When a holder of TBTC on Ethereum is interested in acquiring BTC, they can reverse this transfer through a process called redemption. Redemption allows an Ethereum user, the redeemer, to pay the lot size plus a small signer fee, specify a Bitcoin address, and authorize the 3 signers to jointly produce a signature that completes a Bitcoin transaction transferring the BTC from the deposit to the specified address. This allows a holder of TBTC to exit the tBTC system back to the Bitcoin chain, and returns the signers’ bonds to their respective pools of available funds to back new deposits. The signers split the signer fee paid by the redeemer.

Incident Timeline

We consider the incident to have started when tBTC was deployed, a process completed on March 15, 2020 at 15:52 UTC with the creation of the tBTC system’s sortition pool. The deployment of the sortition pool itself did not place any funds at risk, however: the sortition pool is used to randomly select signers who have enough bond to back new deposits. Signers must opt in to using the sortition pool by placing ETH into a bonding contract. That contract then requires an additional authorization from the signer to enable use of the funds they placed into the bonding contract for the specific sortition pool used by the tBTC system. Lastly, the signer must register themselves with the sortition pool to populate the data used during deposit opening to select signers.

Over the course of the next 3 or so days, several signers made bond available and authorized the sortition pool, but only 3 signers registered themselves with the pool to be used during deposit opening.These 3 signers were controlled by a single person, who reached out after being fully set up to help the team as we tested deposits and redemptions during that time.

Deposits were broadly available via an alpha dApp at https://dapp.tbtc.network/, limited to a 0.001 BTC deposit size. The ETH bonds made available by the 3 signers that were operational also placed an upper limit on how much TBTC could be minted, since each TBTC deposit required 1.5 times its value in ETH to be bonded. The dApp was taken down briefly on the evening of May 15, 2020 while investigating a potential issue, but re-enabled early on May 16th once that issue was understood. Additionally, several members of the community set up local versions of the dApp and used them to open deposits with larger than 0.001 BTC lot sizes.

On May 18, at 2:29 UTC, the operator who controlled the 3 signers attempted to redeem a deposit they had opened and could not complete the redemption process. They reached out to a member of the team, who noticed an earlier communication from another team member, indicating that high gas prices on the Ethereum chain were causing the relay’s updates on Bitcoin state to be a few blocks behind. We communicated to the operator that this was likely the issue he was seeing, and the relay began using a higher gas price and caught up by 3:07 UTC.

At 3:13 UTC, the operator flagged that they were still having trouble redeeming, and we used a local version of the dApp to investigate the issue, at which point we observed an error from the deposit contract, “Tx sends value to wrong pubkeyhash”. This error indicated that the proof the dApp was constructing to show the Ethereum chain that redemption had completed successfully was incorrect; in particular, that the proof did not successfully prove the transaction sent the deposit’s BTC holdings to the correct redeemer address.

After 30 minutes of investigation, we developed a suspicion that the issue was not necessarily in the dApp’s client proof, but rather with the particular Bitcoin address used for redemption and its provability on the Ethereum chain. Before verifying this suspicion, Matt Luongo was notified to ensure 2 of the 3 people needed to trigger an emergency pause were on standby in case it proved necessary. Then, we investigated the implications of the issue being in the smart contract and determined there was the potential for signer bond danger. Because signer bonds can only be seized after 6 hours without a redemption proof, the decision was made to continue investigating and confirm the contract issue before taking further action.

At 4:43 UTC, Matt notified James Prestwich, who wrote much of the first iteration of tBTC contracts, has extensive Bitcoin experience, and is the author of the bitcoin-spv and relay libraries used to verify Bitcoin transactions on the Ethereum chain, to request a corroboration of the finding. James corroborated the finding at 5:02 UTC, at which point we immediately began redirecting the hosted dApp URL to the tBTC homepage to prevent further new deposits being opened.

At 5:18 UTC, after confirming the issue and confirming that it was not fixable outside the contracts, the decision was made to trigger the single-use 10-day emergency pause available in the tBTC system contract. This function prevents new deposits from being opened for 10 days, but does not affect the functionality of any already-open deposits. For security, the process to trigger any tBTC system updates requires 2 of 3 members of a technical wallet team to manually craft an Ethereum transaction, then use air gapped machines to sign the transaction’s information, and finally submit the transaction and the signatures to the Ethereum chain. This process was completed at 5:45 UTC.

The next morning (Eastern time), we realized that, though the hosted dApp’s landing page was redirecting to the tBTC homepage, other pages on the hosted dApp, such as specific deposit pages, were not. Rather than risk any existing deposits triggering the redemption bug unintentionally, the remainder of the hosted dApp’s pages were redirected to the tBTC homepage at 14:11 UTC.

Technical Issue Description

The issue itself was rooted in the process that proves a redemption transaction has in fact gone through on the Bitcoin chain. Under normal circumstances, signers who provide a valid signature for a Bitcoin transaction might have their bonds immediately released, leaving the redeemer in charge of broadcasting that transaction on the Bitcoin chain. If the tBTC system released the signers from their economic obligations at this stage, however, the signers would have the opportunity to produce a different signature, for a transaction sending the funds to an address other than the redeemer’s, and broadcast it before the redeemer has a chance to broadcast their own transaction. As such, the tBTC system only releases signer bonds once the signers have produced a valid signature and proof that the transaction was accepted on the Bitcoin chain.

The proof that a redemption transaction has been sufficiently confirmed on the Bitcoin chain has a few sanity checks applied; one of these is verification that the Bitcoin transaction sends the funds the signers jointly control to the requested redemption address. These checks are performed by the redemptionTransactionChecks function:

The error we observed was in the last check, “Tx sends value to wrong pubkeyhash”:

Bitcoin has several types of output scripts. The most common kinds — pay to pubkeyhash (p2pkh), pay to scripthash (p2sh), pay to witness pubkeyhash (p2wpkh), and pay to witness scripthash (p2wsh) — can be encoded as addresses. We’ll refer to these as standard output types. The address represents a 20- or 32-byte hash, a checksum, and information about the type of output script. The type information is used to insert the hash into a standard template. This creates the corresponding output script.

Output scripts vary in length. As a result all output scripts are prefixed with 1 byte that encodes the length of the script in bytes. For example, a standard p2pkh output script will be 25 bytes long, or 26 counting its length prefix. The value of an output is represented as an 8-byte little-endian integer, serialized immediately before the output script. Thus a standard output is between (8 + 1 + 22 =) 31 and (8 + 1 + 34 =) 43 bytes long.

BTCUtils.extractHash() extracts the hash from a standard output. It does this by inspecting the output script prefix and suffix to determine the location of the hash. If the output script is non-standard, it returns an empty bytearray.

Already we can see some patterns. Output bytes 0–7 are always the value. All legacy types have postfixes, while all witness types do not. All types except p2pkh will begin the hash on the (8 + 1 + 2 =) 10th byte of the output, which is at index:

This expression takes bytes 8, 9, and 10, and concatenates the hash. For witness types, byte 8 is the length-prefix, while bytes 9 and 10 are the template prefix, so it is easy to see that concatenating them to the hash will produce the (length-prefixed) output script. However, for p2sh addresses, this expression will not append the template postfix. For p2pkh addresses, it will extract only 2 bytes of the prefix, and will (again) not append the postfix. This means that the expression modifies legacy output scripts, and will never output a valid legacy output script.

This code is equivalent to the deployed code. After accidentally modifying legacy scripts, it then compares them to unmodified legacy output scripts. Whenever _d.redeemerOutputScript is a legacy script, this equality will always fail, and the transaction will always be reverted.

Neither the redeemer nor depositor is harmed by this bug — that is, the user’s funds are safe. In fact, because this code validates the redemption proof, it is only run after the redeemer has received BTC.

However, because the system cannot verify that redemption succeeded, the signer bonds become available for seizure as if redemption had failed. In particular, if a redemption transaction has not been proven to have occurred to the deposit contract 6 hours after the signers have provided the signature for that transaction, the redeemer can notify the contract that the redemption proof has timed out.

Notification of redemption proof timeout is treated as a signer abort, meaning a situation in which signers did not fulfill the system’s requirements, but which the system does not treat as malicious intent. During redemption, this means the system cannot verify that the redeemer received their funds. In this scenario, the system seizes the signer’s bonds, and gives the redeemer the full bond value as compensation. This recourse is taken to prevent a scenario where the signers produce a signature over the requested redemption transaction, but also collude to produce a signature over a different transaction, and then race to confirm their transaction before the correct one is confirmed.

In normal cases, a signer producing a bad transaction can be punished by proving fraud, showing that they signed data that they were not authorized to sign. The effect of this proof is the same, seizing signer bonds and giving the full amount to the redeemer. However, unless the system validates the redemption transaction’s output script, the signers could simply prove their malicious transaction instead. As such, the redemption proof has to check the output script, requiring both validation at redemption request time and an adjustment at redemption proof time.

How This Code Landed

The initial design of tBTC restricted redemptions to p2wpkh addresses, and enforced this restriction in the redemption process. In early February, our Head of Engineering, Antonio, committed a change that loosened redemption transactions to allow not only p2wpkh output scripts, but also any other output scripts. During redemption, this was intended to allow deposits to accept an arbitrary Bitcoin output script, to give redeemers the flexibility to accept funds in whatever wallet they prefer to use. The code in question is a holdover from the previous validation system. Other artifacts of the p2wpkh verification code (e.g. the no-longer-necessary abi.encodePacked) are present.

The commit message from above notes that ‘The result passes all current tests, though there are no tests for non-p2wpkh output scripts in the repo yet.’ This did not change over the coming months, though Bitcoin testnet addresses generated by Green wallet, which are pay-to-scripthash (p2sh) addresses, were tested repeatedly on testnets. However, due to a bug in the redemption dApp in use at the time, the proof step of the redemption flow never occurred. These p2sh addresses would have failed validation had the proof step occurred, but reliance on the dApp’s display of a completed state meant the team assumed the redemption had completed successfully, when it in fact had not.

Additional Observations

The issue above was not the only problem with the redemption code. In fact, even if the proof code had been correct, a malicious redeemer could potentially still have specified an output script that yielded an invalid Bitcoin transaction due to the consensus rules regarding  OP_VERIF  and  OP_VERNOTIF . This would force a situation where the transaction would never be included in a Bitcoin block. In such a scenario, being able to confirm the transaction’s output script is irrelevant, and the redeemer would be able to guarantee receipt of the bond (while leaving the BTC with the signers). As such, in addition to the incorrect validation  once redemption had occurred , there was missing validation  when redemption was requested . As a consequence, future versions must support only standard address types.

Of note, if a redeemer specified an output script that would result in an invalid Bitcoin transaction, though they would be able to seize the signer bonds, the signers would remain in control of the BTC deposit. Thus, the overall value loss would be less than a valid transaction that can’t be proven to the deposit contract. Still, the overcollateralization of signer bonds means that the malicious redeemer would still have an overall gain from this scenario.

The bugged verification code was also present in a dead codepath that has been removed in the bugfix PR.

What Went Wrong

Ultimately, several issues were at play here:

• First and foremost, we failed to capture an action item to add further test vectors outside of the original commit message. As a result, we missed an opportunity to catch the issue during development.
• During manual dApp-based QA, we did not verify that a successful redemption in the UI resulted in a closed deposit on-chain. As a result, we missed an opportunity to catch the issue during manual QA.
• We did not fully consider input validation at the entry point to redemption. This is one of relatively few completely user-controlled pieces of data in the system, and as such should have been top of mind for input validation.
• We did not spend enough time generating Bitcoin test vectors for unit tests. In particular, fuzz testing of the redemption flow would have potentially caught both the requested output script validation and the transaction proof issues.

Next Steps

In addition to the existing actions to redeem outstanding deposits in the running system and ensure the operator who was bonding ETH at the time receives their bonds back, we have already taken several actions, and will be taking several more, as we look to relaunch the tBTC system.

What We Have Done

• James Prestwich has opened a PR with a proposed fix on GitHub. We’ll be adding test vectors before merging over the coming days.
• We have adjusted the scope of the already-planned Trail of Bits audit that started on Monday from an aggressive focus on the Go clients in the network to instead split time between the Go clients and the smart contracts in the system.
• We have communicated the issue and fix to both our previous auditors, ConsenSys Diligence, and our current auditors, Trail of Bits, for confirmation and further feedback.
• We have retrieved 99.83% of the TBTC supply by offering a 1.005 BTC-to-1 TBTC purchase of outstanding TBTC. We will be using the secured TBTC to redeem open deposits and free up bonded ETH.

What We Will Be Doing

• We will be clarifying and improving our processes for capturing follow-up work in cases where pull requests are merged with future changes still pending.
• We will be working across the team to improve our processes for blocking pull request merges for cases where user-controlled input is used in the system and sufficient testing is not present.
• We will be importing the failed redemptions as test vectors into our test suite.
• We will be generating additional test vectors based on various known address types for our test suite.
• We will be exploring the feasibility of finding or building a Bitcoin transaction fuzzing tool.
• We will be working with Trail of Bits to expand and automate more integration and system tests for tBTC, as well as to add fuzz and property testing to various components of the system that currently only have simpler unit tests.
• We will be working with Trail of Bits to identify any additional areas of the system that may merit additional scrutiny.

What’s Next

In addition to the technical and process changes we’ll be making, in the coming days we will also announce how we plan on approaching a redeploy of the tBTC system, and how that will impact existing plans around the KEEP token distribution stakedrop and Playing for Keeps.

We’re looking forward to showing the world a stronger, more secure Bitcoin on Ethereum.