Wednesday, January 7, 2015

Bitstamp problem and warm wallets

We are publishing an exciting report on Peatio exchanger soon and I've got quite a few thoughts on how to make exchangers' architecture and wallets more secure.

Then I see this. Five. Million. Dollars. In a hot wallet.

Ok, sure it's not everything they had. It's a small part of their assets. But I'm not going to believe this hack is not a big deal for them. I bet they are a little bit upset right now.

Bitcoin exchangers must understand one simple thing: you're going to be hacked. That's the truth you have to accept and build your entire architecture around this axiom (think of Erlang's fault tolerance "let it fail") . And your business shouldn't collapse after it.

Bitstamp clearly wasn't ready to be hacked. That's the point.

The following idea might be useful for all medium-sized+ exchangers:

The warm wallet application is an application on a separate server with only one API call exposed to main application: createWithdraw. This dramatically reduces attack surface. Ideally someone should create, thoroughly audit and open source such app, so any exchange with any stack would benefit from using it.

Another thing warm wallet should do is basic calculation of your total assets: bitcoins, fiat and payment systems. Previously btc-e's private key was bruteforced, attackers "stuffed" the exchanger with fake "add funds" requests and it costed them over 4000 BTC. Before every major withdrawal it must reaudit all exchanger's assets to ensure 2+2=4.

With warm wallet system the attackers have to hack application server first. Then they have to hack the warm wallet app which, surprise, has only 1 API call. They know nothing else about the warm wallet. After hours of playing around they will steal your tiny hot wallet and your admin will patch the hole.

If you want to successfully run an exchanger you need to deal with the fact Bitcoin apps are now target #1 for cyber criminals:
  1. While blockchain is not exactly anonymous, it's nearly impossible to track the stolen money. You cannot get them back. Ever.
  2. Most apps were written by "web developers" (read "amateurs"), not enterprise-level bank engineers (i'm not stating those are any better but they at least know what transaction is).
  3. Do you know how to make a bitcoin exchanger's developer cry? Say "race condition".
  4. The result of attackers' work is money. Not passwords, not l33t deface or private data on pastebin. Just cash. Awesome!
This post makes an assumption Bitstamp didn't have a warm wallet-like system. The breach details are not public yet.

If you have any questions contact our security firm


  1. In order for the warm wallet to know if a authentication is OK, that means they would need a copy of the database from the live site.

    If my understanding is correct, the warm wallet only has one API call- Withdraw.

    But if you tell the warm wallet "withdraw 10000 btc to xyz account" it still needs to be able to check to see if 'xyz' account indeed has 10k bitcoins available. In order to do that, it needs a copy of the database for the live site that is completely up to date. Even if you had a hourly clone of the database, that would not have stopped the bitstamp hack which apparently lasted for a day.

    So if you can hack the application server and modify your user row to say it has 10k bitcoins, how can you stop the warm wallet from trusting it?

    1. I cant edit the above, so wanted to add that manual approval for withdrawls above the hot wallets balance likely wouldnt stop some hacks, especially if they did not go to ask for a huge amount of bitcoins.

      If I modify by account user row to say it has 10k bitcoins in a convincing way and only go to withdraw 500 at a time over a day or so, the undoubtedly low level employee in charge of approving this is not going to see a problem and likely approve the transaction anyways.

    2. Manual approvals will stop it.

      >If I modify by account user row to say it has 10k bitcoins in a convincing way and only go to withdraw 500 at a time over a day or so, the undoubtedly low level employee in charge of approving this is not going to see a problem and likely approve the transaction anyways.

      Good point. Let's take it to the next level - warm wallet will also be responsible for depositing money. We can add another method getAddressForUser(id) so it's easy to audit funds right on the warm wallet server.

      When application server is compromised bitcoind balance can be faked, but warm wallet is not. We just need to extract funds-related features to another service.

    3. Anyway 500*few times is not 19,000

    4. Might be wrong but if you look back at to history this was not taken in one go but in 500 times a few

  2. It would be cheaper to hire staff to manually move trades than lose 5 million

  3. First having such API wont help you at all, as by controlling the main server you control the API as well.

    Second you don't even know how large exchangers work. You can't be tied to wallet at all, what you suggest is simplified implementation of P2SH, but you can't implement it on exchange as theres literally millions transactions per day and you can't move funds constantly.

    Third believe it or not some exchanges have security protocols like real banks. You tie everything to hack ignoring inside job.

  4. Very important: have your customers deposit into cold storage.

    After Bitstamp shut down their system, money was still flowing into the compromised hot wallet.

    It seems the initial withdrawal was "only" 3100 bitcoins, probably their hot wallet size.
    Then over the course of 24 hours, another 15000+ bitcoins we're withdrawn.

    By having deposits flowing into cold storage, it is easy to stop the flow into the hot wallet.

    Your architecture will be a bit more complex, when you need to monitor addresses outside of the hot wallet, to administer customer account balances. With hierarchical determistic wallets came into existence, it became a lot more practical.

  5. Cold or warm wallet they are not a BIP standard so it's hard to say one way or another how safe they really are. I am developing a HD-BIP32 business wallet - this will enable businesses to protect their assets adding multi-sig wallets you can now manage thousands of multi-sig wallets that can be approved with a single click - here is a video of my multi-sig BIP32 wallet - I am looking for investors