I thought jellicle's advice to Zhou Tong was insightful 248 days ago. Time has proven its wisdom.

http://news.ycombinator.com/item?id=2973803

-- systems that work with money are attacked hard and often, by intelligent skilled people

-- in fact some of the people who attack your system are likely to be both more skilled and more intelligent than you are

-- systems that work with money that fail, fail spectacularly ("What do you mean someone withdrew $8 million last night?")

-- banking websites, Paypal, etc. are all like icebergs - you don't see 9/10ths of the things they've done to prevent spectacular failure

-- spectacular failure is your destiny if you don't work very hard to prevent it

-- spectacular failure may be your destiny even if you do work very hard to prevent it

You should plan accordingly.

EDIT: panarky has now updated his post above to make it clear that the advice was to Zhou Tong.

I don't wish to link the Bitcoinica fiasco with my leave because I sold Bitcoinica a few months ago. And I lost my control in January. Basically everything didn't go as I planned. (I'm VERY conservative about wallet security.)

I wanted to build an independent exchange system to replace hedging, so that most funds can stay offline. But the new owner didn't like the idea.

I know it failed like predicted, but my involvement in the failure is highly limited. I sold the site for the same reasons: What if someday I'm hacked, or caught? (The valuation (P/E) was less than 1!)

I'm a web developer, not a security expert. I know how to protect the API keys but I'm not good at encrypting a wallet.dat. That's all I thought since day one.

By comparison, these bitcoin folks seem quite content to store what is essentially money on run-of-the-mill servers in run-of-the-mill data centres. Linode? Rackspace? Are you people fucking serious? It keeps happening, and I keep wondering why any thinking person would trust plain old data centre security staff with their money like this.

Seriously, the banking industry is hardly a paragon of security. Many startups give you better ways to protect your cat pictures than the average bank gives you to protect your money.

It's all about the cost of doing vs. the cost of fixing it and who ends up paying.

    > The thief stole from us not you.
    > All withdrawal requests will be honored.
    > http://www.bitcoinica.com/

Instead we get online banking with password restrictions like: 1. 8 to 12 chars 2. No non-alphanumerics.

Trust me, lots of payment data goes through secure servers at Rackspace.

/Not a rackspace employee, but I know

Might I suggest a post-mortem on the various security problems you dealt with, for those of us who weren't following closely? I'm sure you learned a lot, even if what not to do.

I don't think many programmers, especially web developers in the consumer startup scene, are faced with such security pressures as Bitcoinica was, so it would probably be very useful for many to read a behind-the-scenes accounting of that.

All my 3-year Rails experience was sufficient to prevent the most direct web attacks: Injection, CSRF, XSS, etc.

The rest is simple, keep wallet.dat securely. It's just everything the hacker wants.

So, the experience isn't very applicable to other startups, I think.

The recent hack is not my fault. It just destroyed the only reason that I stay in Bitcoinica.

Trying to run a financial site without the chops to do it was seriously wrong. Of course, it was wrong of users not to do more due diligence on the service, its record, the team (you, I guess), etc - after all, it was (was) their money.

The recent hack is not my fault.

You've got a lot to learn.

He was also warned when he announced Bitcoinica on HN, yet he chose to ignore those warnings.

However, I really did everything possible to prevent security attacks. I failed at trusting other partners too much - especially Linode. (You can say that the wallet wasn't securely stored, which could be my fault, but the hacker possessed the ability to log in to every single Linode server!) The recent email leak is because I added a guy who is responsible for security into our mailing list that was used for password resets.

I would say that almost none of the comments in the original post provided the right warning or predicted the right cause of failure.

Again, they are useful (and the described things could happen instead), but they are irrelevant.

Any business can fail because of various reasons, and predicting something will eventually fail will always turn out right, just like predicting a baby will die one day. I appreciate the warnings because I didn't make any mistake they mentioned - no SQL injection, no mass assignments, no unauthorized back-end access, no CSRF/XSS, no firewall break-in (the hacker got the database from the server backups, not from the live database server). I think I've handled everything that I'm able to handle well.

The warnings I referred to were the ones saying that it is guaranteed that you will be hacked. It was never a question of if it would happen - it was always a question of when. Thats simply the facts of running a service dealing with large amounts of money. For that I think you were ill-equipped, despite that you knew that this would happen. Now, I admit that I think you handled the fallout quite well, but I don't know if you were as prepared for it as you should have been. From my very distant view, it looks like you weren't, but I don't have all the facts, so...

I would say that almost none of the comments in the original post provided the right warning or predicted the right cause of failure.

I'm assuming this is where experience would have saved you (or if not prevented a hack, reduced the damage done). Its difficult to cover all angles at the best of times, so its doubly so when you are inexperienced. I'm not saying that the problems would definitely have been prevented and I think perhaps I'm being unfairly critical. If your service had been dealing with anything other than money, I honestly would have no problem at all with anything you did - like I said, I think you have skill and you did handle it well after problems occurred. Its just that if I entrust my money with someone, I expect that they are well prepared for anything that could go wrong.

I think I've handled everything that I'm able to handle well.

I agree, you probably have, but thats where the problem that I see is (or was, I guess). Its not the things that you're able to handle that I'm worried about - its the things that you're not able to handle.

So with that said, let me close by saying that I'm glad you came out of this alive (ie not bankrupt - I hope!) and I wish you the best of luck in your future endeavors. I imagine you learned a hell of a lot running Bitcoinica.

I have officially left the Bitcoin economy so my next project will not be anything financial related. The most important thing that I have learned is exactly what you said - "Its not the things that you're able to handle that I'm worried about - its the things that you're not able to handle."

Also, trust is another issue. Almost the whole Bitcoinica system fell under the hacker's hands, except for one part - AML verification documents (customers' passports and other extremely confidential information). This is because this part of system is still solely under my control - not even the new owner could access the information. I didn't trust anyone with that data, and it turned out to be the most secure system after all.

I should probably identify my strengths and weaknesses carefully next time.

His forthrightness with the incidents, covering of customer losses, etc. has made me respect him.

Without money I can finally rationally re-think my destiny.

I hope you are successful and that you have banked well in the Bitcoinica deal.

Congratulations on your insights.

Why not make it easier for people to use bitcoins in day to day transactions? It feels to me like more effort in the bit coin community is going into making financial products for finance people than financial products for normal people doing normal things. Solutions such as those that make bitcoin transactions from one cell phone to another cell phone would be a valuable addition to the community.

We're seeking 200k BTC for a self-green powered mining farm at our own DC up here in Maine, so we profit from the mining farm itself, having nearly no cooling needs for the DC, selling power generated back to the grid, and also renting excess space to other companies who want out of the way DC space in a quiet part of the country.

Why would someone downvote for asking for clarification..?