It was a late night in 2012, phone blowing up with texts. "The wheels have come off the wagon!" My business partner was losing it while I was at the gym. Rushed home, jumped on the computer, and found the problem. I had configured an AWS LoadBalancer wrong. It was spinning up new servers for every single user instead of scaling based on traffic. No sticky sessions either.
Big AWS bill, yeah. But that wasn't the real problem.
Our infrastructure was backwards. User data and critical files were stored locally first, then queued to transfer to S3. With instances spinning up and down constantly, those files just vanished. Gone. A flaw I hadn't wanted to face was now staring me down. Would've been catastrophic at scale.
Spent three weeks rebuilding everything to be stateless. Fixed it.
Then 2014. We demoed our MVP to major decision-makers. Actually it was an MMMMMVP. Every click we made: orange screen of death. They signed anyway. They believed in what we were solving, broken demo and all. That same customer still uses the product today.
I guess the point is simple. I screwed up. Multiple times. The AWS thing could've killed us. The demo should've been embarrassing. But we moved fast, fixed what was broken, and kept going. That's it. No magic formula.
I've screwed up plenty. The difference was fixing it fast and not pretending it didn't happen.