Anatomy of an Outage
ยงMarch 10, 2025
How do you know that people love your application?
When it's down for thirty minutes, you get a flood of emails asking what happened.
Mission Critical Software, Baby!
Last week, multiple users reported they couldn't access our application through Azure DevOps, though interestingly, GitHub and GitLab integrations remained unaffected. Which was a bit of a puzzle, but it could also be that because DevOps is some 40% of users, the Gits haven't caught up yet.
This was a challenge to debug because:
- It was an unfamiliar codebase. Even though we acquired GZ three months ago, I still haven't been able to completely go through it as other things have taken precedence.
- It was an unfamiliar language. ISS.NET deployment is very different from Heroku + Python!
- There was no evidence of recent code changes being the issue.
Within an hour, I had a couple calls with affected uers so I could witness the problem. Users couldn't refresh their authentication tokens. Our system is set up for a refresh every forty-five minutes. But the process was failing. When users tried to manually reset their authentication, they were met with error pages - effectively locked out of the system entirely. We had login credentials to Azure DevOps, AWS, and the remote server. But since these were systems that were only used a couple times before, I had no idea what I was looking for. Within a few more hours, I called in the Big Guns (jmduke). Justin found the root cause. Our ability to give users Azure tokens had expired, and was therefore revoked. 
The system had been silently carrying on until it hit this expiration date, at which point everything came crashing down. The solution involved generating a new token and updating our encrypted vault on the server. After a half-day outage, we were back online. We will need to refresh this token in the future, but I'm working on a way to programmically do this. What should we change going forward?
- Documentation of Critical Infrastructure : During or after an acquisition, we should be documenting the critical infrastructure points extensively. This takes time but could save us in the long run. This should be done in conjunction with the previous owner, if possible.
- Is it worth it to move things to our preferred infrastructure? On the priority of TSC things, is this in the top five? If we had unlimited capacity, this would be a no-brainer. But it wouldn't be trivial to move GZ if we wanted. My contention would be moving things to the preferred architecture would allow for faster debugging and development.