My First Post Mortem

Report of my first serious incident as a software engineer

TLDR: I had my first ever major incident as a software engineer soon after joining a company. In this article i share, how it felt & what i've learned from it

They say, that there is first time for everything. So, today was my first time, causing a serious incident to a production grade software system. I know, the term: "serious" sounds a bit vague, but trust me, the whole company felt the turbulence. it was a real deal. The thing that makes this incident much worse for me, is that i've just joined the company, it was kind of like my 1 month anniversary gift to the company.

Take this fact, add on top of it average person's overthinking "abilities" and i hope you'll get an idea of what kind of a day i've had. Even though i'm worn out from stress and non stop thoughts about how i could have avoided it with a bit more focus & other imposter syndrome thoughts, i still want to dive right into creating my first post mortem document. All this is just to understand clearly what mistakes i made, to reflect on them & what can be learned.

For those of you, interested in what i mean by post mortem. I generally mean a writing, where one discusses own career & personal life failures. Not for a blame-game but to extract as much learnings from it as possible. The big idea behind it is, to write down in details all the moments when one is at the lowest point in life, try to retrospect on a deeper level about it & share it to public. With this, embracing all kinds of failures, learning to treat them as first class citizens of one's daily life & learning to live with it.

Disclaimer: As a first time "Post Mortem" club member, expect to read biased feelings & thoughts that i've had during this process, what i learned ( or am learning ) from it & details of how stressful it was for me. Don't expect to hear advices about how to deal with incidents or how to write post mortem-s, as i'm the last person on planet to talk about that.

To get a bigger picture of the situation, i'll share a cause of the incident. It was a (very) small software bug(Small in a way, that it could've been avoided easily) that had a big negative impact on the whole system. When you are a newbie in the company, you expect your reputation, skills & confidence to increase. For this kind of improvement you need a margin for error, as you got to be able to test with lots of things. Usually this margin is created by a set of people who share your responsibilities. This is exactly what i had. Having 2 other reviewers of my contributions, made me forget that there always is place for human error. The problem slipped right through all 3 of us. Even though my manager handled this situation in a very strategic manner, solving the problem ASAP & communicating the information to me in a non-blaming manner i still felt an enormous stress.

This was kind of my worst nightmare becoming real. Imagine getting a job that you've been trying to land for quite some time & in your first month messing in such a way. This fact automatically initiated a burst of negative thoughts like a domino-effect, the stress increased & feeling of shame with it.

After letting it sink for a while, the time for reflection has arrived & these are the 3 main learnings i got from this experience.

Key Learnings

• Making mistakes is inevitable, treat it like a common case instead of an outlier
  Instead of optimizing your life to not make any mistakes, treating failures as outliers & mourning over them. Assume that failing at something is a common case, embrace it & focus on handling them effectively
• Learn to emotionally detach from an incident, this is the only way you can learn anything from it The only way to actually reflect & grow, is not to be overwhelmed by the emotions & stress about making a mistake or failing. Learn to control your emotions and not let it flourish. You need to be clear headed and calm to go through the mistake in details like a detective
• Don't let any incident(no matter how severe) seed any kind of fear or self doubt that will mess with you in the future The fact that you made a mistake, doesn't mean that you lack some natural abilities & that you should generalize it on whole other aspects & that you are flawed at your core. Don't let the fear of not being good enough take over you. You are good enough, even more then good enough, you are unique

The learnings above might sound easy to master, but i'm sure, that i'll need to spend a lot more time & make a lot more mistakes to actualize these learnings.

This way my life's hardest part is still ahead of me.