When Systems Fail, People Get Blamed

Introduction

When things go wrong, the immediate question is usually

Who's at fault?

However, instead of throwing the blame around, we should examine the system.
Why did it allow things to go wrong?

Blame Game

When things go horribly wrong, people usually try to deflect the responsibility and assign the blame to someone else, especially if what went wrong is affecting several people. Take the following scenario, for example.

You get called at 3 in the morning. Production is down because of a fix that was deployed recently.

• You did the fix yourself and tested it. It was fine.

• QA team was responsible for testing and verifying the fix.

• The business team performed user acceptance testing and approved the release.

• DevOps team was responsible for the deployment and verification of production stability.

What would be your first response to such a scenario? Be honest.
I will not assume to know the answer for this imaginary scenario, but something went horribly wrong.
So, who should get the blame then?

You did the fix, but it went through multiple teams before it went live.
QA team tested the fix, but it went through other teams before it went live.
The business team verified the fix, but it went to DevOps before it went live.
DevOps team deployed the fix, but they followed the deployment instructions.

So, who's really at fault here?

Analyze

To be able to determine the root cause of an issue, you need to analyze the process.

The analysis phase is not meant to find the person responsible or throw the blame on a certain team.
This phase is meant to understand the full process of developing a product, a feature, or fixing an issue.

Business Requirements

The business requirements are usually the first drivers of a product or feature. When requirements are unclear or not analyzed correctly, they can cause issues in other phases.

• Define the business requirements in a clear and correct way, based on the customer and business needs.

• Review the requirements with the technical team to understand the limitations of the current system, if any.

• If a future phase requires redesigning a significant portion of the current implementation, it may indicate that the original scope was underestimated.

• Communicate proper timelines with stakeholders. Estimating how long a feature or product will take should be discussed with the technical team first.

Do some of these points sound harsh? maybe. But if you want to enhance the process, you need to point to the root cause of an issue, and sometimes it can be hard to say the right thing to push for a better process.

Does that mean the issue is from the business team? No.

Technical Requirements

Defining the technical requirements rests solely on the technical team. During the technical requirements phase, which usually comes after the business requirements, the following should be considered and observed.

• Do the business requirements align with the current system capability?

• What are the limitations of the current system that could cause an issue?

• What are the technologies, tools, and strategies used to develop the new product or feature?

• What is the timeline required to develop the new product or feature?

• What are the requirements that should be implemented?

• Is the technical team well equipped to develop the new product or feature? Or do they require support?

Technical requirements should be clear and understandable. It should not be only the responsibility of the person developing the new product or feature to define these requirements.

When the technical requirements are not clear, there's a strong possibility of wrong implementation and issues that could reach production and cause an incident.

Does that mean the issue is from the technical team? No.

Testing Strategy

The testing strategy is one of the most important phases in the development cycle. It's used to make sure nothing breaks in the production environment. However, something to keep in mind is that there's no product without issues. You cannot possibly account for all possibilities, and you can't prevent issues entirely.

But you can minimize the number of critical issues if your process is strong enough.

• Define proper and real-world test cases. The test cases should simulate the behavior of real users as much as possible.

• Ensure the test coverage is enough for the new product or feature.

• Account for edge cases: what if an API was not working? What if a specific service was not working?

• Regression testing must be performed before every release, even if the change was small.

• Understand the business and technical requirements before starting with testing cycles.

The testing phase is an essential part and a safeguard against issues. It's used to ensure the product or feature is ready to be used by the customers. Rushing this phase to meet a deadline often leads to far more work after the release than the time it takes to test it properly.

Testing the product is the responsibility of the testing team, so they must be at fault, right? No.

Deployment Strategy

The deployment strategy is an essential part of the development cycle. However, by the time a release reaches this phase, the opportunities to prevent issues are much more limited. But still possible.

• Implement automated testing in the deployment pipeline.

• Implement monitoring and use analytic tools to catch issues early.

• Implement a notification system to alert you once an incident happens in production so you can act quickly.

A good deployment strategy is more than simply deploying the application. It should include proper visibility to detect issues quickly and enough safeguards to recover before users are significantly affected.

Does that mean the issue is from the DevOps team since they are responsible for the deployment and monitoring? No.

Blame

Blame focuses mainly on who made the mistake, not on how it happened.
Blaming the person who made the mistake and throwing everything on their shoulders creates an environment that is filled with:

• Fear.

• Defensiveness.

• Hidden mistakes.

This environment will not avoid issues. It will create more.
People will start hiding their mistakes out of fear, making future incidents even harder to prevent.
You cannot improve a process if people are afraid to report mistakes or discuss them openly.

Accountability

If no single team is entirely responsible for the incident, does that mean there's no accountability? It doesn't.

Accountability must still exist, but it should focus on ownership, learning, and continuous improvement rather than punishment. It also means owning the solution, not just the mistake.

Reviewing the incident is an important part of accountability. It helps the team learn from mistakes and improve the process.

But if accountability becomes a trial, people will start hiding mistakes instead of learning from them.

Enhancements

Every incident should leave the system stronger than it was before.
Implementing enhancements to the system and process can be done in different ways, depending on the organization, resources, and team structure.

• Implement or improve monitoring to be able to understand the cause of an issue easily.

• Implement a rollback plan and feature flags to resolve the issue once it's detected.

• Implement automated testing such as UI and integration testing to catch issues early.

• Improve the release strategy to avoid production incidents.

• Define proper guidelines for requirements to ensure nothing is missed during the early phases.

Post-incident review is important to understand the issue, the root cause, and how it can be prevented in the future.

Finally, implement deployment checks and deploy the new feature or product in a pre-production environment to verify everything is working as expected.

Conclusion

The developer may have introduced the issue, but the system allowed it to pass through requirements, design, testing, release, and monitoring without being detected.

The question should not be "Who caused the incident?" but rather, "What allowed one mistake to become a production incident, and how do we prevent it from happening again?"