DevOps Foundations: System Level Thinking

Welcome back to another week of DevOps Insights!

This week I am going to dive into a not-so-popular skill that is critical for DevOps engineers and software engineers in general: system-level thinking.

Software isn't just about the lines of code. It's a complex ecosystem of interconnected components. Think about it: your backend API, networking, security configurations, and infrastructure resources—it's all part of the puzzle. System-level thinking is about grasping how these intricate pieces fit together seamlessly.

It’s about understanding that your backend API needs to have ingress traffic on port 8080, so you need to ensure that is open on your security group. It’s about being aware that your email service needs to connect to your message broker to publish and subscribe to topics. It’s about understanding that a build server needs to authenticate with a bastion host to make updates to a server within a private VPC. There’s an infinite number of examples, but I think you get the gist.

Useful and valuable software isn’t about a single service running code. It’s code that connects multiple services that work together cohesively to produce results for end users.

Being able to harmoniously configure and allow components of the system to interact with one another is a skill that sets good DevOps engineers apart from the great ones.

Key Principles

System-level thinking consists of several core principles and concepts. One key concept is understanding that the system is greater than the sum of its parts. Multiple components exist in a software system at any given time. It’s important to consider how those components interact and influence one another. A single component isn’t that useful by itself. For example, a UI is pretty useless without a backend service to supply data to it.

Another principle in system-level thinking is fast feedback loops. It’s crucial to have fast feedback loops on how those components are working together. It helps identify bottlenecks or even disconnections between services. We’ll talk about this in just a moment.

Lastly, taking a broader problem-solving approach is to any new problems in the system. The identification of root causes instead of surface-level symptoms must be done to ensure that we are not masquerading as a deeper system-level issue.

In DevOps, we have continuous integration and continuous deployment, but we also have continuous improvement. Continuous improvement of the system and evolving it to be the best it can be.

Learning System Level Design

So, how do you cultivate this skill? It starts with understanding your system's architecture. If your system does have architecture diagrams, study them, understand them deeply. If they don’t exist, create them. Clarity around the system architecture empowers you to make informed decisions, detect flaws, and enhance resiliency. Additionally, it enables you to identify bottlenecks and prepare for future challenges, making your approach to software development proactive.

Being aware of your architecture will not only help you become more of a system-level thinker but also will help you realize potential flaws in the system. This clear knowledge of the system empowers your team to make informed decisions, prioritize tasks effectively, and allocate resources where they matter most. It’s a proactive way of approaching software development.

Also, understanding the system as a whole can help you better understand where software bottlenecks exist and where resiliency may be a problem now or in the future.

Tools of the Trade

There’s a massive plethora of tools available at your disposal to make life easier. I’d prefer to talk about the “groups” of tools that you can use to make life easier.

Automation Tools

Using automation tools such as Terraform, CloudFormation, Azure Resource Manager, and other infrastructure as code tools is great for multiple reasons. Firstly, it allows you to create your infrastructure in a repeatable manner. Secondly, you can identify what resources your components need to run since they are written as code. If there’s no external deviation, it’s extremely easy to piece together an architecture diagram of how things work from the infrastructure as code.

Orchestration

Orchestration tools such as Docker Compose and Kubernetes are great for simplifying your workload provisioning. It allows you to connect your services through it’s networking principles. They allow you to scale up and down services seamlessly without extensive configuration.

Monitoring

Monitoring is critical in the DevOps world. One of the main tenets is fast feedback loops. This doesn’t apply to development, but also in production environments. If something happens to a service, whether that be high response time, disconnections, or errors, you want to know about them as fast as possible. Ideally, these monitoring tools should be connected to the next class of tools: collaboration tools.

Collaboration

Most companies will have collaboration tools available at all times, such as Slack, JIRA, teams, and many more. If you want to monitor a specific component in your system, the information about that component needs to be tracked and reported somewhere. For outages and errors, I’d recommend your messaging system whether that be Slack or Teams. Create a dedicated channel and report errors there for quick responses. It’s also important that when an incident occurs, a ticket is created to investigate the root cause and create a post-mortem in the next group of tools.

Documentation

Nobody enjoys writing or reading documentation, but it is crucial for a high-functioning software engineering organization. Documenting how system components work, their connections to external services, and disaster recovery steps are common items to document, but not an exhaustive list. Also, it’s great to have an area dedicated to post-mortems that is reviewed on a regular cadence, quarterly, or biannually is great. This review will give the team a better understanding of why certain changes were put into place and can also spark improvement ideas.

Conclusion

In conclusion, mastering system-level thinking is your ticket to success as a DevOps engineer. In a field as complex as software engineering, understanding the bigger picture and orchestrating its various elements is what sets great engineers apart. So, dive into your system, embrace the complexity, and become a true system-level thinker!

Software engineering is complex. Systems are not the easiest to manage. Learning how to be a system-level thinker will only help you succeed in your career as a DevOps engineer.