PagerDuty Operational Reviews Documentation

Operational Reviews

The documentation on this site offers up a framework for how to run operational reviews at digital organizations. It was first put together by listening to PagerDuty's 10,000+ customers and asking them how they measure, or wished to measure, the impact of operations on business outcomes. What we learned was that getting managers together on a regular basis, and from different levels of the organization, to review metrics is a critical aspect to keeping leaders aligned on how the way they operate impacts their ability to reach their business goals. What follows is meant to be a framework for setting up Operational Reviews at your digital business, and it is the framework that PagerDuty is adapting within its own organization. For more information and a detailed background, visit our About page.

What Are Operational Reviews?#

Operational Reviews are a series of regularly scheduled activities that offer Engineering, IT, and Operations leaders the opportunity to understand the impact of their real-time digital operations on business outcomes.

Operational Reviews are metric-driven activities, preferring quantitative data ahead of qualitative. That data, along with the analysis and discussion among those involved, is meant to highlight the impact of technical service operations on key facets of a digital organization; namely people, costs to the business, and customers.

The documentation that follows here offers a guide for how to run regular Operational Reviews in your digital organization, including the impact of running operational reviews, what to review and when, who to involve, what data you’ll need, and even how to run meetings.

Not all of the guidance provided here will be applicable for every organization, and varying levels of digital operations maturity will impact how hard or easy it is to follow this guide. The intent of this documentation is not to prescribe a one-size-fits all approach to running operational reviews, but rather to provide a framework for any digital organization to adopt and make their own, and allowing them to refine, expand, and improve the review process as the organization's operational maturity increases.

Why Conduct Operational Reviews?#

Why spend the time and effort to run operational reviews? It all comes down to business impact. You likely don’t run your organization on the principles of DevOps just because everyone else does; you do it so you can deliver better business outcomes through more autonomy and better-quality work. Simply put, running a digital business means that the smallest issues or inefficiencies can have outsized impact on your business.

Business impact and outcomes aren’t just smart-sounding words—they are real things you can optimize for. Outcomes are the impacts felt by your people, the time and employee costs your business absorbs, and the impact experienced by your customers.

One way to ensure that your real-time digital operations yield positive business results rather than negative business outcomes is to regularly pause, review the impact of operations on business outcomes, and align all stakeholders on how to invest in the future for better business outcomes.

Running regular operational reviews allows you to optimize for the health of your people, the costs of running your digital business, and the experience your customers have. At the speed of digital, failing to measure and understand these impacts can deliver outsized negative outcomes before you even realize it.

Don’t just take our word for it. The data shows it’s true. Research done by PagerDuty on digital operations maturity shows that improving operations leads to measurable impact. For example, digital organizations that model highly mature behaviors acknowledge incidents 7 minutes faster, mobilize respondents 11 minutes faster, and resolve incidents a full 2 hours faster than their immature peers. At an average of 7 major incidents per month, this adds up to at least 14 hours less downtime each month. Those results have a significant impact on people, too. Those same highly mature organizations experience a 21% lower employee attrition rate among their on-call responders.

If you are convinced that digital operations have major impact on business outcomes, and you want to start optimizing by putting a process for operational reviews in place, the below documentation will help you get started.

Getting Started#

Highly mature digital organizations run a variety of reviews, and they look different at every company. However, there is a common set of reviews that the most mature organizations all tend to conduct. Each of them aligns with a specific layer of organization management, covers an intentional timeframe and information scope, and seeks to answer a short, specific list of questions. These three reviews work together to create focus and alignment across the organization on how operations impact business outcomes.

The three commonly shared reviews at digital organizations are: a weekly on-call review, a monthly service review, and a quarterly business review. The timing of these reviews is deliberate and common. It’s hard to understand the impact of large investments or initiatives until weeks or months have passed, while understanding the impact of on-call on your people requires fresh information and perspective week in and week out.

If you come across any words you don’t recognize at any point in our guides, take a look at the definitions section.