Turning continuous integration and continuous delivery (CI/CD) pipelines into emissions‑controlled workloads is a strategy businesses adopt to reduce the carbon footprint of their pipelines. This article examines how a Carbon-Aware DevOps strategy can help optimize CI/CD pipelines for sustainability by reducing their carbon footprint while also decreasing costs and meeting the increasing demand for innovative, eco-friendly solutions.
Understanding the Problem
Today’s organizations take advantage of CI/CD pipelines to streamline and accelerate their software delivery. However, these CI/CD pipelines can become a hindrance to sustainability because of the resources required to create, build, test and ultimately deploy software into production.
Typically, CI/CD pipelines run tens of thousands of automated builds, tests and deployments daily, thereby consuming significant compute and energy resources. If you don’t take the steps necessary to reduce the emissions generated by these software applications as they scale, they can seriously undermine your organization’s sustainability initiatives over time.
What is Carbon-Aware DevOps?
Carbon-Aware DevOps is the practice of implementing and creating CI/CD pipelines in an organization to reduce carbon emissions, balance sustainability objectives and forecast carbon consumption in real-time. It is an evolution of traditional DevOps that includes the ability to manage CI/CD pipelines based on the amount of carbon they produce, including carbon-aware scheduling, telemetry and intelligent orchestration of CI/CD workflows.
Carbon-Aware DevOps and Green DevOps are separate but complementary approaches that help reduce the environmental impact of software. Green DevOps aims to reduce total energy consumption through architecture and pipeline efficiency, while Carbon-Aware DevOps modifies how workloads are executed based on real-time incident data regarding the availability of renewable green energy.
How Does it Work?
Carbon-Aware DevOps works by adding a carbon-based decision layer to the CI/CD pipeline, enabling it to build, test and deploy workloads when emissions are at their lowest. In practice, the fundamental idea is that CI/CD processes are executed as usual, but in this case, a scheduler will evaluate carbon-intensity data/metrics and determine when and/or where workloads will be executed as part of the CI/CD process, while using that data/metrics and policy rules to guide where a workload should be executed based upon actual emissions data collected.
Figure 1 below shows how you can turn CI/CD pipelines into emissions-controlled workloads.
Figure 1: Demonstrating Carbon-Aware DevOps
The Need for Carbon Awareness in DevOps
In today’s technology world, automated CI/CD processes are typically always on. While digital transformation has enabled us to provision and deploy more resources than ever before, it has also placed heavy daily workloads on an organization’s cloud infrastructure.
Traditional CI/CD pipelines often consume substantial cloud resources. DevOps automation processes and practices for building, testing and deploying software applications consume significant energy. As a result, they contribute to increased carbon emissions.
The carbon footprint of your CI/CD pipeline is determined by:
- The energy source used to run the data center
- The efficiency of the infrastructure, i.e., how much work can be accomplished per watt of energy consumed
- The timing of execution, since the carbon intensity of the grid fluctuates by hour
Reducing carbon usage in CI/CD pipelines provides benefits beyond reducing carbon emissions; it also creates business value through lower operational costs and less wasted compute time. Improvements in pipeline operational efficiency often yield faster feedback, less load on infrastructure and fewer test reruns.
Why are CI/CD Pipelines Key Areas for Reducing Carbon Emissions?
There are several reasons why CI/CD pipelines are a key area of focus for reducing carbon consumption.
- First, a CI/CD pipeline is one of the most compute-intensive and centrally managed phases of modern software delivery processes in an organization. Hence, even small reductions in energy usage can result in significant reductions in carbon emissions.
- Second, it provides a very clear location to measure, monitor and reduce energy consumption because builds, tests, scans and deployments are all completed using the same workloads defined by their respective specifications.
- Finally, CI/CD pipelines enable you to run tests and security scans on a schedule and to rebuild images in different geographic regions without impacting end users, thereby enabling you to implement carbon-aware scheduling and workload batching as needed.
Key Criteria for Identifying Carbon-Aware Scheduling
There are three criteria to consider when determining whether your workload is an appropriate target for carbon-aware scheduling:
- Low Urgency: There is very little urgency associated with a job, such as a daily documentation build or a nightly full regression suite build. Such jobs are good candidates for processing when carbon emissions are low.
- Known Duration: If you know how long it would take to complete a job, you can evaluate the earliest time it can be completed while still meeting its deadline, rather than simply evaluating what time would cost the least.
- Geographic Portability: If your job does not have an established requirement for where it is processed, you can route it to a processor that currently runs on renewable energy.
A Brief Overview of Green DevOps and GreenOps
Green DevOps is a new approach that seeks to improve the sustainability of the software development life cycle (SDLC) as organizations adopt increasingly automated methods to rapidly build, test and deploy software applications to enhance business growth and profitability.
The Green DevOps methodology encourages developers to adopt energy-efficient development practices, such as the following:
- Reducing the number of processes that run in the CI/CD pipeline
- Optimizing source code to reduce the amount of work required to build, test and deploy applications
- Using lightweight containers to reduce resource utilization
- Decommissioning unused resources
As more enterprises adopt a greener approach to their operations, GreenOps, a new approach to managing energy use efficiently, has emerged, enabling them to continue meeting regulatory requirements while enhancing sustainability and their reputation. GreenOps promises to maximize data center energy efficiency and minimize carbon footprints through the following benefits:
- Consistent use of real-time carbon data to schedule resources
- An AI-driven approach to resource demand forecasting that helps predict future workloads
- The use of edge computing to alleviate or eliminate workloads within a data center
Though the terms GreenOps and Green DevOps are sometimes used interchangeably, they differ in certain aspects. GreenOps generally describes how systems are developed and coded, while Green DevOps focuses on how they operate.
GreenOps represents the evolution of Green DevOps into a combined approach for managing energy usage across both the development and operation of IT solutions. Organizations can gain an edge over competitors by using AI-based resource scheduling tools that leverage actual carbon data, comply with regulations and improve their brand image.
Key Takeaways
- Carbon-Aware DevOps doesn’t mean we have to slow down delivery or produce lower-quality output due to the constraints it introduces.
- Right-sizing your build agents, auto-scaling and similar techniques can help reduce idle energy consumption.
- Proper scheduling of the workloads that consume the most energy during periods of high renewable generation can help reduce your indirect emissions.
- Optimizing the efficiency of your artifacts, where possible, will help reduce the energy cost of your build time and storage.
- Instrumenting your pipeline to report energy per build allows you to tie this back to the continuous improvement of your pipeline.
- Use the right type and sizes of build agents for your jobs to avoid over-provisioning.
- Ensure that the strategies you adopt to reduce emissions do not adversely affect throughput or reliability.