Engineering teams ideally want to spend all their time writing code to create applications and services for their users! In reality, many end up spending a significant portion of their time on other tasks, such as maintaining DevOps infrastructure.
In Azure DevOps, Microsoft-hosted agents (aka Azure Pipelines agents) provide a fully managed, low overhead way to get started with Azure Pipelines. Many customers find that these agents are not flexible enough to meet their needs – not enough power, not enough memory, an inability to connect to private networks, etc. In these cases, teams can use self-hosted agents for maximum flexibility, but at the cost of a significant increase in overhead and maintenance costs.
Azure Virtual Machine Scale Set agents addressed the overheads problem by enabling teams to create agent pools based on Azure VM Scale Sets with any SKU, image, storage or network that teams wanted. While the maintenance overheads were less than what teams were spending on maintaining self-hosted agents, they were still spending significant amount of time maintaining and troubleshooting their pools. Since the VMs were being created in the team’s subscription, it was also difficult for the user and support teams to troubleshoot when things went wrong. We saw an opportunity to further improve the experience of creating and managing custom Azure DevOps pools.
Today, we’re excited to announce the public preview of Managed DevOps Pools (MDP), a feature of Azure DevOps that enables dev teams or platform engineering teams to quickly spin up custom DevOps pools that suit your team’s unique needs. It combines the flexibility of Scale Set agents and the ease of maintenance of Microsoft Hosted agents. It enables engineering teams to establish consistency and best practices while maximizing performance, security, compliance, and cost-efficiency for their custom DevOps Pools. Managed DevOps Pools was inspired by a Microsoft internal service called “1ES(One Engineering Systems) Hosted Pools”. You can read more about the problems Microsoft solved with “1ES Hosted Pools” in Managed DevOps Pools – An Origin Story.
Managed DevOps Pools for Azure DevOps
The Benefits
By using Managed DevOps Pools, teams can expect to see the following key benefits.
Hosted on your behalf
Managed DevOps Pools is a fully managed service where VMs powering the agents are created/managed by Microsoft services in Microsoft owned Azure subscriptions. VM agents are not created in team’s own Azure subscription like it is in Azure DevOps Elastic/VM Scale Set Pools. The host on behalf model reduces the time spent in managing agents, improves reliability and won’t have other services in the same subscription competing for compute cores with your CI/CD agents.
Time spent in Management
Managed DevOps pools will drastically reduce time spent in management of agents that are based on on-premises infrastructure or manually maintained. Teams that are using Azure Virtual Machine Scale Set agents can further reduce the time spent in creating, managing and troubleshooting Scale Sets, by switching to Managed DevOps Pools. Teams that used the service in private preview were able to create new pools in under a minute and spent little time actively managing the pool.
Specific Pools
Due to the ease with which new pools can be created, organizations can very easily create multiple team-specific or workload-specific pools. Creation of specific pools helps teams tailor pools to accelerate the workloads that run on it and eliminates the noisy neighbor problem. A team in private preview created a pool with a memory-optimized SKU, for a build workload that was memory intensive. They then switched only the pipelines that would benefit from a more expensive SKU to the new pool to significantly reduce the duration of their build pipelines.
DevOps Billing
Managed DevOps Pools helps optimize a team’s DevOps bill through many features. It makes it easy for teams to find an optimal balance between a pool’s QoS/performance and cost. Managed DevOps Pools offers advanced features to manage Standby agents and therefore helps teams to better manage cost than Scale Set agent pools. We observed private preview customers that were able to reduce their Azure billing by as much as 50%.
Scalable
Teams that use on-premises infrastructure are limited to the hosts they own. Teams that create agents manually are limited in scale to the maximum agents they create. Scale Set pools perform less optimally when its scale approaches 100 parallel agents. Managed DevOps Pools solves the scaling problem by automatically orchestrating multiple virtual machine scale sets under the hood and enables Managed DevOps Pools to scale to 1000s of agents in a single pool. This also helps Azure DevOps pools better respond to bursty traffic and reduce queuing of pipelines.
Notable Features
Microsoft-hosted Quick-starter Images
Teams can create pools with quick-starter images that contain the same software present in Microsoft hosted agents. This provides an easy transition for teams currently using Microsoft hosted agents that want to use a more powerful agent, require the agents to be stateful, or need to connect securely to their private network.
Standby Agents
Teams can choose to make their pipelines start more quickly by deciding how many agents they want pre-warmed during specific hours of the week or choose the “automatic” option that uses historical data to create standby agents.
Private Networking
Teams can create pools that connect to resources on their private network, such as package registries, secret managers, and other on-premises services.
Bring your own Image
Teams can create pools with images that the team has created with pre-requisites that are unique to their scenario.
Stateful Agents
By default, MDP pools are stateless and a new agent is created for every pipeline job. However, teams can choose to reuse the same agent in multiple jobs to improve performance of their pipelines because of not needing to re-download files or not needing to re-compute operations due to local cache hits. Managed DevOps Pools implements best practices for stateful agents by auto-recycling agents based on time or the agent running out of disk space.
Pick any SKU
Azure offers a variety of compute families that are tailored for various workload requirements. Teams can pick an Azure SKU family and a size that matches their workload’s unique core/memory/disk usage profile to make them more performant or cost effective.
Additional Storage
If teams need extra disk space but are content with the number of cores and memory on their SKU size, they can just add an additional disk without needing to go up a SKU size.
Geographic co-location
Teams can create pools in a geographic location that is closest to the rest of their resources for improved performance due to low network latency or to meet compliance requirements.
Get started now
Managed DevOps Pools for Azure DevOps is now available public preview, and you can access the service directly in the Azure Portal. During the public preview period, you can get started with no extra service-related billing apart from the cost of Azure resources created as part of the environment. To learn more about pricing for compute, storage, networking, and other Azure resources, check out the Azure pricing calculator.
To learn more about Managed DevOps Pools and getting started with the service, visit the Managed DevOps Pools page. You can request features or report bugs in the Azure DevOps Developer Community portal.
The post Announcing Public Preview of Managed DevOps Pools (MDP) for Azure DevOps appeared first on Azure DevOps Blog.