When I took on the job of designing and building CI/CD pipelines for a project running Azure DevOps Server on-prem, the community already had pipeline definitions available. I assumed I had a head start. I was wrong. Every one of those pipelines was written for Azure DevOps Services. Cloud-managed agents, public internet access, Microsoft-hosted authentication — none of it applied to our environment.
I could not adapt those pipelines. I had to make deliberate design decisions about how our pipelines would work, what standards agents would follow and how we would validate changes before they reached upstream. A junior engineer worked alongside me to implement these decisions. This article covers the design choices that mattered the most and why they were necessary, specifically on-prem.
The Fundamental Shift: Everything You Relied on is Gone
The first thing to internalize about on-prem Azure DevOps is that Microsoft-hosted agents do not exist. This is not a configuration option. It is an architectural fact. Microsoft’s own documentation states it plainly: Hosted agents are only available with Azure DevOps Services. On-prem, every agent is self-hosted, which means you provision it, configure it, maintain it and manage its tool dependencies yourself.
This has a cascading effect. Cloud pipelines assume a freshly imaged VM for every job, pre-loaded with Node, Python, Docker and dozens of other tools. On-prem, your agents persist between runs and carry only what you install. Every capability your pipeline needs, including compilers, package managers and container runtimes, must be explicitly provisioned and registered as agent capabilities. If a job requires something the assigned agent does not have, it queues indefinitely with no useful error message.
Authentication is also completely different. Cloud Azure DevOps authenticates over the public internet using Microsoft Entra ID or personal access tokens. On-prem, you authenticate against an intranet server using Windows Authentication, Active Directory, NTLM or Kerberos. Service connections, which are the mechanism pipelines use to access external resources, must be manually configured against your internal server and are not transferable from cloud pipeline definitions.
Agent Design: Stability is the First Problem to Solve
My first week was spent debugging agents that registered successfully and then went offline the moment a job was queued. The root cause was the agent service not recovering gracefully after brief network interruptions. This is a problem that simply does not exist with cloud agents because each job gets a fresh VM.
The fix was configuring the agent as a system service with automatic restart on failure, not as a foreground process. It also required tuning connection retry settings so the agent could survive transient network issues without dropping out of the pool. Do this before writing a single pipeline. An agent that silently goes offline causes jobs to queue indefinitely rather than fail fast. That is far harder to debug than an outright failure.
Explicit capability declarations in pipeline YAML are equally important on-prem. In cloud pipelines, vmImage handles agent selection for you. On-prem, you specify a pool and rely on demands to route jobs to agents that have the right tools. Without precise demands, any available agent picks up the job. If the tool is missing, the failure happens mid-run.
Here’s the pool configuration pattern that worked for us:
pool:
name: OnPremAgentPool
demands:
– Agent.OS -equals Linux
– docker
– make
Artifact Publishing: Not the Same as Cloud
In Azure DevOps Cloud, the publish task automatically uploads artifacts to Microsoft’s infrastructure. On-prem, you need to think deliberately about where artifacts go. Your on-prem Azure DevOps Server manages artifact storage directly, and you need to make sure retention policies, storage limits and feed configurations are in place before pipelines start producing artifacts at scale.
One on-prem-specific issue I ran into: Artifacts published in one stage are accessible to downstream stages using the download task, but the behavior depends on how your server is configured and which version of Azure DevOps Server you are running. Cloud documentation for pipeline artifacts assumes a specific service layer that the on-prem version may implement differently depending on the server version. Test artifact passing between stages explicitly before building multi-stage pipelines that depend on it.
Here’s a minimal multi-stage artifact pattern that worked reliably on-prem:
stages:
– stage: Build
jobs:
– job: BuildJob
steps:
– checkout: self
submodules: recursive
– script: make build
displayName: Build
– task: PublishPipelineArtifact@1
inputs:
targetPath: $(Build.ArtifactStagingDirectory)
artifact: build-output
– stage: Test
dependsOn: Build
jobs:
– job: TestJob
steps:
– task: DownloadPipelineArtifact@2
inputs:
artifact: build-output
– script: make test
displayName: Run Tests
Submodule Validation: Design for Two Tiers
Before I designed the validation pipeline, submodule changes were never tested against the broader system until they were upstreamed. The failure pattern was consistent: A change would pass its own tests, merge cleanly and then break the parent build hours or days later. By then, the connection back to the submodule change was not obvious and the cost of fixing it was high. I designed a two-tier validation system to eliminate this failure category.
The design uses two distinct validation tiers based on how tightly a module is coupled to the rest of the system.
For standard modules, the validation is straightforward: When a PR opens, build the module and run its unit tests. If both pass, the PR can be merged. This keeps feedback fast and the pipeline focused on what changed.
For tightly coupled modules, where a change can ripple across multiple dependents, the validation is deeper. When a PR opens for one of these modules, the pipeline builds every dependent module, assembles the full integration image, replaces the newly built package in that image and runs the full integration test suite. This catches integration failures before merge rather than after, which is the only point at which they are cheap to fix.
The key design decision is treating these as two separate pipeline definitions, not two branches of one pipeline. Each has its own trigger, its own required status check on the PR and its own pass/fail criteria. Mixing them in a single pipeline with conditional logic creates noise and makes failures harder to diagnose.
Network: Mirror Everything Before You Need It
On-prem agents have no assumed internet access. Cloud pipeline YAML that calls npm install, pip install or docker pull against public registries will fail silently or with misleading errors the moment it runs on an on-prem agent in a restricted network.
The right approach is to mirror every external dependency to an internal registry before writing your first pipeline — package registries, base container images and build tools. Configure your agents to use internal mirrors by default, not as a fallback. Two specific patterns that caused the most pain: Package managers that silently fall back to cached or stale versions when a registry is unreachable, producing builds that appear to succeed but use outdated dependencies; and proxy configuration that needs to happen at the agent service level, not just in pipeline environment variables. Some agent operations bypass pipeline-level proxy settings entirely.
Code Coverage Publishing: V2 Won’t Work the Way You Expect
If you are running Azure DevOps Server 2022 on-prem and plan to publish code coverage results, you will likely encounter a frustrating problem: The documentation says PublishCodeCoverageResults@2 is supported on Server 2022, but when you try to use it on a self-hosted agent in a corporate network, it fails. The error points to a connection problem with dev.azure.com. This is not a configuration mistake. The v2 task has a cloud-dependent processing step that makes an outbound call to Azure’s infrastructure to generate its coverage reports. In an on-prem environment without that outbound connectivity, it simply does not work.
The practical answer is to stay on PublishCodeCoverageResults@1. It works reliably on-prem and the constraint is format: v1 only supports Cobertura and JaCoCo. Whatever language your build uses, the coverage output needs to be in one of those two formats before you can publish it to the pipeline UI.
For Go repositories, this requires an extra conversion step. Go’s native coverage tool produces its own format that Azure DevOps cannot parse. The gocover-cobertura converter (github.com/t-yuki/gocover-cobertura) transforms Go’s output into Cobertura XML, which v1 can then publish. The full sequence in the pipeline YAML is:
# Run Go tests and generate native coverage output
– script: |
go test -coverprofile=coverage.out ./…
displayName: Run Go Tests with Coverage
# Convert Go coverage format to Cobertura
– script: |
gocover-cobertura < coverage.out > coverage.xml
displayName: Convert to Cobertura
# Publish using v1 — the only task that works reliably on-prem
– task: PublishCodeCoverageResults@1
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: coverage.xml
A few things worth noting: gocover-cobertura needs to be installed on the agent, which means adding it to your agent-provisioning script rather than installing it inline during the pipeline run. Installing tools inline works in cloud pipelines because each run gets a fresh agent. In the case of on-prem self-hosted agents, repeated tool installs per run accumulate overhead and can cause version conflicts. Treat agent-level tools as infrastructure. Install them once, document them in your agent capability matrix and declare them as demands in your pipeline YAML.
The Takeaway
On-prem Azure DevOps is not a simplified version of the cloud product. It is a different operational environment that requires deliberate design decisions around agents, authentication, artifact storage and network connectivity that cloud users never have to think about. Community pipelines written for cloud Azure DevOps are not a starting point. They are a map of every assumption you need to invalidate.
The most impactful decision I made was the two-tier submodule validation design. Before it existed, upstream breakages from untested submodule changes were a recurring problem with no systematic fix. After it was in place, that entire failure category was caught at PR time before anything reached the main branch. The technical implementation was not the hard part. The design decision to treat submodule validation as a first-class pipeline concern and to direct my team to build and maintain it accordingly was what made the difference.
The engineering that makes on-prem CI/CD reliable is invisible when it works. When it does not work, the failures are subtle and the documentation is thin. I hope this article saves you some of the time it took me to figure these things out.