When to Set Up Self-Hosted CI/CD Agents in Azure DevOps?

Regardless of which Continuous Delivery platform you’re using, you should always aim to utilise it in the most efficient way. Build and delivery times, as well as the overall costs, are the core factors to be considered. The build agents play a major role in this process. Many modern CI/CD systems offer built-in workers, but also self-hosted machines to run the pipelines. Which approach is better? The answer is not straightforward, and it depends on the specific needs of your project. My goal is to show you in which cases you can benefit from each of these options. This article is based on the Azure DevOps platform, but the insights can be useful on any CI/CD platform which offers the possibility to create custom build agents.

What Is a Build Agent?

Each CI system needs a place to run the pipeline jobs. A build agent is a machine that’s dedicated to this purpose. Depending on the platform, it might be called “agent” or “worker”, but what matters is that this entity is a crucial component when it comes to the pipeline’s performance and costs.

Types of Azure DevOps Agents

Most CI systems offer vendor-hosted agents. That means they can be used as a PaaS resource. Depending on the platform, you can choose how many agents, parallel jobs or build minutes you want to pay for. In Azure DevOps, each organisation by default has one free parallel job with a limit of 1800 minutes for private projects. Additionally, it comes with the possibility to add one self-hosted agent. If that is not enough for, you have two options. You can either buy additional jobs with unlimited minutes or buy more “licenses” for self-hosted agents. What’s worth mentioning here, is that you will receive one more self-hosted agent slot for each person in the organisation who has a Microsoft Developer Network subscription.

For the open-source projects, there are 10 parallel jobs available for free and you can also create an unlimited number of self-hosted agents.

Advantages of The Parallel Jobs

The reason I’m referring to jobs and not agents is because a Microsoft-hosted job can run on many different machines. The pipeline allows you to specify if you want it to run on a specific version of Windows, Linux or even MacOS. This poses the question, why would you need additional jobs, if you can build multiple different pipelines with applications for any platform with just a single agent? While technically it’s possible, it’s not a good idea, and I will show you why.

For this example, I created a simple pipeline with three jobs. Each of these jobs performs the same steps for one of the sample .NET applications:

1.  Install NuGet
2.  Install .NET Core 3.1
3.  Restore packages
4.  Build the application

I used the devops-project-samples repository, which contains a bunch of sample applications prepared specifically for testing pipelines. First, I ran the pipeline on a Microsoft-hosted agent:

The whole pipeline execution took 3 minutes 23 seconds (the sum of jobs + time needed for switching context). The MS-hosted agents have already installed both NuGet and .NET, so the majority of the time was consumed on restoring packages and building the app. Next, I ran the same build on a self-hosted agent pool:

The agents were clean Ubuntu machines, so the first run took a little longer, because they needed to install all the required tools. That said, the total build time was similar to the MS-hosted scenario, because they were run in parallel. However, the most interesting part is the next run on self-hosted agents:

Since they have already installed all the tools, once again, the time was spent on restoring packages and compiling the application. But some of the packages were already in the cache. The entire pipeline took one minute to run — that’s the execution time of the longest job. This leaves us with 3:23 vs 1:00 on a bunch of simple boilerplate apps. Now imagine the difference on an enterprise-level solution. That’s a significant improvement.

Accelerating The Process

Regardless of the framework, the most of time is usually consumed on restoring packages and dependencies. They’re restored every time the build is performed on the hosted agent. That’s because you get a fresh new machine for every build. To overcome this, Azure DevOps allows you to use Pipeline Caching, but not every CI platform has such a feature. This is where you can benefit from a self-hosted agent.

When you’re using self-hosted agents, each pipeline has its own working directory. But the machine tools and cache are preserved between runs. The first run will always take some time to download all the dependencies. However, it won’t be needed for subsequent runs, so the process will be much faster.

There might be cases when you actually want to perform a clean build. When that’s the case, you can set up a maintenance job for self-hosted agents to clean them up, e.g. once a week. For scale set agent pools, there’s an option to tear down virtual machines after every use.

Set Up a Custom Agent

Creating a self-hosted agent is a fairly simple task. There are multiple ways to do it in Azure DevOps:

•  Manually. You need to create an agent pool in the Azure DevOps portal. Then, you just have to copy the generated script and run it on the selected machine. You can use any cloud virtual machine, on-premises machine, or even your personal computer. There are separate scripts for each operating system. The script will install the agent on the selected VM and ask for organisation/project URL, and the agent pool which it should join.
  
  
•  With Azure scale set. First, you have to create a scale set on Azure Cloud. Then, you need to generate a new agent pool and choose “Azure virtual machine scale set” as the pool type, and connect to it. You also need to set up service connection to the subscription or resource group where the scale set was created. The agents will be installed automatically on the virtual machines. This approach lets you manage and scale your agents easily. You can also use the custom VM image with all the required software already installed.
  
  
•  Using container with Docker or Kubernetes. You can prepare a custom container image with all the needed software and the agent, and optionally pack it with helm if you want to run it in AKS (or any other Kubernetes provider). This method is similar to using a scale set, you also can scale your agents, but they won’t be scaled automatically by the Azure DevOps server. Choose this approach if you want more control over managing the agents, but remember — maintaining them may also require more effort. 

Additional Benefits

Let’s continue discussing the actual added value provided by the self-hosted agent. Speeding up the pipelines is one advantage. The other benefit is the possibility to customise the agent tools. Imagine that you need to run performance tests using Apache JMeter. Although the hosted agents come with a variety of tools, JMeter isn’t included. For some of the tools, there are installer tasks which can be used during the build to install and/or set the proper version of the given tool. It’s really convenient, but remember — each installer impacts the overall build time. In most cases, it’s better to have the tools already in place. Using a custom agent, you can prepare a VM or a container image, and use it as a base for the agent. That saves a lot of time during the build.

An Alternative Approach

If you can’t use the custom agents, but you need to use specific software which takes a long time to install, there is one more option. You can use a container job in Azure DevOps. This way, you’ll be able to prepare a container image with everything you need to use as context during the build. That means all the steps will be executed inside the container. The only issue here is the time required to download and setup the image. But in some cases, it may be the optimal approach.

The Cost of Setting Up Self-Hosted Agents

The financial aspect of self-hosted agents may also be beneficial. However, it depends on how you’re going to use your virtual machines. The discussed prices are valid at the time of publishing this article, but they may change in the future. According to the billing, each additional Microsoft-hosted job costs 33.74 €. For this price you get a job which runs on a DS2_v2 tier VM (2 cores, 7GB RAM, 6400 IOPS). On Azure, a VM of this size costs 81.26 €. At first glance, it might seem that getting an MS-hosted job is significantly cheaper. But the question is, how much computing power do you really need for an agent? Let’s look at another example.

For the price of 27.70 € (which is less than one MS-hosted job) you can get an Ubuntu VM of size B2s (2 cores, 4GB RAM, 1280 IOPS). How many agents can you host on a VM this size? Well, it depends on your build requirements. For some simple tasks, it may be enough for 3 or even 4 agents (if you set them up in containers). But in some cases, the build task/script may require much more RAM or CPU power, and this machine may not be enough even for one agent. On the other hand, for a similar amount of money, you can set up a scale set with 4 VMs of the B1 size (1 core, 1 GB RAM, 320 IOPS) and it will cost you 27.84 €.

As you can see, it’s not easy to determine which approach is cheaper. If you have a big number of not-demanding builds, use a scale set with low-tier VMs. If this is not enough to match your needs, buying an additional job may be the right decision. But there might be some edge-cases when even the hosted job agent is not enough (for instance, in game dev build scenarios) and you may need a bigger, more expensive VM.

The examples above are based on Azure Virtual Machines, but remember, you can use any other provider for a self-hosted agent. So in order to determine what is cheaper — a self-hosted agent or an MS-hosted job — you will have to calculate which is a better fit for your needs.

Keep the Agents Up-To-Date

There’s one more aspect of self-hosted agents that we have to cover: maintenance. The vendor’s infrastructure will always provide you with fresh tools. But you can also prepare your own setup, which will keep your agents up-to-date. In this case, the container agents would be the easiest to maintain. You can set up a pipeline which creates new images and deploys them, e.g. once a month. It’s important to keep the software updated, especially for the security reasons. In Azure DevOps, there also are some features that require a specific version of an agent. But this particular component can be easily updated in the portal.

Summary

Both the vendor-hosted and self-hosted agents have their pros and cons. You can achieve faster build times, install custom software and optimise costs with the self-hosted approach. There are multiple ways to set up custom agents, starting with simple virtual machines and ending with containers. On the other hand, when you pick a vendor-hosted solution, you don’t need to worry about the system or software updates. There’s no single right answer that would be optimal for every project. I would recommend taking the approach that better suits your organisation’s needs — you can always ask a DevOps consulting partner to help you pick the right option. If you want to optimise costs and speed, you can consider setting up your own agents. But if you’re looking for an easy solution, which simply does its job, you can keep using the ones provided by the vendor.