How to Save Time Building WebRTC

Photo by Lex Photography from Pexels

In this post, my colleague, Andrew Francisque, and I are going to discuss how we were able to decrease the amount of time required to build a WebRTC solution from hours to minutes.

Introduction

A complex build can sometimes take hours to complete. Usually, the challenge with these kinds of builds are the dependencies.

In our case, we partnered with Kognitiv Spark, whose mission is to empower industries’ solutions by using augmented reality. RemoteSpark (Kognitiv Spark’s remote worker app) was originally built for Microsoft HoloLens to provide augmented reality solutions for a variety of HoloLens applications like maintenance, reporting, and remote support.

WebRTC is at the heart of Kognitiv Spark’s solution and having a reliable and fast build is crucial to their success. The issue they were facing was in performing an “immediate” code validation when there were changes in the codebase. This was a particularly challenging goal due to the complexity of the codebase which would cause builds to take upwards of 4 hours to complete. In this work, we were helping Kognitiv Spark to tackle their goal by offering a solution that significantly shrunk the amount of time a build would take to complete from hours to minutes.

First, we started off with using the Azure self-hosted agents to create a pipeline for WebRTC-UWP (WebRTC for Universal Windows Platform) build. Then we explored the build time improvement by migrating to Virtual Machine (VM) self-hosted agents. This move enabled us to reduce the build time to less than half. We experimented with scaling up the VM which only marginally improved the timing. Finally, we developed an incremental build solution that provided the most significant reduction in the build time by reducing one hour to minutes. This post is going over all the mentioned approaches we described above and provides a more detailed explanation and outcome of our project.

Technical Discussion

1. Whys and Hows

WebRTC has many components and requires a lot of time to build. While we could have settled for using Azure-hosted agents to build the WebRTC-UWP solution, we decided to instead use custom VMs as self-hosted build agents. This was done for three main reasons:

1. With hosted agents, to build the solution, we would need to download and install dependencies every time. Using custom agents, we only needed to download/install them once. Because they would persist on the machine, we wouldn’t have to do so again.
2. The WebRTC solution is extremely resource-intensive. Because the hosted machines are not as powerful as one would need them to be in order to build the solution quickly, builds took at least 3 hours to build successfully. This not only means that almost half of a workday was spent on a successful build, but that if builds started and were unsuccessful, almost half of a workday was wasted.
3. With Azure-hosted VMs, each build would be completed by a different machine. No matter how small or trivial the change was, the hosted VMs would create the solution from scratch every time a build was triggered. As we will further discuss below, using a VM as our build agent would allow us to only build out the submodules that were directly affected by the new changes that were made, thus drastically decreasing our build time.

Simply put, we used the custom VMs because we knew we could do better.

2. Specs

The setup for the VM is as follows:

VM: D8s_v3 Azure VM (Specs: 8 VCPUS, 32 GB RAM, 16 Data Disks, 64 GB temporary storage)

Because WebRTC builds are fairly large, we added a 500 GB data disk to the VM. Instructions on how this is done can be found here.

After VM was created, we connected to the VM and installed these tools, which were required to build the WebRTC-UWP solution:

• Windows’ Chocolatey Package Manager (we used version 0.10.13)
• Windows 10 SDK version 1803. (Be sure to download this on the VM after downloading the Chocolatey Package Manager)
• Windows Debugger Tool
• Python version 2.7.14

We then followed the steps outlined here, in order to set the VM up as the agent to be used to build the solution. (We used the Windows x86 architecture).

Just by using a custom VM we were able to cut the build time down from the 3–4 hours previously mentioned to about 1.5 hours!

3. Scaling up the VM

As it was mentioned in the first approach, we created a VM to decrease the build time. This solution effectively shrunk the build time to half, from around 3 hours to 1.5 hours, but it was still not an acceptable solution for us and we needed to reduce the build time even more. While running the build locally we realized that it uses all the CPU power in the system to run. Also, in the next phase of linking and merging it uses 100% of the memory. That was indicating the dependency of the build to the system’s configuration.

So in the next step, we started to wonder what will happen if we increase the size of the VM. We expected to half the amount of time for the build by doubling the size of the VM. So in this section, we review how we scaled up the VM.

3.1 Resizing the VM

In Azure resizing the VM happens through changing both memory and CPU level together. By opening the main VM page, one can select Size and select your desired VM size then click on the Resize button as it is shown in the image below:

In our case, we increased the size to Standard_D16s_v3 meaning 16 vCPUs and 64g memory.

3.2 Results

Scaling up the VM by double, only dropped the build time by 30 percent, however, it was not as significant as we expected before. The build decreased into approximately one hour. This is not much progress for such a huge jump in the memory and CPU. If we wanted to keep increasing the memory at some point, first we would not able to see much progress according to Amdahl’s law, second, it is a very pricy path to take. Therefore, we stopped increasing the size at this point and started thinking about other options.

As one of the approaches, we started thinking about the nature of the build and how it works to find out a way to decrease the build time. That approach is called incremental builds and will be discussed in the next section.

4. Incremental Build As a Solution

As previously mentioned, a contributing factor in why builds took so long to complete is that post-build remnants would not persist when using the Azure-hosted agents. Each time, a new machine would be spun up, and builds would be started from scratch, no matter how large or small the change was. As one could imagine, this was not seen as the most efficient use of time.

One of our senior software engineering leads, Tyler Gibson came up with a solution that would allow us to take full advantage of our custom VMs and reduce build times even further — incremental building. Using a custom VM, we were able to store the source code, and utilize (WebRTC’s default) method of building, which uses Chromium’s GN build system. This build system uses a checksum method to ensure that we are only building out the submodules directly effected by incoming changes and leaving everything else intact. Thus, the submodules that have gone through changes are built out, then linked to the rest of the solution.

Note: If too much code is changed, the GN system will build out the entire solution, as opposed to just what was changed and what is dependent on the code that changed, but this was seen as a very small compromise to make.

The combination of increased power and stored dependencies, as a result of using custom VMs, and using incremental building brought our average build times from 3+ hours to (in some cases) 20 minutes! We saw this as a large win for the team.

Conclusion

Some builds take a long time to finish because of so many dependencies. In this article, we discussed different approaches we took to make our build run faster. We worked on WebRTC-UWP build which takes more than three hours to run at first. In this article we explain how using the combination of two techniques: using VMs as self-hosted build agent, and creating incremental build helped us to reduce the build time to minutes.

In the first step, we used a self-hosted VMs to create our build that cut the build time in half. We observed the intense dependency of the build on CPU and memory so for the next step we explored the effect of VM expansion on the build time. The results represented about 30 percent improvement, however, it was not the most desirable outcome considering the additional cost and resource it requires for scaling up the VM. So, we took another step and tried out incremental builds, where we built the whole project once at the beginning and later for any new build it only adjusts the existing build for the changed parts of the code rather than building the whole project. Incremental build gave us the biggest performance jump and ameliorated the build time from an hour to minutes.

“The ability to cut the build time from hours to minutes allowed our development team to test iterations quickly. As the code base is complex, certain changes need to be tested immediately to ensure there are no blockers for other team members, so the decrease in build times greatly increased our productivity as a team.” — Ryan Groom, Co-Founder, and CTO of Kognitiv Spark.

As a result, we were able to decrease the build time in Kognitiv Spark project by using self-hosted agents on VM, and incremental builds to help the business move faster towards their goal.

Thanks to Andrei Ermilov, Tyler Gibson, and Phong Cao.