NetDevOps & How to Apply CI/CD Pipelines to the Network

DevOps has become an aspiration for technology companies everywhere, and the networking space is no exception. Unfortunately, for network engineers, the way to manage the network doesn’t necessarily align with the pre-existing patterns software engineers use to implement DevOps methodologies. However, today’s networks are complex puzzles comprised of many different segments including access networks, data center networks, core networks, LANs internal to a physical location, WANs spread across the world, and virtualized networks, among others. Networks continue to increase in complexity, resulting in more change requests thus a higher risk of human error. Managing these segments is too heavy a load to maintain with direct human interaction. To accelerate network automation and orchestration, network operators need to change the way they think about network changes and configuration management and look to DevOps practices of building and updating software with traditional CI/CD pipelines.

For a deeper understanding of the problem, it’s useful to abstract the traditional CI/CD pipeline into basic phases and look at the approach used in software development. At the outset, it’s necessary to store all of the code and changes. This is best done in some version of source control like git, or CVS. Once established, one can build the pipeline.

• The first phase is the build here, various changes are merged, and code is compiled into its complete and final form.
• The next phase is the testing In this phase, tests (which might include unit, regression, and performance) are run.
• Finally, in the deploy phase, code is put on the target platform. Any time a change is made to source control, a developer can approve the change automatically kicking off the process to push that change to the target platform.

Applying CI/CD Pipelines to the Network

If the traditional CI/CD pipeline is applied to network configurations, some issues arise. Specifically, there isn’t generally a consistent source of truth. There may be several templates that are each responsible for chunks of the configuration, but the actual final built version of each configuration is only available on each specific device. In this scenario, the developer can version control templates and begin a build process each time a template is changed. However, the entire configuration isn’t being built, so the only way to achieve the actual result of the build step is to apply it to the device. This brings up two very big problems.

• First, one doesn’t know if the configuration that being adding to the device is actually going to be accepted by the device. The templates could have errors, or the code that exists on the device might conflict with the changes being made. The device may reject all the changes (which is the desired outcome in this situation). Alternatively, the device may accept some, but not all, of the changes, leaving the device in a potentially unstable state. Even worse, the changes may be made, but the result of the configuration isn’t what was intended.
• The second glaring problem is that changes were deployed before the test phase! As a result, testing is done on the live network instead of matching the desired state.

If anything goes wrong in testing, the device will have to be changed again in order to restore it to its previous configuration. In this scenario, it’s necessary to add a step to backup before deployment.

Often, companies try to mitigate risk by making these changes during scheduled maintenance windows when an outage would have the least impact. However, isn’t a major point of doing CI/CD to make changes as they happen? By implementing this gating mechanism, the core principles of these methodologies are being violated.

Rethinking DevOps for NetOps

In the network universe, it is difficult to “compile” configurations as one typically does in the build phase of CI/CD pipeline. To accommodate, the NetOps pipeline breaks the traditional build phase into two. The first is the build phase where one builds complete configurations from external data and/or blocks of pre-built configurations. The second is the “validate” phase. This phase ensures that the configuration being deployed actually makes sense. Unlike the previous network example (where it is not known whether the constructed configuration would even be accepted by the device), the validation phase checks to see if a configuration is compatible with the device without having to deploy it. The combination of these two phases approximates the type of work a compiler might do during the build phase.

The next phase of the NetOps pipeline is testing services to make sure changes do what they were intended to do. For example, if changing an ACL policy on a device, one can run a series of tests against that policy to simulate what would happen for multiple scenarios.

The final phase is deployment, where the changes are applied to network devices. Most networks operate using distributed sources of truth, where multiple systems have authoritative information about different areas of the network, and these changes to the network may not automatically propagate to all of the other systems. Therefore, in addition to supporting device changes, it’s vital to keep all of the other systems in sync anytime an infrastructure change is made.

A process (as described above) serves to help networking teams adopt modern agile processes through automation. As more and more services are made available, the adoption of Network Infrastructure as Code becomes easier. By abstracting and slightly tweaking to abstract the traditional CI/CD pipeline, organizations of all sizes can accelerate the use of network automation and orchestration.

Itential’s NetOps Pipeline & API Services

At Itential, we’ve been working to make this transition easier for our customers, and in order to help customers embrace Network Infrastructure as Code methodologies, we are building services that better enable network engineers to achieve these goals. We also understand that you can’t just up and change everything you do to conform to some new paradigm. Our services attempt to align with traditional software engineering practices, while staying flexible enough for customers no matter their level of automation. In order to do that, we’ve tweaked the traditional pipeline slightly as the NetOps Pipeline for Network Infrastructure as Code.

To learn more about how Itential can help you incorporate CI/CD and DevOps practices into your network automation and orchestration initiatives to gain great velocity, improve quality, and reduce risk click here or watch this on-demand webinar.