If You’re Not Automating Your Data Gathering, It’s Time to Start

Another Cisco Live has come and gone, and as many of us have recovered mentally and physically from the event, I love to use this time to reflect on all of the great conversations I’ve had with everyone this year.

One absolute fact is that automation is now everywhere.

The majority of people not only had automation tools in different domains, but they were very aware of what was being used and how it was being used. That extra level of awareness may seem insignificant, but in reality, it signals that network teams are adopting more tools, using them seriously, and becoming more interested in what tools their colleagues in other domains are using to automate infrastructure. This means that most of the people I spoke with are advancing through the automation stage rapidly compared to last year and are looking at how to drive further efficiencies by orchestrating end-to-end processes.

One of the easiest ways orchestration can benefit teams, and ultimately the entire organization, is to start by focusing on including data gathering and data transformation as part of your automated process.

What is data gathering, and what is data transformation?

Maybe the best way to frame it is to consider how you run an automation. If we take a program written in python and used to provision a network device for a new server or application, it’s probably written in such a way that it can be applied generally to multiple devices of a particular type in a specific area of the network. For example, a script can work to provision the necessary resources for JunOS switches in a data center. You can give the script the appropriate inputs on the command line for the switch name, interface, VLAN, IP address, netmask or whatever details it needs to make the changes. If the automation tool is a web interface, then the input method might be a form that needs to be filled out before you click the “Submit” button.

No matter what, these automation tools need some critical details in order to make changes.

This data needs to come from somewhere, but where? That’s the question I typically pose in my conversations, and immediately the wheels start spinning…

Next question: How does that data get into your automation tools?

Do you ever need to modify the data for the tool?

Boom! Awareness of data gathering and transformation has been achieved.

In the early automation stages, it’s something that network engineers do so often and so quickly that they don’t realize feeding the automation tools takes time and needs to be automated.

When we break it down even further, data gathering and transformation doesn’t just happen once in a change process, but many times.

Initially, you have to feed the automation tool, so teams manually hunt down the data from various sources like tickets, inventory, IPAM, and more. Before the change is made, there ought to be a series of pre-check tasks that verify it’s safe to make changes to the network. This is also data gathering. Normally, you want to check one or more network devices to ensure that the state of these devices is correct for the change. If we are assigning a new IP, interface, or VLAN, it might be bad if we see those resources are already in use, so part of a typical MOP might be to “run these commands and look for this, if the response looks like this, then continue”. And if these tasks are done before a change is made, then a similar set of data is gathered and evaluated AFTER the change is made — the post-check. Yet another data gathering step.

Transformation occurs in between steps in a change process — you might need to reformat the output from an IPAM in order to pass in the right devices to another system, and so on. Engineers usually do this on the fly without really thinking about it. But even if you can reformat data quickly, it still makes networking harder to scale — process completion will be bottlenecked by manual touchpoints that create a significant delay when multiplied across many different process executions.

Keep in mind that all of this is typically done manually, as bookends that surround the task of using an automation tool to make changes. These data gathering and transformation steps can operate on CLI-based devices or using more modern API-managed solutions.

The real kicker is understanding how much time you spend on these sections of the process and comparing that to the time spent actually running the automation tool. This realization tends to light up people’s brains brighter than that 110 degree Las Vegas sunshine. The point is that all of this SHOULD be automated and with Itential it CAN be automated — and a lot more easily than you might think.

Boom! Awareness of orchestration has been achieved.

The very first steps into orchestration can be understanding where the data comes from that you need to automate changes, and automating those data gathering tasks, transforming the data, and supplying that together with the different automation tools in your different network domains.

Orchestration opens up the possibilities of expanding into other, often time-consuming manual tasks, like managing change request tickets in applications like Jira or ServiceNow, updating documentation, notifying multiple teams of network changes, updating monitoring and observability platforms, and more.

Itential’s platform simplifies how you can integrate all these systems along with your existing network automation tools so you can create workflows to orchestrate these processes on a visual, low-code canvas. To see how to get started with the basics of orchestration in the Itential platform, start with data gathering and transformation — and here’s a perfect video to get you started.