Advancing your Network Automations with Advanced Data Transformations Techniques

Automating network changes shouldn’t be hard, especially when they happen so frequently. Yet, they feel still feel like it sometimes as the simplest network change requests turn into data gathering treasure hunts between multiple applications and IT systems just to complete the data gathering process before the network change can happen.

This traditional manual process of data gathering is not only prone to human errors but chips aways at precious time you could be making more network changes. Because I know these pain points all too well through my experience as a network engineer, I’ve been passionate about sharing why data transformation is an important building block that cannot be skipped when it comes to building network automations.

In the past I’ve focused on why teams should build automations that account for data transformation, but this time I want to shift gears and take you on the journey of mastering  advanced data transform methods to meet the specific needs of network teams.

Why Traditional Methods Don’t Scale with Automation

In the world of automation, data transformation processing is necessary, especially when data comes from different sources of truth. However, when done manually it only slows down the process you’ve spent time automating. That’s why automation must integrate with many different systems for an efficient end-to-end process.

The process of transforming this data into the proper format hasn’t been simple for network practitioners, since the traditional methods of automating between different systems require writing code to integrate with systems and transforming data between those systems. On top of that tedious process, there’s the hidden tax of managing all that code over time and making changes to everything when an API changes or a data format is different. That’s a tall order for an experienced programmer, let alone for a network engineer with no coding experience, a significant mountain to climb.

Enabling NetDevOps Teams to Rapidly Build Reusable Data Transformations

NetDevOps teams cannot reasonably keep up with maintaining code for ever-changing network needs. They need a better way that fits in their automation process, not one that happens separately.

With the Itential Automation Platform, network engineers and developers alike can rapidly build data transformations on a visual canvas, making it easier to “see” each step of the transformation process. Data types are defined by JSON schemas, and conversions between data types are handled by methods put on the drag-and-drop canvas that can be then linked together. And with testing accomplished so quickly, you can rapidly see the data generated by the transformation, iterate, and make changes if needed.

For DevOps teams, the method of creating and managing data transformation through JSON schemas is great as well. Since JSON schemas are assets that are decoupled from the main automation and reused across different automations, it can make management of data transformations much easier.

The result of this process is a reusable, independent automation assets that can be executed as a task in any automation. Simply define the input variables and the data transformation task runs, formats the data as you defined, and outputs data the way your other tasks need it.

Creating Advanced Automations with Advanced Data Transformations

No network is identical to another and because networks are so unique, there is no one size fits all for data transformations that meet the specific needs of network engineers and developers. That’s why Itential has specifically engineered advanced features for data transformations – so you can use what works best for you.

Using Multiple Input Schemas in a Transformation
For instance, since input variables are defined as a JSON schema, you can have multiple input schemas available with variables coming from different sources. This means that a single data transformation can accept data from multiple systems and sources of truth, parse it, and transform it for multiple tasks in the automation workflow. As new applications and APIs are added or updated over time, DevOps teams can deploy bulk updates by updating a single data transformation and any automation that uses those assets will also automatically update, ensuring consistency across the network.

This process allows for flexible automation that drive efficiency for the entire organization. For instance, string manipulation are a common task and Itential provides a number of tools to approach this process. From simple ways to split and concatenate, all the way to complex regular expressions, there’s a way for you to get it done no matter how much coding experience you have.

Generating Human Readable Output for Real-Time Notifications
To complete the end-to-end process, notifications must also be included. One of the most important and often overlooked data transformation processes is making strings for human readable output — like updating a Slack channel with details of an automation that’s running or updating a trouble ticket in ServiceNow or Jira. Data transformation for generating this kind of format is very straightforward in Itential by using a template literal method, which allows you to write freeform text and dynamically imbed variables that can be mapped in from your input variables. The result is a perfectly formatted update for any system you are using. 

Leveraging Queries & Logic for More Efficient Data Parsing
Data transformation can include queries and logic, which provides the ability to parse data from complex objects, evaluate that data, and take different logic paths to manipulate that data based on the evaluation results. By moving this logic from the automation and into the data transformation, you can simplify the workflow logic, and build some very flexible and dynamic data transformations that are more independent and efficient.

If you’d like to see some of these advanced data transformation techniques in action, check out this recent demo or check out our full data transformation demo series here. You can also try it for yourself with our free JSON Schema Transformation tool here.