Turning Tribal Network Knowledge into Organization-Wide Automation

Every network team has a wealth of tribal knowledge, amassed from years of experience, that has become critical to the operation of its network. This knowledge touches every aspect of the network — from creating new services to quickly troubleshooting and resolving network problems across any domain. However, it’s only useful to those who know it.

In the past very little of this tribal knowledge was ever shared, let alone documented. What little was written down was rarely comprehensible for less experienced network engineers. This typically resulted in a call to one of the network elders, who would end up doing the work themselves since it was just faster than documenting and teaching (something that is understandable if the call comes at 4 AM because a critical part of the network is down). While this is a real operational problem which has been tolerated for a while, it cannot stay the norm.

The operational problems that arise from tribal knowledge create delays, especially when compounded with manual changes that are already tedious and time consuming, make it impossible to keep up with network changes. With the expectation put on today’s network infrastructure, teams can’t get by this way anymore. You must find a way to take that tribal knowledge and turn it into organization-wide automation.

Network automation is the only way to overcome the challenges teams are facing in today’s environment, however it comes with its own set of challenges for network practitioners — a tremendous skills gap, a lack of time, and very low confidence in making automated changes to a complex network.

So, the big question is how can you get the core networking team, with its decades of knowledge, to not only impart that knowledge but also participate in building automations using that knowledge?

How to Extract Tribal Knowledge to Increase Automation Participation

While it may not seem like it at first, network automation and tribal knowledge are tightly linked together. Automations that can integrate these tribal procedures will more likely gain confidence by the network team when they run. After all, it’s the same process that the network elders use, and if they are the ones who have built the automation steps, then there’s little question whether they will work. The same person who you called at 4 AM to help with a network issue has now helped write the automation that you (and anyone) can use to resolve the same problem, without ever having to pick up the phone or send a message.

Sounds good in theory, but let’s talk about how to put it into practice so you can help the network team document this incredibly useful and critical tribal network knowledge AND participate in building automations. All of this can easily be accomplished with Itential’s command templates, a feature of the Itential Automation Platform specifically engineered to enable anyone in the network team to understand and follow the same process that someone might use on the command line of a network device.

When you need to determine the state of a network device, think of what command you would run. Now, after the command is run, what part of the output from the network device are you looking for? And what part of the output identifies whether something is working or not working? As an example, if you were on a Cisco device that runs IOS and you wanted to determine if an interface was working, what would be the first command you would execute? Perhaps, “show interface ethernet-xyz”. From that command, you could look for a line of output like “line protocol is up,” which would help you determine that at least the interface is up.

That common scenario is how Itential’s command templates work in their simplest terms. A network engineer can enter commands in the application, just as they would on the command line, and parse the output for the correct information to determine whether that command passes or fails the determined test. As the network engineer starts building these steps, they never have to leave the application because they can execute commands on devices in real-time and see the output it generates. This lets them define the rules based on that output.

These command templates may start out as a single test, but your network team will find that you can run multiple tests over the output of a single command, and string together multiple commands and tests in a single command template. Because this is a very natural method of expressing that network tribal knowledge, they will be able to create command templates quickly, test them out, and iterate over them without the need to learn to write any code — all while documenting it and making it available to other team members.

Leveraging Tribal Network Knowledge to Build Reusable Automations

Once your network team has transferred their tribal knowledge to command templates, they become logical assets that are used in automations. In a single step within a workflow, a command template can run on any device in the network in real-time, and it will return a pass or fail condition with full details on the device output.

Command templates can become the heart of a workflow that can be used for a myriad of network tasks including troubleshooting network problems, identifying whether interfaces are up or down, whether routes exist or not, or to build automations that run across many devices to accomplish time consuming and repetitive tasks, like finding where a MAC address is on the network.

As your automations progress into making network changes across different network domains, these same command templates will become critical in determining pre-check conditions (to determine that it’s ok to make changes) and post-check conditions (whether those changes just made are working and nothing else on the network has been impacted). Your core network team can participate in these automation efforts immediately without any prior experience in writing code and can rapidly build automations that the whole team can run, while feeling confident that they will execute successfully.

Seems too good to be true, doesn’t it? Let me show you that’s not the case. You can see it for yourself in this recent demo I did showing step-by-step how you can put this into practice and turn tribal network knowledge into organization-wide automation with Itential’s command templates.