Shining a Light on the Invisible Issues Blocking Your Automation Goals

Most of us in networking have had a situation where, for one reason or another, trying to reach our objective has turned into a slog.

Maybe it’s a technical deficiency of a system that you’re having to work around. Maybe it’s a source of truth that you thought was pretty consistent but is in fact a complete dumpster fire. Or maybe it’s that Joe on this cross-functional team is generally a pain to work with and keeps throwing up hurdles left and right.

In some situations, the issue is obvious (yeah, we’ve all worked with Joe before). In other situations, the more obvious challenges can overshadow some of the more subtle ones. And for many of us, we have grown so accustomed to these challenges and the workarounds we use that it’s difficult to clearly see what’s hindering us.

Automation and orchestration goals are no exception to these situations. In most businesses, the truly transformational and value-driving goals cross many teams and domains, heightening the complexity and number of opinions.

Working with our customers at Itential, I’ve seen that it takes people with a vision to become the champions of automation and solve the impossible challenge. They are the leaders within their organization pushing their peers and colleagues to innovate and produce meaningful, valuable solutions. These champions have learned to point to the invisible issues, as we’ve come to call them, and name them and overcome them.

I’m in a somewhat unique position — I get the chance to work with many different network teams, not just one. I’ve seen many automation and orchestration journeys play out in many different ways. With the benefit of that bird’s eye view, I’d like to share patterns and actionable insights: the trends and invisible issues facing automation efforts, and the best strategies for overcoming them.

Not all teams are created equal in terms of their ability to create and consistently manage sources of truth for inventory and when there is no mandate to use just one, each team has a tool of choice. If a team needs to know inventory data to be able to connect to relevant devices for new service, they have two choices: create a new inventory source of truth and build from scratch, or find a way to take what you can from all of the sources of truth with their varying levels of consistency.

The first option can sometimes mean delaying value realization for a year to focus on what some might consider hygiene activities. The latter can mean using NetBox for one team for inventory, CMDB for another team, SolarWinds for yet another team and a half-complete, error-ridden spreadsheet for everything else.

When champions talk through the art of the possible with their organization’s leaders on how they are going to drive transformation, these are not the activities they want to get stuck on. But left unsolved, inconsistent inventory can slow progress, create friction (“if Joe and his team would just clean up this mess!”), and create new work that was never scoped and is difficult to explain to organization leaders.

Have you ever read a message and immediately knew it wasn’t written by the person who sent it? I’m looking at you, pioneers of ChatGPT!

It’s because everyone writes and speaks a little differently. The same is true with config. I overheard a conversation the other day with a customer leader related to their Day 0 and Day 1 configuration consistency objectives and brownfield network challenges. One team member said he could tell who deployed the devices based on the configurations, which “had their fingerprints all over it!” Each person on the team has their own word doc (or device) that has the ‘right’ configuration that they use for new devices deployed on the network.

Champions that are charged with auditing configuration standards on the network have a long and hard road in front of them. Having seen many of our customers go through this journey, we’ve seen how much of a blocker configuration consistency can be to future network automation goals. We’ve put together an in-depth, step-by-step walkthrough of how to implement Itential’s Golden Configuration methodology for network consistency — go through the steps here.

Network experts are constantly being asked to learn new skills and transform the way that they interact with the network. A common hurdle is finding the “needle in a haystack” — aka a meaningful endpoint that allows them the functionality they need for their automation or orchestration.

Many vendors do an excellent job of creating and exposing their API endpoints to enable automation, but searching for the endpoint in a 150 page API document is always going to be demoralizing. While it’s useful to learn how to effectively identify the right endpoints to deliver information needed for an automation, it’s not a requirement to start automating — individuals can learn over time. In the meantime, it’s tedious at best and a serious blocker at worst.

For leaders hoping to keep the expertise of those who know the network best at the core of their team, solving the API navigation challenge is essential to achieving network goals.

Many of our champions are charged with creating a Center of Excellence (CoE) and crafting the standards for their organization to bring together cross-functional expertise and participation around automation. In most cases, these champions have very complex environments to navigate. Often they are required to build a framework which other internal customers are mandated to use in order to accomplish their automation goals. While that sounds simple at the outset, it can quickly become complex when each team brings their own skillsets, goals, and requirements around release management and changes to the network.

At the end of the day, champions cannot say to leadership that they will not support a strategy that a team is using if it has produced results or represents core functionality that cannot be simply turned off. If Team A has elected a strategy around python scripts, they must have a mechanism for support in the framework. If Team B has no background in scripting, they must be offered a mechanism through which to contribute. We have seen time and time again that if every team is forced to adopt a single contribution strategy, automation adoption across the organization will suffer and subsequently impact the velocity of the organization.

While many of our champions function as a group of specialists who will gather requirements and build the network services for their internal consumers, having an option that allows others to contribute in a standardized way is key to increasing the organization’s velocity as a whole while also enforcing some guardrails around technical and functional requirements of the contributions. The tools around network automation are diverse, and the contribution framework provided by these teams also needs to reflect this diversity as well.

You don’t have to be reminded that at the end of the day, if the security team has flagged something in their program as insecure, it will need to be remediated immediately.

Automation can be an incredible tool for organizations to transform the way they do business, but it can also be another medium through which malicious actors can exploit vulnerabilities. Valuable services and network agility can only go so far if the overall program is insecure, and in the same way that organizations attempt to hedge against security breaches, champions will also need to consider the reputational damage that is incurred when their program is flagged for a lack of best practices around security.

The most important priority is to ensure your automation and orchestration strategy aligns with security needs. That usually means two sets of things — first, the ability to make software upgrades, push vendor patches, update firewall policies, etc. as quickly as possible. Second, the ability to integrate network workflows with any security platforms in order to communicate data back and forth, trigger security alerts, or trigger remediation workflows in response to alerts. Adopting automation and orchestration technology that enables and supports your security needs is essential.

The champions that we have seen achieve the most success and visibility within their organization and beyond are those that have always got their finger on the pulse of value realization.

Many teams gather data before they begin development of a network service that helps make the case for its prioritization (e.g. “we do this process X times per year, it requires at least two engineers during the change window for each site, we have 30 sites or perhaps we’ve had 75 outages in the past year due to this specific activity not being done correctly and it has cost us X$”).

While this is a great best practice and should absolutely be a standard process, value shouldn’t just be a topic for requirements gathering and prioritization. Our customers don’t always get to choose the work (that old adage the squeaky wheel gets the oil comes to mind), but we do get to control how success is measured and how those metrics are communicated across the organization. It is not enough to do good work; truly successful champions showcase the accomplishments and value realization of their team whether requested or not.

The more complex your environment, the more likely you’ll face complex ‘invisible issues.’

But that complexity is exactly why we need automation in the first place.

So for automation champions, driving successful automation adoption requires identifying and solving large-scale challenges that range from technical realities (inconsistent inventory) to business and reputational needs (demonstrating automation ROI). It’s a complex problem, but it’s one that’s made easier when people build the right strategies, adopt the right platforms, and build the right culture for success.

What invisible issues have you seen in your team or organization? How have you worked to solve them? Reach out — it’s always great to hear how teams are innovating to overcome key automation challenges!