The Network is Down! 4 Words No Network Engineer Wants to Hear & How to Avoid Network Outages with Automation

The network is down.

These four words (or any of several iterations of this phrase) can cause instant panic in end users and an immediate adrenalin rush to a network team. Any outage, especially those that occur at the most inopportune moment (thank you, Murphy), will try anyone’s patience and always seem to take longer to restore than it should. However, many times there are outages that occur and the ugly truth is that they do take far longer to resolve than necessary.

Let’s take a look at what happens between network engineers when this all too frequent phrase is said.

Disclaimer: The following scenario is a dramatization based on real events, but names and details have been changed to protect the identities of the networking practitioners involved.

Network Engineer 1: We’ve received a call that the network is down at site XYZ.

Network Engineer 2: What’s the IP of their router? Let’s try to ping it.

Network Engineer 1: Let me look it up. (Searches spreadsheet.) It looks like it’s 192.168.111.34.

Network Engineer 2: That can’t be right, we’ve migrated off of those subnets, are you looking at the right spreadsheet?

Network Engineer 1: You’re right, this was the previous one. I need to get rid of that. (Opens correct spreadsheet.) Ok its 10.12.11.34.

Network Engineer 2: That sounds right. Hmm, it’s not responding. I’ll continue to troubleshoot this, but you go ahead and get a new router ready in case this one has died.

Several minutes elapse with several more calls coming in from management and onsite staff, wondering how much longer it will be before the outage is fixed.

Network Engineer 2: Yeah, we tried power cycling the router at the office and it’s not coming up, looks like it’s dead. Did you get another router ready?

Network Engineer 1: I have another router here.

Network Engineer 2: Is it the same model?

Network Engineer 1: I’m not sure, it wasn’t noted in the new spreadsheet, but this is the one we’re deploying now.

Network Engineer 2: Hmmm, ok, let’s just use this one. We’ve got to get this thing back up, people are calling non-stop now. Ok, do you have a copy of the router’s config?

Network Engineer 1: No, I haven’t touched that router since it was deployed. Do you have one?

Network Engineer 2: (Deep sigh of frustration) I don’t know, let me check my laptop. Nothing there, I’ll check my desktop.

More precious time elapses, and then finally…

Network Engineer 2: Well, I found a backup. It’s from the beginning of last year, but this will have to do. We have to get this site back up, and whatever we’re missing we can try and figure out later.

Avoid Outages with Automated Device Inventory

It’s an understood fact of life that outages will occur due to any number of reasons – network hardware fails, configurations get changed or wiped out, and backhoes actively seek out buried cables (once again thank you, Murphy). In some cases, networks are built redundantly to withstand certain outages, but in other cases an outage means service is completely down. In those cases, recovery must be accomplished quickly, effectively, and with absolute confidence that you are restoring the network device with the same configuration it had before the failure.

In order to guarantee that this occurs, it’s important that network teams have modern configuration and compliance tools that provide an inventory of devices, from across your entire network, that is up to date and always accessible. These tools should also ensure that a backup configuration of every device in this inventory list is consistently logged, according to a pre-defined schedule and located where members of the network team can find it the moment it’s required. These are foundational processes that every enterprise should have implemented as they provide an insurance policy when a network disaster strikes.

Network teams shouldn’t waste minutes or hours figuring out the details of how to access a device and then track down a configuration backup for the device that may or may not even exist. By rebuilding a network device’s configuration based on an older backup file, there are several potential problems at play:

• Missing security policies that can leave the device vulnerable to DoS or hacking attempts.
• Incomplete security ACLs that now allow access from previously unauthorized networks.
• Routing features that are misconfigured or no longer exist.
• Performance or redundancy functionality that is now absent.

Itential’s Configuration Manager Provides a Federated Inventory for the Latest Backup

In the rush to restore network services, many network teams are forced to take “the best worst option” because they lack the tools that can help them restore services quickly and effectively. They will bring the device back online using whatever configuration backup file is available and try to piece together what’s missing based on context, experience, other device configurations, memory, and pure willpower. Despite their best attempt at restoring a lost configuration, anything less than having the most recent configuration backup file before the outage amounts to guesswork.

Using Itential’s Configuration Manager application, network teams have a modern tool that supports a federated device inventory. That means any device in the network, whether it’s a traditional CLI-managed device, a device managed through a network controller, or even a cloud-based network service like an AWS VPC, is identified and can easily be located in the inventory list. For each device in the list, complete configuration backups can be taken at any time and stored in single location that anyone on the network team can search for and find in a matter of seconds.

In addition, Configuration Manager can also automate configuration and compliance across both cloud and on-prem networks, automate remediation of devices, and validate proposed configuration changes to ensure compliance before they are applied to a device. To learn more about Itential’s approach to modern configuration and compliance click here or check out and register for our webinar series on “Modern Network Compliance.”