Recent Security Advisories released this month by Cisco PSIRT have once again put security vulnerabilities top of mind for network teams. At least twice a year, Cisco releases advisory Bundles, but those only contain the planned vulnerabilities. On top of that, there are unplanned vulnerabilities. Auditing all your devices for vulnerabilities and remediating any issues is always important, but it’s far too easy for it to slip down the list of priorities.
When it comes keeping up with your networking vendor’s security advisories, there are probably a lot of “skeletons in the closet.” These skeletons come in the form of network routers and switches that haven’t had their configurations backed up or reviewed — let alone updated — in a long time. Sometimes that time stretches beyond the tenure of most of the team. (We won’t even think about what version of firmware they might be running — that’s another skeleton in a different, but related, closet.)
And to be fair, this backlog doesn’t exist because the network team lacks motivation, or skill, or knowledge. It usually comes down to the fact that everything on a day-to-day basis is moving so fast, there’s a need to prioritize everything coming at us, and these ‘maintenance’ tasks tend to get pushed down the list. Outages, performance issues, new deployments, and day-to-day changes that keep the business running are at the top of the list. If there’s time to read the latest advisory and check out the most important network devices, we will get around to it.
Of course, there’s a way for config compliance to immediately leapfrog to the number one spot — critical advisories that have been exposed because of an outage or security breach and make the rounds in the media. The phones start ringing, Slack and Teams channels don’t stop pinging, and network teams start creating time to scour devices to make sure they aren’t exposed and then generate reports to validate that is the case.
Ok, I didn’t mean to panic anyone or bring up past trauma, but this scenario plays out far too often. Everyone who has felt this pain would agree that there needs to be a better, faster way to go from vendor advisories to full network audits and reporting. With a large number of network devices, automation and orchestration are the key to this, but there are some things to consider when looking at solving the problem with a new set of tools.
Auditing, Reporting, Remediation: How Do I Get There?
Successfully automating and orchestrating config compliance means doing several things. Ask yourself these questions to build a strategy that checks all the boxes to help you get where you want to be:
Velocity, or How fast can we integrate with our network devices?
This is not just identifying these devices, but also “speaking the language” of the devices, whether that’s a specific vendor CLI, and API, or through some other management protocol.
How do we build rules and templates that express the advisory process?
Network engineers can quickly read an advisory and understand the commands and process they specify, but the bigger issue is how to express that into the solution so that it can be executed and AT SCALE!!! Skills gaps that require learning programming languages (or complex data formats) can slow down everything and keep multiple people from participating and sharing the workload.
How fast can we start running audits and generate reports?
Time is of the essence, and closing down windows of opportunities is critical for those critical advisories, so turning templates into real action is a big part of the solution. Pay attention to scaling and any needs for infrastructure to support your scale and features that can help do more in less time, like parallel audits, prioritization, and staggered scheduling.
How will we remediate issues when they are found?
Usually this means “Can we automatically remediate the issue?” A lot of this will depend on how critical the issue is, how critical the device with the issue is, your organization’s processes, and your team’s ability to confidently automate changes. Some changes are so critical they need to be fast-tracked, but in most cases, it’s safer to follow approvals and change management processes to ensure changes are correct.
Your solution should give you the flexibility to automate changes confidently and follow the required processes around change request and approvals by orchestrating these tasks together to reduce the time of exposure. It’s possible to automate changes for the next scheduled maintenance prescribed by the CAB upon approval, but not every solution can do all of this together (but they should).
Some other thoughts:
Configurations change, and sometimes they change back (Skeletons rising again?). Once dismissed these configurations should not come back, so ensure that continually scanning can be automated on a schedule, but just as important, make sure different teams are aware that these scans are occurring and they have the reports that have data that are relevant to them. This means that it would be great for your solution to enable report generation in different formats and deliver them to different teams that may use different systems, like email, Slack, MS Teams, ServiceNow, etc.
Start Doing Things Differently: It’s Time to Embrace Orchestration
Not all advisories are the same, some will point out configuration commands to avoid and others may require certain operational information be verified as part of a process. A solution should support both configuration and operational methods and processes and be simple enough for network teams to understand, build, and modify over time.
The goal should be to minimize your window of exposure—as soon as vulnerabilities are released, how fast can I scan and audit across all devices, how fast can I generate reporting, and then how quickly do we want to remediate for each set of issues?
My network is not just multi-vendor, but also cloud. If you are old enough to remember a single vendor network, you might just be a skeleton yourself! Just kidding, but that’s just not reality anymore, when even a single vendor has multiple flavors of devices, CLIs, software and differences (Does that sound familiar?!?!). The ideal solution is one that can operate across different network vendors and cloud platforms, using multiple methods of integration to provide a consistent and flexible set of features that multiple teams can use.
This is certainly not a complete list, and you’ve probably realized some of these considerations already, but as networks continue to grow and bring more complexity and network vendors are moving quickly to identify problems and issue more advisories it is becoming a requirement for network teams to figure out how to operate networks differently.
At Itential, we can help you to leverage automating network operations, like configuration audits, reporting, and remediation as part of an orchestrated workflow that can integrate with your IT systems for change management, team notification, email (and more) in order to overcome these modern challenges and confidently slay some pretty nasty network skeletons and ensure they never return again. If you’d like to see just how fast this can be done, check out this recent webinar where I demo step-by-step how you can quickly go from PSIRT Advisory to Network Audit and Reporting with the Itential platform or check out our full range of configuration management features.
Joksan Flores
Senior Solutions Engineer ‐ Itential
Joksan Flores is a Senior Solutions Engineer at Itential. Joksan's passion for putting both systems and software together lead him to spend 10 years as a Networking Architect at Cisco prior to Itential. Throughout his career, Joksan has supported enterprises and service providers with massive customer bases to solve their IT challenges, designing cloud peering connectivity, WAN, and data center networks. While helping organizations solve complex network challenges, Joksan always finds a way to leverage automation, either by devising ways of making work more streamlined or by helping customers achieve their project goals faster.
