According to Gartner Insights, a staggering 80% of all incidents stem from planned and unplanned configuration changes.
“Through 2015, 80% of outages impacting mission-critical services will be caused by people and process issues, and more than 50% of those outages will be caused by change/configuration/release integration and hand-off issues.”
– Gartner RAS Core Research Note, Ronni J. Colville, George Spafford
Despite that, most approaches to troubleshooting and handling incidents are reliant on alerts and performance monitoring rather than asking quite literally, any recent changes in configuration.
In other words, no matter what, the majority of outages almost always have something to do with configuration changes. But that fact is often ignored, understandably due to a lack of actionable configuration data.
Enter SIFF, a network configuration management system that manages rapid changes in DevOps, network, and IT operations offering visibility into IT environment changes by providing an audit trail for all configuration changes. The SIFF platform scrutinizes data from a diverse range of sources and then distinguishes between planned and unplanned configuration changes, aiding in the isolation and identification of complex incident root cases.
Below, we outline the best uses of configuration data and why harnessing the power of SIFF will help minimize outages, accelerate troubleshooting faster, and automate policy compliance.
1. Utilize Configuration Data for Faster L2/L3 Support Resolution
In most scenarios, L2 / L3 support is confronted with outages or disruptions and struggles with limited visibility into the configuration changes that have transpired across their network infrastructure. They’re forced to diagnose outages without the ability to access valuable information siloed within the various departments.
Think about it another way: Imagine a doctor attempting to diagnose a patient without permission to ask questions about what has recently changed for the patient – their living conditions, what they’ve been eating, and other pertinent questions. There’s no structure or logical decision-making to narrow the scope of the problem.
By understanding that the majority of outages are attributed to configuration changes, it’s quite obvious that change data can help support teams identify the problem. Still, the information you need is most likely fragmented across various tools and silos within the organization.
Even if support teams start asking questions about recent configuration data, it’s still a bit of a needle-in-a-haystack approach. It requires bridge calls across departments and teams – increasing resource waste and taking up valuable time to inevitably find resolution.
SIFF does this by collecting all configuration changes and making the data easily accessible through a change activity stream where users can correlate service-impacting changes with actual change details and DIFF comparisons. The synergy between real-time visibility and configuration data in a solution like SIFF contributes to faster resolution of Level 2 and 3 support in a few ways:
- Isolate and identify the root cause of service outages by analyzing recent configuration changes that may have impacted the service
- Visibility to configuration changes across multiple functional groups in a single place (networks, servers, applications, cloud, containers, security, etc.)
- Identify planned vs. unplanned changes – Ad-hoc or unauthorized changes are good candidates for analysis
- Configuration change details and history – what, when, who
2. Enforce Configuration Policy and Security Practices
Security and network engineering teams often implement a configuration strategy called the Golden Template. This approach aims to enforce configuration policies by comparing configuration elements from a device against the Golden Template. In practice, the device-centric template approach quickly becomes difficult to manage and maintain due to the numerous yet valid exceptions that must be catered for e.g. location, function, customer, or service.
An alternative approach implemented by SIFF is to align compliance rules directly with the configuration or security requirements rather than by device or vendor. Additionally, SIFF collects and monitors configuration data from all sources regardless of how they are configured and managed (EMS, scripts, automation, UI). Having a single place where you implement and verify configuration policies allows you to manage and scale your configuration governance.
SIFF offers a solution by automating configuration policies. This ensures alignment with security guidelines and policies, creating a unified configuration monitoring approach for network integrity – which is essential to ensure consistent process and centralized integration.
It achieves this through the following approach:
- Configuration Policy Monitor: Continuously examining all configuration changes against defined policy rules, integrating with SIEM, Fault Management, or ITSM systems to ensure that the process is not only automated but centralized.
- Search and Report on Policy Violation Changes: Search config change activity and generate audit and compliance reports, helping review planned vs. unplanned changes.
- Configuration Validation Automation: Comparing device running-config vs. startup-config, helping verify the integrity of device configs while comparing configs between components in a cluster.
3. Track Changes: Planned vs. Unauthorized
Traditionally, the change management and review process helps provide a check-and-balance to prevent incidents and outages from occurring. Simply put, a change request describes the intention of the change and then outlines how the changes will be made.
However, when a change is carried out, inadvertent errors frequently happen – caused by both human action and automation. That’s because the conventional approach overlooks something critical: permission is given to make the change, but how the change is implemented is not monitored or reviewed. In other words, imagine a coach of a sports team calling an offensive play, but then not watching the execution.
If there’s an error in the implementation, it is extremely difficult to review and identify the fault since the config changes were not captured. Requiring the technician to manually capture the changes made is prohibitively time-consuming when changes are often applied to many devices and services.
The significance of this challenge is amplified by the inherent resiliency within networking environments. Problems from a change might not appear immediately but rather accumulate and pop up later, causing disruptions days or even weeks after implementation.
The SIFF platform takes an innovative approach to the change and implementation review process. SIFF automatically records all configuration changes and correlates them with change requests to identify planned vs. unauthorized changes. This practice not only bolsters security but also assists in compliance and regulatory requirements.
SIFF also gives you these benefits:
- Safety Net for Infrastructure Changes: Acting as a safety net, ensuring the security and integrity of your infrastructure changes.
- Historical Configuration Data: Examining configuration data historically, helping facilitate forensics, review, and compliance.
- Automated Change Association: Changes seamlessly linked with change requests, allowing for easy and early review to minimize errors and preempt incidents.
4. Identify Devices with Vulnerable Configuration
Conventional tools have primarily catered to backup and restore functionalities, with limited accessibility to configuration data. Some NCM tools offer the capability to detect devices susceptible to known vulnerabilities or common vulnerabilities and exposures (CVE) – but they fall short in other crucial areas.
Many cases demand the identification of devices with non-optimal security configurations that don’t qualify as CVEs, such as using HTTP instead of HTTPS or employing default passwords. Given that, the necessity to search, analyze, and report on configuration specifics occurs frequently, yet most NCM tools lack this functionality.
SIFF addresses these limitations with detailed reporting of configuration details, providing the following:
- Comprehensive searches and insightful analysis
- Efficient identification of vulnerable configurations, enhancing network security
- Adherence to best practices by providing the means to identify and address potential security weaknesses beyond CVEs
5. Ensure IT Audit, Reporting, and Compliance
Achieving IT and security compliance should be a priority for all teams – and it encompasses a variety of processes. Many security standards emphasize the need for visibility into all configuration changes, resulting in a myriad of configuration backup tools.
Data is essential for driving automation, so in order to ensure compliance, teams are stuck dealing with an inefficient process to perform reviews and audits. Managing multiple systems prevents the establishment of automated policies and best practice enforcement.
As a solution, SIFF provides a unified repository of your configuration data, helping you and your team streamline and perform regular audits and security reviews. Think of it as a one-stop shop for all your compliance needs.
- Conduct thorough searches, comparisons, and analyses of detailed configurations
- Generate reports for audits with ease
- Showcase audit trails and historical logs to meet compliance certification
6. Use Post-Incident Analysis and Security Forensics
If you’re not analyzing what you’re doing, what are you doing? Following an incident, it’s vital for teams to examine what happened and how it can be prevented in the future.
Despite the fact that configuration changes are crucial for understanding incidents or security breaches, the configuration history provided by most NCM tools’ is typically insufficient – often updated on a daily or batch-based basis. This time window means critical changes might be missed, making it challenging to understand timelines for incidents.
SIFF integrates with configuration data on a granular level, tracking time-based changes on a more micro level. After an incident, this allows teams to get further into the nitty-gritty, understanding what broke and why. It does this by:
- Utilizing granular time-based change data for accurate correlation of changes across teams and technologies.
- Leveraging service dependency information in larger environments to identify changes impacting specific services.
- Enhancing post-incident analysis and security forensics with precise identification of change-related impacts.
7. Enable Automation in a Changing Environment
Despite the value of automation, resistance towards automation initiatives comes from the idea that “bad data” hinders progress. The irony is that data is the major driver for improving automation, so limiting the scope to simpler tasks reduces opportunities. Further, waiting until the data is cleaned up creates an environment lacking motivation to automate.
For instance, if a device configuration is using a “public” SNMP community string, a policy based on data can trigger an automation to update the device to the correct community string.
The point is that the interplay between data and automation is the key driver for action – whether manually or fully automated.
- Recognize data’s pivotal role in driving effective and forward-looking automation initiatives.
- Embrace data as the foundation for adaptive and impactful automation strategies.
- Highlight data-driven approaches to enable automation in changing environments.
8. Synchronize Inventory Management Systems
Network engineering teams often rely on inventory management systems to plan and manage networks. However, as you know, these systems can become outdated due to network changes made independently by various teams using different tools. Plus, multiple types of inventory systems can further complicate the situation – and existing tools lack efficient ways to consolidate data.
As a solution, SIFF offers an inventory management system, and a consolidated repository for configuration data from various sources. The consolidation eliminates challenges in keeping inventory systems updated and synchronized, streamlining network management and increasing accuracy. You will also be able to:
- Use data-driven automation for inventory data reconciliation
- Address challenges stemming from multiple management systems
- Overcome integration difficulties caused by having consolidation data across multiple sources
Utilize Configuration Data to Eliminate Outages and Improve the Future of Your Organization
Configuration data is not just a byproduct of network activities – it’s one of the most useful tools for effective IT management. Despite that, traditional approaches to problem-solving in this world ignore it as a useful piece of information.
But that’s because there hasn’t been a simple way to access configuration data. For the longest time, it’s been stuck in different silos, forcing troubleshooters to look for solutions similar. And to that, we say – there’s got to be a better way.
And that’s what we’re here for.
Unlock the full potential of your data with SIFF’s configuration data, and finally say goodbye to endless bridge calls.
