Modern enterprise networks are complex ecosystems that demand intelligent automation for effective management. However, network automation data collection presents the most fundamental challenge that organizations must solve first. The journey from raw device outputs to actionable insights requires systematic data collection processes that can handle the scale and complexity of modern network infrastructures.

The Network Automation Data Collection Challenge

Managing modern networks requires automation, but establishing reliable network automation data collection processes is often the first and most critical hurdle. Network engineers face the daily reality of gathering information from dozens or hundreds of devices, running commands manually, and interpreting outputs that were designed for human consumption, not automated data collection systems. This approach doesn’t scale, introduces human error, and prevents the systematic analysis necessary for strategic network decisions.

The Importance of Structured Data

Structured data forms the foundation of effective network automation. While raw command outputs provide information, they lack the consistency and format needed for programmatic analysis. Without structured data, network teams cannot build reliable automation workflows, perform trend analysis, or generate meaningful reports for management. The transformation from unstructured text to organized, query-able data is the critical first step toward intelligent network operations.

Our Approach

At Netodata, we’ve developed a practical demonstration that showcases multiple approaches to network automation data collection, parsing, and reporting. Repository exemplifies our philosophy of providing flexible, scalable network automation data collection solutions that address real-world challenges. Our Simple Facts toolkit, available at:

Git: Simple Facts repository

Part 1: The Problem – Network Automation Data Collection and Structuring

Network engineers face several fundamental challenges when implementing network automation data collection:

• Device Diversity: Modern networks consist of devices from multiple vendors including Cisco, Arista, Juniper, and others, each with unique command structures and output formats. Effective network automation data collection must account for these variations while maintaining consistency in the collected data structure.

• Manual Processes: Traditional network management relies heavily on manual data collection processes. Engineers SSH into devices individually, run commands, copy outputs to spreadsheets, and manually analyze results. This approach prevents scalable network automation data collection and introduces significant operational overhead.

• Inconsistent Outputs: Raw command outputs are designed for human consumption, not machine processing, creating blocks of text that require complex parsing to extract meaningful information. Even within a single vendor’s platform, output formats can change between software versions.

• Scale Challenges: As networks grow, traditional approaches become unsustainable. Implementing efficient network automation data collection becomes critical when managing hundreds of devices where manual processes could take days or weeks to complete.

The Structured Data Requirement – Why Raw Text Isn’t Sufficient

Raw command outputs present several problems for automation:

• Inconsistent formatting makes parsing unreliable
• Human-readable layouts waste computational resources
• Text-based outputs are difficult to query and analyze
• Version differences between devices can break parsing logic

Machine-Readable Formats

Effective network automation requires data in formats that programs can reliably process:

• Consistent key-value structures
• Standardized field names across device types
• Queryable data that supports filtering and analysis
• Format stability that survives software updates

Enabling Programmatic Decisions

Structured data provides a systematic way to convert raw text into organized information, making it easier to manage and use within various network automation tasks. With properly structured data, network teams can:

• Build automated compliance checking
• Perform trend analysis over time
• Generate executive-ready reports
• Implement predictive maintenance

Common Challenges

Network automation projects frequently encounter these obstacles:

• Authentication Complexity: Managing SSH access and authentication across multiple devices requires careful credential management and often involves different authentication methods.

• Device Type Variations: Different network operating systems require platform-specific handling for connection methods, command syntax, and output parsing.

• Output Parsing Difficulties: Converting unstructured command outputs to structured data traditionally requires complex regular expressions or custom parsing logic.

• Scalability Requirements: Effective network automation data collection across large networks demands parallel processing capabilities and efficient error handling to manage inevitable connection failures or device timeouts.

• Inventory Management: Maintaining accurate device inventories with current credentials, IP addresses, and device types becomes a project in itself.

Part 2: Our Solution – A Multi-Approach Network Automation Data Collection Tool

Overview of the Simple Facts Project

Our Simple Facts toolkit demonstrates four distinct approaches to network automation data collection and processing, each highlighting different aspects of modern network automation. This isn’t just academic exercise – these approaches reflect real-world scenarios where different technical requirements demand different network automation data collection strategies.

Project Philosophy: The toolkit embodies our belief that network automation should be:

• Modular: Different components solve specific problems
• Extensible: Easy to adapt for unique requirements
• Educational: Clear examples that teams can understand and modify
• Production-Ready: Robust error handling and professional output

Complete Workflow Architecture

The project follows a logical progression: Data Collection → Parsing → Reporting. Each stage builds upon the previous, creating a complete pipeline from raw device access to executive-ready reports. This architecture mirrors the approach we use in client network automation data collection engagements.

Approach 1: Direct SSH with Paramiko

Technical Implementation

Our simple_facts.py script demonstrates direct SSH connectivity using Python’s Paramiko library. Paramiko provides a handy Python SSH library for connecting to devices, offering fine-grained control over the connection process.

Key Features

• Keyboard-interactive authentication support
• Direct command execution and response handling
• Minimal external dependencies
• Full control over connection parameters

When to Use This Approach

Direct Paramiko implementation works best for network automation data collection when:

• Working with devices that require specific authentication methods
• Needing precise control over connection timing and error handling
• Integrating with existing systems that don’t support higher-level frameworks
• Learning fundamental network automation data collection concepts

Code Architecture Highlights

The implementation handles connection establishment, command execution, and graceful disconnection while providing detailed logging for troubleshooting. Error handling ensures that authentication failures or connection timeouts don’t crash the entire network automation data collection process.

Approach 2: Network Automation with Nornir

Framework Advantages

Nornir is a Python framework explicitly made for network devices, developed and maintained by the same people who created NAPALM and Netmiko. Our nornir_facts.py implementation showcases the framework’s strengths in production environments.

Scalability Features

Nornir is multi-threaded and significantly faster compared to serial execution approaches. Key capabilities include:

• Parallel task execution across multiple devices
• Built-in inventory management system
• Plugin architecture for different connection methods
• Robust error handling and result aggregation

Production Benefits

Nornir allows multi-threaded task execution without using a domain-specific language, built with Python and designed to be used with Python. This approach provides network automation data collection capabilities including:

• Inventory-driven automation that scales with network growth
• Plugin ecosystem supporting various device types
• Consistent task execution model
• Professional logging and result handling

Inventory Management

The Nornir approach includes structured inventory files that separate device information from automation logic. This separation enables teams to manage device credentials and groupings independently from the network automation data collection scripts.

Approach 3: Structured Data Parsing with TextFSM

The Parsing Challenge

TextFSM provides a systematic way to convert raw text into structured data, making it easier to manage and use within various network automation tasks. Our structured_facts.py demonstrates how to transform vendor-specific command outputs into consistent data structures.

TextFSM Integration

TextFSM parser helps extract structured output from show monitoring commands sent to network devices, allowing information to be easily extracted without using regular expressions. The implementation includes:

• Automatic template selection based on device platform and command
• Error handling for unparseable outputs
• Fallback mechanisms for unsupported commands
• Integration with community template libraries

Template Ecosystem

The ntc_templates repository on GitHub contains a collection of pre-built templates for many common network commands across different vendors. This community resource significantly reduces development time by providing tested parsing templates for standard network commands.

Vendor-Agnostic Benefits

The TextFSM approach enables consistent data structures regardless of the underlying device platform. A structured interface list looks the same whether it comes from Cisco IOS, Juniper JUNOS, or Arista EOS, enabling unified analysis and reporting.

Approach 4: Comprehensive Reporting

Professional Output Generation

Our report.py script transforms collected and parsed data into professional reports suitable for both technical teams and management. The reporting engine includes:

• Markdown generation with company branding
• Executive summary sections highlighting key findings
• Detailed technical appendices for engineering teams
• Customizable report templates

Below we have a report example generated by the script.

# Network Discovery Report

## Overview

* **Report Generated**: 2025-05-24 14:02:35
* **Data Collected**: 2025-05-24 11:10:35
* **Total Devices**: 3
* **Total Interfaces**: 8
* **Total LLDP Neighbors**: 18

## Device Inventory

### Device Types
* **Arista**: 2

### OS Versions
* **4.29.9.1M-38626060.42991M (engineering build)**: 2

## Interface Statistics

### Interface Types
* **Ethernet**: 4
* **Loopback**: 2
* **Management**: 2

### Interface Status
* **up**: 8

## IP Addressing

### router1
| Interface | IP Address |
|-----------|------------|
| Ethernet1 | 10.1.1.1/30 |
| Ethernet2 | 10.1.2.1/30 |
| Loopback0 | 1.1.1.1/32 |
| Management0 | 172.20.20.11/24 |

### router2
| Interface | IP Address |
|-----------|------------|
| Ethernet1 | 10.1.1.2/30 |
| Ethernet2 | 10.1.4.1/30 |
| Loopback0 | 2.2.2.2/32 |
| Management0 | 172.20.20.12/24 |

Data Analysis Features

Beyond simple data presentation, the reporting module performs analysis:

• Network health assessments based on interface states
• Compliance checking against organizational standards
• Trend identification across multiple collection cycles
• Risk assessment highlighting potential issues

Business Value Communication

The reports translate technical data into business language, helping management understand network investment needs, compliance status, and operational efficiency metrics. This translation is crucial for securing automation project funding and demonstrating ongoing value.

Part 3: Implementing the Solution

Setting Up the Environment

Python Environment Configuration

The toolkit requires specific Python packages that work together effectively:

# Core automation libraries
pip install paramiko nornir nornir-netmiko
pip install textfsm ntc-templates
pip install pyyaml jinja2 markdown

Inventory Structure Design

Proper inventory organization enables scalable automation:

# hosts.yaml structure
router_001:
  hostname: 192.168.1.10
  platform: cisco_ios
  groups: [core_routers, building_a]

switch_001:
  hostname: 192.168.1.20
  platform: cisco_ios
  groups: [access_switches, building_a]

Security Best Practices

Using environment variables for credential management provides security while maintaining automation capability:

export NET_USERNAME="automation_user"
export NET_PASSWORD="secure_password"

Running the Tools

Network Automation Data Collection Process

Each approach provides different capabilities for network automation data collection:

1. Paramiko Direct: Individual device connections for testing and small-scale collection
2. Nornir Framework: Parallel network automation data collection across device groups
3. TextFSM Integration: Structured parsing of command outputs
4. Report Generation: Professional documentation of findings

Execution Workflow

The typical network automation data collection execution follows this pattern:

# Collect raw data using network automation
python simple_facts.py

# Parse with TextFSM for structured data collection
python3 structured_facts.py output/simple_facts_YYYYMMDD_HHMMSS.json

# Generate reports from collected data
python3 report.py output/structured/structured_facts_YYYYMMDD_HHMMSS.json

Error Handling Strategy

Production implementations must handle common failure scenarios:

• Authentication failures
• Network connectivity issues
• Unsupported device commands
• Parsing errors for unexpected output formats

Extending the Solution

Adding Device Support

Extending support for new device types involves:

• Adding platform definitions to inventory
• Creating or locating appropriate TextFSM templates
• Testing command compatibility
• Documenting platform-specific considerations

Custom Analysis Integration

The modular architecture supports custom analysis modules:

• Compliance checking against organizational policies
• Performance baseline comparisons
• Security posture assessments
• Capacity planning data collection

Enterprise Integration

Production deployments often require integration with existing systems:

• ITSM ticket creation for identified issues
• Database storage for historical trending
• Dashboard integration for real-time monitoring
• API endpoints for external system consumption

Conclusion

The transformation from raw network device data to actionable insights requires systematic approaches that address the realities of modern network environments. Our Simple Facts toolkit demonstrates that effective network automation data collection doesn’t require expensive proprietary tools – it requires thoughtful architecture, appropriate technology selection, and professional implementation practices.

The Foundation of Success

Reliable network automation data collection provides the foundation for all advanced automation workflows. Without systematic data collection and parsing capabilities, organizations cannot build comprehensive automation strategies. The approaches demonstrated in our toolkit provide this foundation while remaining accessible to network teams with varying automation experience.

Benefits of Modular Solutions

The multi-approach nature of our solution reflects real-world requirements where different scenarios demand different technical approaches. Whether you need the simplicity of direct SSH connections or the scalability of framework-based automation, having multiple options ensures that your automation strategy can adapt to changing requirements.

Adapting to Your Environment

Every network environment presents unique challenges. The examples in our Simple Facts toolkit provide starting points that teams can customize for their specific requirements. The combination of direct device access, structured parsing, and professional reporting creates a complete automation foundation.

Next Steps for Network Automation

Organizations beginning their automation journey should start with data collection and structured parsing before moving to more advanced workflows. The Simple Facts approach provides immediate value while building the foundation for more sophisticated automation projects.

The path from manual network management to intelligent automation starts with reliable data. Our toolkit demonstrates practical approaches that network engineering teams can implement immediately, providing both technical capability and business value from day one.

Additional Resources

Project Repository

• Simple Facts Toolkit – Complete source code and documentation

Technical Documentation

• Nornir Framework Documentation – Comprehensive framework guide
• TextFSM Templates Repository – Community parsing templates
• Netmiko Documentation – SSH library for network devices

Community Resources

• Network Automation Community – Open source projects and tools
• TextFSM Online Tester – Template development and testing
• Nornir Community Plugins – Extended functionality modules