Building on our ContainerLab foundation to create a complete network automation environment.
In Part 1 of our ContainerLab series, we established a solid foundation by deploying Google Cloud VMs with ContainerLab. Now we’re ready to build the complete network automation environment by deploying a dedicated NetOps server and configuring the essential routing that enables seamless management of our containerized network devices.
This post focuses on two critical objectives:
By the end of this guide, you’ll have a fully functional network automation environment where your NetOps server can manage and automate containerized network devices through proper routing configuration.
Our target architecture includes:
At Netodata, we’ve developed a comprehensive setup script that automates the deployment of all necessary NetOps dependencies. This approach ensures consistency across environments and reduces deployment time from hours to minutes.
High-Level Requirements
Our NetOps server includes these essential components:
Core Infrastructure:
Network Automation Stack:
Directory Structure:
/opt/netops/
├── scripts/ # Automation scripts
├── inventory/ # Device inventories
├── templates/ # Jinja2 configuration templates
├── logs/ # Execution logs
├── backups/ # Configuration backups
├── data/ # Runtime data
└── venv/ # Python virtual environmentQuick Deployment
Deploy your NetOps server with our automated script:
# Deploy with URL format for GitLab raw files
curl -sSL https://gitlab.com/netodata/netodata-toolkit/-/raw/main/netops_server/deploy.sh | sudo bashOr:
# Download the script first
curl -sSL https://gitlab.com/netodata/netodata-toolkit/-/raw/main/netops_server/deploy.sh -o deploy.sh
# Inspect the script before running
head -20 deploy.sh
# Make executable and run
chmod +x deploy.sh
sudo ./deploy.sh
# Verify installation
source /opt/netops/venv/bin/activate
python -c "import nornir, netmiko, napalm; print('NetOps environment ready')"Complete script and documentation:
# Other ways to confirm deployment after installation
(venv) ubuntu@lettered-grison:~$ python /opt/netops/verify_install.py
NetOps Installation Verification
------------------------------
Python version: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0]
Platform: Linux-5.15.0-140-generic-aarch64-with-glibc2.35
Nornir version: 3.5.0
Netmiko version: 4.5.0
NAPALM version: 5.0.0
All required packages are installed correctly.
NetOps environment is ready to use.For enterprise environments, consider these advanced deployment methods:
Infrastructure as Code (Terraform):
CI/CD Pipeline Integration:
Note: Detailed implementation of these advanced approaches will be covered in future enterprise-focused posts.
Before configuring routes, it’s essential to understand how Google Cloud’s Software Defined Networking differs from traditional networking approaches.
Key Concepts:
Why VM-Level Routes Don’t Work:
Many network engineers initially attempt to configure routes using traditional Linux commands:
# This approach WILL NOT WORK on Google Cloud
sudo ip route add 172.20.20.0/24 via 10.128.0.50Even when this route appears in the VM’s routing table, inter-VM traffic will still fail with “network unreachable” errors because Google Cloud’s VPC doesn’t recognize VM-level routes for inter-instance communication.
Method 1: Google Cloud Console (GUI)
For users who prefer graphical interfaces:
netops-to-containerlab-mgmtRoute NetOps traffic to ContainerLab management subnet172.20.20.0/241000containerlab-serverus-central1-a
Method 2: gcloud CLI (Recommended)
For automation and documentation purposes:
# Create custom route in GCP VPC
gcloud compute routes create netops-to-containerlab-mgmt \
--destination-range=172.20.20.0/24 \
--next-hop-instance=containerlab-server \
--next-hop-instance-zone=us-central1-a \
--priority=1000 \
--description="Route NetOps traffic to ContainerLab management subnet"
# Verify route creation
gcloud compute routes list --filter="name:netops-to-containerlab-mgmt"Why GCP-Level Routing is Essential
Google Cloud’s VPC routing requirements exist because:
Without proper VPC routes, packets destined for the ContainerLab management subnet are dropped with ICMP “network unreachable” messages, regardless of VM-level routing configurations.
Once VPC routing is established, containerized network devices need default routes pointing to their management gateway.
Sample ContainerLab Topology:
name: netops-routing-lab
topology:
defaults:
kind: linux
nodes:
spine1:
kind: ceos
image: ceos:4.28.3M
mgmt-ipv4: 172.20.20.10
leaf1:
kind: ceos
image: ceos:4.28.3M
mgmt-ipv4: 172.20.20.11
leaf2:
kind: ceos
image: ceos:4.28.3M
mgmt-ipv4: 172.20.20.12
links:
- endpoints: ["spine1:eth1", "leaf1:eth1"]
- endpoints: ["spine1:eth2", "leaf2:eth1"]Arista cEOS Configuration:
# Connect to cEOS device
docker exec -it clab-netops-routing-lab-spine1 Cli
# Configure default route
configure
ip route 0.0.0.0/0 172.20.20.1
copy running-config startup-config
exitNokia SR Linux Configuration:
# Connect to SR Linux device
docker exec -it clab-netops-routing-lab-leaf1 sr_cli
# Configure default route
configure
/network-instance default static-routes route 0.0.0.0/0 next-hop 172.20.20.1
commit save
exitFRRouting Configuration:
# Connect to FRR container
docker exec -it clab-netops-routing-lab-leaf2 vtysh
# Configure default route
configure terminal
ip route 0.0.0.0/0 172.20.20.1
write memory
exitBasic Network Connectivity
Test fundamental network connectivity from your NetOps server:
# Test management subnet reachability
ping -c 3 172.20.20.10
# Verify routing table entries
ip route show | grep 172.20.20
# Test TCP connectivity to device management ports
nc -zv 172.20.20.10 22 # SSH
nc -zv 172.20.20.10 443 # HTTPS managementDevice Management Protocol Testing
# SSH connectivity test
ssh admin@172.20.20.10 "show version"
# SNMP connectivity test
snmpwalk -v2c -c public 172.20.20.10 1.3.6.1.2.1.1.1.0
# HTTP/HTTPS management interface test
curl -k https://172.20.20.10/command-apiPython Connectivity Test Script
Create a comprehensive connectivity test using Nornir:
#!/usr/bin/env python3
"""
NetOps Server Connectivity Validation Script
Tests connectivity to all ContainerLab management interfaces
"""
from nornir import InitNornir
from nornir.plugins.tasks.networking import netmiko_send_command
from nornir.plugins.functions.text import print_result
import ipaddress
import subprocess
import sys
def test_icmp_connectivity(mgmt_subnet="172.20.20.0/24"):
"""Test ICMP connectivity to management subnet"""
print(f"Testing ICMP connectivity to {mgmt_subnet}")
network = ipaddress.IPv4Network(mgmt_subnet)
reachable_hosts = []
for host in list(network.hosts())[:10]: # Test first 10 hosts
result = subprocess.run(
['ping', '-c', '1', '-W', '2', str(host)],
capture_output=True,
text=True
)
if result.returncode == 0:
reachable_hosts.append(str(host))
print(f"✓ {host} - Reachable")
else:
print(f"✗ {host} - Unreachable")
return reachable_hosts
def test_device_connectivity():
"""Test device connectivity using Nornir"""
nr = InitNornir(config_file="/opt/netops/config.yaml")
# Test basic connectivity
result = nr.run(
task=netmiko_send_command,
command_string="show version"
)
print_result(result)
return result
if __name__ == "__main__":
print("NetOps Server Connectivity Validation")
print("=" * 40)
# Test ICMP connectivity
reachable = test_icmp_connectivity()
if reachable:
print(f"\nFound {len(reachable)} reachable devices")
# Test device management connectivity
test_device_connectivity()
else:
print("No devices reachable - check routing configuration")
sys.exit(1)Issue: Route not appearing in routing table
# Check VPC routes
gcloud compute routes list --filter="nextHopInstance:containerlab-server"
# Verify instance exists and is running
gcloud compute instances list --filter="name:containerlab-server"Solution:
Issue: “Network unreachable” errors despite correct routes
This typically indicates routing asymmetry or missing return routes.
# Test from NetOps server
traceroute 172.20.20.10
# Check ContainerLab server routing
gcloud compute ssh containerlab-server --command="ip route show"Solution:
Issue: Containers unreachable despite VPC routing
# Check ContainerLab status
sudo containerlab inspect --all
# Verify bridge configuration
ip addr show clab-br-*
# Check container networking
docker network ls
docker inspect <container_name>Solution:
Issue: Default routes not persistent after container restart
ContainerLab containers may lose configuration after restarts.
Solution:
# Example: Persistent route configuration in topology
nodes:
spine1:
kind: ceos
image: ceos:4.28.3M
mgmt-ipv4: 172.20.20.10
exec:
- ip route add default via 172.20.20.1Once routing is configured, validate your complete environment:
# From NetOps server, test full automation workflow
cd /opt/netops
source venv/bin/activate
# Test Nornir inventory
python -c "
from nornir import InitNornir
nr = InitNornir(config_file='config.yaml')
print(f'Loaded {len(nr.inventory.hosts)} devices')
for host in nr.inventory.hosts:
print(f' - {host}: {nr.inventory.hosts[host].hostname}')
"
# Test device automation
python scripts/device_discovery.pyYour completed environment should demonstrate:
With routing configured and connectivity established, you now have a complete foundation for network automation. In Part 3, we’ll explore:
Your environment is now ready to support sophisticated network automation use cases that mirror production enterprise networks.
We’ve successfully built a complete network automation environment by:
This foundation enables network automation workflows that scale from lab testing to production deployment. The combination of containerized network devices and cloud-based automation servers provides the flexibility and reliability needed for modern network operations.
Key Takeaways:
At Netodata, we help organizations implement similar network automation solutions in production environments. Our expertise in network automation, combined with proven deployment methodologies, enables enterprises to achieve operational excellence through automation.
Related Posts:
Ready to implement network automation in your organization? Contact Netodata to learn how we can help you build robust automation solutions tailored to your network infrastructure.
