There is nothing worse than getting 40 alerts on your phone at 3:00 AM because a single network switch blinked for three seconds. When you monitor a complex lab with physical switches, hypervisors, and dozens of virtual machines, standard monitoring tools have a bad habit of crying wolf. A host drops, and instead of telling you the hypervisor is down, it screams that every VM, database, container, and website on that hypervisor is simultaneously dead.
When I deployed Zabbix 7.0 LTS in my lab, my primary goal wasn’t just collecting metrics—it was killing alert fatigue. I wanted a monitoring system that tells me exactly what broke, without waking me up for transient hiccups or burying the root cause in a storm of child alerts. Here is how I set it up, the architecture, and the Python automation I wrote to build a real dependency tree.
1. The Monitoring Architecture
I spun up a dedicated virtual machine (zabbix01 at 10.1.20.17) running on my XCP-ng cluster. It watches our entire footprint across 10 VLANs:
| Layer | Target Devices / Hosts | Collection Method | Poll Rate |
|---|---|---|---|
| Network Fabric | UDM Pro, SW-CORE-01, SW-OFC-01, APs | SNMPv2 / ICMP Ping | 60 seconds |
| Hypervisors | XCP-ng R710, Windows Server Hyper-V (HV2) | Zabbix Agent 2 / WinRM / ICMP | 60 seconds |
| SAN Storage | Dell EqualLogic Array (Out-of-Band) | SNMP on 10.1.1.24 |
120 seconds |
| Virtual Machines | AD1 (Domain Controller), docker02, wazuh01 | Native zabbix-agent2 daemon |
Real-time active checks |
| Web Services | WordPress, Vaultwarden, Authentik | Zabbix Web Scenarios (HTTP 200 checks) | 60 seconds |
2. The SAN Monitoring Trap
Early on, I added all five of our Dell EqualLogic storage IPs (including the isolated 10.1.99.x SAN target ports) directly into Zabbix. Immediate mistake. Our firewall rules strictly isolate VLAN 99 from the rest of the network—nothing outside the storage fabric can talk to it, not even the monitoring VM.
Lesson Learned: Never monitor storage targets on non-routed SAN VLANs from a general server subnet. Zabbix will flag them as permanently dead. Instead, monitor the array strictly through its dedicated out-of-band management interface on VLAN 1 (
10.1.1.24).
3. Killing Alert Storms with a Python Dependency Tree
Zabbix has a powerful feature called Trigger Dependencies. If Trigger B depends on Trigger A, Zabbix will suppress Trigger B if Trigger A is already firing. But configuring dependencies by hand for dozens of hosts and hundreds of triggers in the web UI is impossible to maintain.
To automate this, I wrote an idempotent Python script (zabbix-dependencies.py) that talks directly to the Zabbix JSON-RPC API using an API token stored in my central vault. It models the real physical and virtual topology:
PARENT = {
# Core Switch sits behind Gateway
'SW-CORE-01': '_gateway',
'AP-DDWRT-01': '_gateway',
# Hypervisors, storage, and office switch sit behind Core Switch
'SW-OFC-01': 'SW-CORE-01',
'XCPNG-R710-01': 'SW-CORE-01',
'HV2': 'SW-CORE-01',
'SAN-EQ1-MGMT': 'SW-CORE-01',
# Workstations sit behind Office Switch
'Deathstar': 'SW-OFC-01',
'Ubuntu Laptop': 'SW-OFC-01',
# VMs sit behind their physical hypervisors
'XOA': 'XCPNG-R710-01',
'docker02': 'XCPNG-R710-01',
'wazuh01': 'XCPNG-R710-01',
'AD1.ittechstl.local': 'HV2',
'dockervm': 'HV2',
'homeassistant.ittechstl.local': 'HV2'
}The logic is dead simple but bulletproof:
- Every child host’s “Unavailable by ICMP ping” trigger depends on its PARENT host’s ICMP trigger.
- Every host’s internal agent and service triggers (e.g. “Zabbix agent is not available”) depend on its OWN ICMP trigger.
The result: If XCPNG-R710-01 goes offline for maintenance, I get exactly ONE alert saying the hypervisor is down. Zabbix automatically suppresses the alerts for docker02, wazuh01, and XOA. When the hypervisor comes back up, everything clears cleanly.
4. Sensible Alert Routing (No 3:00 AM Noise)
An alert system that cries wolf gets muted, and muted alerts cause outages. I established strict alerting tiers through our internal postfix mail relay ([email protected]):
- Warning & Average Severity: Must remain active for at least 10 minutes before sending an email. If a backup job briefly spikes CPU to 98% or a laptop drops off Wi-Fi for 60 seconds, no alert is sent.
- High & Disaster Severity: Sent immediately. If a hypervisor stops responding, an iSCSI path drops, or an uninterruptible power supply kicks over to battery, alerts dispatch instantly.
5. The Single Point of Failure (And What Comes Next)
No engineering runbook is complete without being honest about architectural trade-offs. Right now, zabbix01 runs as a virtual machine on XCPNG-R710-01. If the entire XCP-ng host loses physical power or hardware fails, Zabbix goes down with it—meaning it can’t send an email saying its own host died.
To close this gap, we’re currently building an external heartbeat monitor on our out-of-band network that checks Zabbix from the outside. But within the hypervisors, switches, and Docker stacks, Zabbix 7 has given us complete visibility with zero alert fatigue.