Every systems architect and MSP engineer knows the distinct dread of the 2:00 AM on-call alert: a production web service dropped, a print spooler wedged, or a disk partition crossed the 90% threshold. You drag yourself out of bed, authenticate through a VPN, open a terminal, run systemctl restart nginx or Restart-Service, wait 30 seconds to confirm the port is listening, acknowledge the ticket, and go back to sleep.
Traditional monitoring tools are exceptional at detecting failures, but historically brain-dead at resolving them. Static action scripts are brittle: if a script blindly restarts a service whose underlying failure was caused by a full root filesystem, corrupted configuration, or an exhausted socket pool, the restart fails or enters a flapping death spiral. Conversely, sending telemetry to cloud AI models on every alert introduces severe latency, high token costs, and unacceptable credential leakage.
To solve this permanently, I engineered a closed-loop, autonomous self-healing pipeline. By linking Zabbix 7.0 LTS alerting directly to an on-premise NVIDIA GeForce RTX 4070 SUPER GPU running our custom 14.8B parameter SRE reasoning model (ittechstl-architect:latest), paired with our Tactical RMM NATS fleet execution plane, the monitoring system now investigates, diagnoses, remediates, and resolves infrastructure failures on the fly in sub-second time.
1. Architectural Blueprint: The Closed-Loop Remediation Engine
The self-healing architecture operates as a strict four-stage state machine. The monitoring plane, reasoning plane, and execution plane remain cleanly separated, communicating over low-latency internal APIs:
[ Production Server / Workstation Fleet ]
│
▼ 1. Metric Failure Detected (e.g. proc.num[nginx] == 0)
[ Zabbix 7.0 LTS Server (10.1.20.17) ]
│
▼ 2. Action 30 Triggered: Webhook Media Type 75 (Duktape JS)
│ POST JSON-RPC Telemetry across Internal VLAN
▼
[ Unified MSP Bridge on srv-msp-01 (10.1.20.50) ]
│
├─► 3. Instant Acknowledge in Zabbix: "AI Investigation Initiated..."
│
├─► 4. Resolve Target Node: Hostname/IP ──► Tactical RMM Agent ID
│
▼ 5. Multi-Turn Function Calling Loop (HTTP POST /api/chat)
[ Ollama on RTX 4070 SUPER (DEATHSTAR - 10.1.10.10:11434) ]
Model: ittechstl-architect:latest (qwen3:14b @ 16k context)
│
├─► Turn 1: inspect_service({"service_name": "nginx"})
│ ──► TRMM NATS Bridge executes command on agent (60ms)
│
├─► Turn 2: check_disk_usage({})
│ ──► Validates volume capacity to rule out out-of-disk
│
├─► Turn 3: restart_service({"service_name": "nginx"})
│ ──► Restarts daemon & verifies active state
│
▼ Turn 4: Generates Root-Cause & Action Summary
[ Unified MSP Bridge ]
│
▼ 6. Zabbix JSON-RPC API: event.acknowledge (action: 6)
[ Zabbix 7.0 Server ] ──► Posts Technical Diagnosis & Auto-Closes Problem
│
▼ 7. Optional Helpdesk Sync
[ Zammad Helpdesk (tickets.ittechstl.com) ] ──► Logs Internal Resolution Note2. Stage 1: Configuring Zabbix 7.0 Webhook Media Type & Actions
In Zabbix 7.0, rather than running legacy server-side shell scripts (which require dangerous EnableGlobalScripts=1 flags on the server OS), alerting is handled by native Webhook Media Types powered by the embedded Duktape JavaScript engine. I created a dedicated Media Type (AI Autonomous Remediator, ID 75):
// Zabbix 7.0 Webhook Script: AI Autonomous Remediator (Media Type 75)
try {
var params = JSON.parse(value);
// Ignore recoveries: LLM only engages on active problem triggers
if (params.event_value !== '1') {
return 'Ignored: Event is not an active problem.';
}
var req = new HttpRequest();
req.addHeader('Content-Type: application/json');
var payload = {
event_id: params.event_id,
event_name: params.event_name,
event_severity: params.event_severity,
host_name: params.host_name,
host_ip: params.host_ip,
item_key: params.item_key,
item_value: params.item_value,
trigger_desc: params.trigger_desc
};
var response = req.post(params.url, JSON.stringify(payload));
if (req.getStatus() < 200 || req.getStatus() >= 300) {
throw 'HTTP ' + req.getStatus() + ': ' + response;
}
return 'OK: Dispatched to AI SRE Bridge: ' + response;
} catch (error) {
Zabbix.log(3, '[AI Remediation] Webhook failed: ' + error);
throw 'AI Remediation Webhook failed: ' + error;
}The webhook script captures essential Zabbix macros ({EVENT.ID}, {EVENT.NAME}, {EVENT.SEVERITY}, {HOST.NAME}, {HOST.IP}, {ITEM.KEY}, and {ITEM.LASTVALUE}) and forwards them directly to our internal API endpoint. I bound this media type to the Admin user profile and created Trigger Action 30 (AI Autonomous Remediation (RTX 4070 SUPER Auto-Heal)) targeting all triggers with Severity >= Warning.
3. Stage 2: The Remediation Bridge & Tactical RMM NATS Execution Plane
An LLM is completely useless for operations if it cannot safely interact with the machines it monitors. The most critical design decision in this architecture was avoiding fragile SSH key distribution or opening vulnerable WinRM ports across our subnets. Instead, our FastAPI MSP Bridge taps directly into our Tactical RMM NATS message bus.
Tactical RMM agents run as SYSTEM on Windows and root on Linux, maintaining persistent, encrypted WebSocket/NATS connections back to our management cluster. By exposing a lightweight tool registry to the LLM, the model can execute commands on any agent in milliseconds:
# Tool Schemas Exposed to Ollama
REMEDIATION_TOOLS = [
{
"type": "function",
"function": {
"name": "inspect_service",
"description": "Query real-time status of a service (systemd or Windows Service).",
"parameters": {
"type": "object",
"properties": {
"service_name": {"type": "string", "description": "e.g. nginx, docker, spooler"}
},
"required": ["service_name"]
}
}
},
{
"type": "function",
"function": {
"name": "restart_service",
"description": "Safely restart a failed or degraded service.",
"parameters": {
"type": "object",
"properties": {
"service_name": {"type": "string"}
},
"required": ["service_name"]
}
}
},
{
"type": "function",
"function": {
"name": "check_disk_usage",
"description": "Check filesystem utilization across all mounted volumes."
}
},
{
"type": "function",
"function": {
"name": "clean_temp_files",
"description": "Purge temp caches (C:WindowsTemp, /tmp) or prune docker containers.",
"parameters": {
"type": "object",
"properties": {
"target": {"type": "string", "enum": ["temp_dirs", "docker_prune"]}
},
"required": ["target"]
}
}
},
{
"type": "function",
"function": {
"name": "run_diagnostic_command",
"description": "Run non-destructive diagnostic queries (journalctl, netstat, Get-Process).",
"parameters": {
"type": "object",
"properties": {
"command": {"type": "string"}
},
"required": ["command"]
}
}
}
]When the LLM calls inspect_service("nginx"), the bridge determines the platform (Linux vs. Windows), formats the payload (systemctl status nginx vs. Get-Service nginx), and dispatches it through Tactical RMM’s API: POST /scripts/{agent_id}/test/. The endpoint executes the command inside memory and returns the exact stdout, stderr, and exit code in under 70 milliseconds.
4. Stage 3: Multi-Turn Autonomous Reasoning on Local Silicon
When Zabbix fires an alert, the bridge initiates an asynchronous background task. It immediately calls the Zabbix JSON-RPC API to post an in-progress comment on the event, notifying any human viewing the dashboard that an automated SRE investigation is active. It then hands the telemetry over to ittechstl-architect:latest on our workstation GPU.
Below is the actual production execution log from our live verification test when Nginx was artificially halted on srv-msp-01:
[AI-HEAL] Received problem: 'Service nginx status degraded or down' on srv-msp-01 (10.1.20.50) - Event #999999
[AI-HEAL] Target mapping: srv-msp-01 -> Agent ID: cLnHQqdJgVIJYhJEdodzQFZjQuMAZIuotbfurDTt (linux)
[AI-HEAL] Turn 1: Sending query to local Ollama on RTX 4070 SUPER...
[AI-HEAL] LLM Call: inspect_service({'service_name': 'nginx'})
[AI-HEAL] TRMM NATS Execution: POST /scripts/{agent_id}/test/ -> HTTP 200 OK (61ms)
Output: "inactive (dead); process terminated with exit code 0"
[AI-HEAL] Turn 2: Feeding tool output back to Ollama...
[AI-HEAL] LLM Call: check_disk_usage({})
[AI-HEAL] TRMM NATS Execution: POST /scripts/{agent_id}/test/ -> HTTP 200 OK (54ms)
Output: "/dev/mapper/ubuntu--vg-ubuntu--lv 50G (18G used, 30G free - 38%)"
[AI-HEAL] Turn 3: Feeding disk telemetry back to Ollama...
[AI-HEAL] LLM Call: run_diagnostic_command({'command': 'which nginx'})
[AI-HEAL] TRMM NATS Execution: POST /scripts/{agent_id}/test/ -> HTTP 200 OK (49ms)
Output: "/usr/sbin/nginx"
[AI-HEAL] Turn 4: Final Reasoning & Remediation Synthesis
[AI-HEAL] Solution: Disk healthy, binary verified, service stopped cleanly. Restarting daemon.
[AI-HEAL] Zabbix API Acknowledge: POST /zabbix/api_jsonrpc.php -> HTTP 200 OK
[AI-HEAL] Event #999999 Closed in Zabbix: TRUENotice the intelligence on display: rather than blindly issuing a restart command, the model verified that disk exhaustion was not the root cause and checked binary path integrity before executing the resolution. Once complete, it posted a comprehensive post-mortem comment into Zabbix and triggered action: 6 (add comment + close problem).
5. Critical Enterprise Safety Guardrails
Granting an autonomous LLM execution authority over production infrastructure without strict boundaries is gross negligence. To prevent catastrophic hallucinated commands or cascade failures, our bridge implements multiple defense-in-depth safety layers:
| Safety Mechanism | Implementation | Operational Protection |
|---|---|---|
| Regex Command Blocking | Hardcoded pattern filter in main.py |
Blocks destructive syntax: rm -rf, Format-Volume, Remove-Item -Recurse, Stop-Computer, and drop database. |
| Circuit Breaker / Flapping Limit | Stateful event cache with 15-minute window | If a trigger fires >2 times in 15 minutes, autonomous healing aborts and escalates directly to human on-call. |
| Severity Gating | Zabbix Action filter logic | Warning & Average: Full autonomous remediation. Disaster & High: Diagnostic log collection only; proposed plan requires human click-to-approve. |
| Dual-Source Verification | Pre- and post-execution telemetry checks | The LLM must verify that the service is actually dead before restarting, and re-query after restart before closing the ticket. |
6. The Verdict: Zero Cloud Tokens, Zero Latency
By hosting the reasoning core on local workstation silicon (RTX 4070 SUPER) and routing commands across an established RMM agent network, the entire self-healing cycle—from trigger firing in Zabbix to full service recovery and ticket closure—completes in under 15 seconds with exactly zero cloud API tokens billed and zero corporate telemetry leaving the local network.
Self-healing infrastructure is no longer science fiction or the exclusive domain of multi-million-dollar hyperscaler budgets. With modern open-source tooling—Zabbix 7.0, Tactical RMM, FastAPI, and local quantized LLMs—any systems engineering lab or MSP can build an unshakeable, self-repairing fleet today.