People always ask me: “If you have local AI models with access to your network, why don’t you just let them fix things automatically?”
The short answer: because I run a real business, and I actually like having working servers.
AI models are fantastic at reading system logs, correlating complex alerts, and writing diagnostic scripts. But the moment you give an LLM unchecked SSH root keys or admin API access, you are exactly one hallucinated command away from a blown partition table or a dead domain controller.
I built Jarvis—a local, voice-capable AI assistant running on a dedicated GPU in my lab. It monitors Zabbix alerts, inspects UniFi switch ports, and pings me over Signal messenger when something needs attention. But it operates under one golden rule:
AI Proposes. A Person Approves. Nothing changes itself.
Here is how I built the approval pipeline and why human-in-the-loop is the only sane way to run AI automation in a real homelab.
1. The Problem with “Full Autonomy”
An LLM does not understand the real-world blast radius of an operating system command. A model might notice high disk utilization on a database volume and decide that running rm -rf /var/lib/mysql/* solves the disk space problem. Technically, the drive is empty! In reality, your data is gone.
To prevent this, I split Jarvis’s capabilities into two distinct zones:
- Read-Only Inspection (Unrestricted): Querying container status, checking Zabbix problem triggers, reading Wazuh SIEM events, and inspecting UniFi device health.
- State-Mutating Actions (Strictly Gated): Restarting a daemon, editing a config file, bouncing a PoE switch port, or modifying DNS records.
2. The Two-Phase Approval Workflow
When something acts up in the lab—say, a monitoring collector daemon stops responding—Jarvis follows a strict two-phase process:
- Phase 1: Diagnosis & Staging: Jarvis checks the service logs, confirms the failure, and stages a proposed fix in
jarvis-proposals.json. - Phase 2: Mobile Notification & Approval: It sends an end-to-end encrypted notification directly to my phone via Signal:
[Jarvis Proposal #42]
Target: docker02 (Zabbix Agent 2)
Action: Restart crashed collector service
Command: sudo systemctl restart zabbix-agent2
Reason: TCP agent checks timing out since 01:14 AM
Reply 'approve 42' or tap [Approve] in Lab OS.If I reply “approve 42” from my phone or click Approve on my Lab OS dashboard, the command executes and the result is verified. If I ignore it or reject it, nothing happens. The script aborts and times out.
3. Local-First: Keeping Everything on My Hardware
I don’t send my network topology, internal IP addresses, or server credentials to third-party cloud APIs. Jarvis runs on a dedicated GPU workstation (DEATHSTAR) right in my office. Voice pipelines run through Home Assistant Assist, telephony alerts route through our on-prem FreePBX Asterisk server, and passwords are retrieved on-demand from our self-hosted Vaultwarden vault using ephemeral tokens.
4. The WordPress Audit Trail
Every single proposal, whether I approve it or decline it, gets permanently written to our AI Action Log. If a service ever restarts or a configuration changes, I never have to wonder what happened or dig through messy shell history. There is a clean, searchable timestamped record right here on this site.
Automation should save you time, not create emergency recovery drills at 2:00 AM. Keeping a human in the loop gives me all the speed of AI troubleshooting with zero of the nightmare.