Calling My AI: How I Hooked Up a Local LLM to FreePBX and Hardware Desk Phones

Interacting with artificial intelligence through a web browser or a smartphone chat app gets old fast. Typing out prompts while you are knee-deep in a server rack with both hands full of Ethernet cables is clumsy and impractical. I wanted something much more direct: I wanted to pick up a physical desk phone, dial an extension, and talk to my local AI assistant.

In our lab, we run a full enterprise VoIP environment powered by FreePBX 17 and Asterisk. By bridging Asterisk’s audio stream directly into our local AI pipeline on DEATHSTAR, I can pick up any Yealink desk phone in the office, dial Extension 102, and have a voice conversation with Jarvis. Here is how the telephony pipeline works and how we got latency low enough for natural speech.

1. The Telephony Foundation: FreePBX in Host Mode

FreePBX runs containerized on docker02 within our dedicated VoIP network (NET-VOIP-VLAN60 at 10.1.20.19). Because real-time audio (RTP UDP ports 10000–20000) suffers severe jitter and NAT issues when pushed through Docker’s virtual bridge network, the container runs in host networking mode:

# FreePBX compose configuration excerpt
services:
  freepbx:
    image: escomputers/freepbx:17-nofail2ban
    network_mode: "host" # Required for clean UDP/RTP media streaming
    restart: unless-stopped
    volumes:
      - ./freepbx_data:/data

Our office hardware phones (Yealink SIP-T46S endpoints) auto-provision over VLAN 60 through our internal TFTP/HTTP provisioning service, while softphones (BareSIP and Linphone) run on laptops for mobile testing.

2. The Audio Pipeline: Bridging Asterisk to Local AI

Making a phone call work with a large language model requires stringing together four discrete micro-operations in sub-second time:

1. Inbound Call Capture (Asterisk AGI): When Extension 102 is dialed, Asterisk triggers a custom FastAGI daemon written in Python. It answers the call, establishes a bi-directional raw audio stream (8kHz 16-bit PCM mono), and listens for speech.
2. Speech-to-Text (Whisper): Audio packets stream into our local faster-whisper engine running on the DEATHSTAR GPU. As soon as a natural pause in speech is detected (VAD silence threshold), Whisper transcribes the speech to text (typically ~180ms).
3. LLM Inference (Ollama / Local Models): The text transcript passes to the local LLM with our lab system prompt and tool definitions. Jarvis checks whether the query is a question (e.g. “What is the status of docker02?”) or an operational request.
4. Text-to-Speech Streaming (Piper): Rather than waiting for the entire LLM response to generate, tokens stream directly into Piper TTS. Piper synthesizes speech chunks in parallel, converts the audio to 8kHz μ-law format, and pipes it back into the Asterisk channel in real time.

3. Defeating Audio Latency

The Golden Rule of Voice AI: Anything over 1.2 seconds of conversational latency feels awkward and robotic. If you wait for a 400-word LLM reply to finish generating before starting speech synthesis, the phone call feels dead.

We achieved an average conversational turnaround of 650–750 milliseconds by implementing sentence-boundary streaming: the moment the LLM generates a complete clause (a comma or period), Piper immediately begins audio playback on the phone while the LLM continues writing the remainder of the sentence.

4. Real-World Use: Rack Diagnostics & Disaster Callouts

Being able to dial your AI from an office phone has proved surprisingly practical:

• Hands-Free Rack Diagnostics: When I’m in the server closet troubleshooting cabling, I pick up the cordless handset, dial 102, and ask: “Jarvis, is SW-CORE-01 port 14 linking up?” Jarvis checks the UniFi API and speaks the port state directly in my ear.
• Automated Emergency Callouts: If a Disaster-level trigger fires in Zabbix (e.g., an EqualLogic iSCSI path drops or a host loses power), FreePBX can automatically initiate an outbound SIP call to the office desk phone, announce the incident over speakerphone, and ask for acknowledgment.

It combines 30-year-old bulletproof telephony protocols with cutting-edge local machine learning—fast, completely private, and running entirely within our own walls.