Hunting Network Ghosts: Centralizing Syslog from UniFi, Linux, and Windows with Grafana Loki & Promtail

The most frustrating experience in systems administration is knowing an anomaly occurred—a switch port flapped, a rogue IoT device attempted an unauthorized DHCP request, or an edge firewall dropped a burst of malformed packets—and having to SSH into five different physical appliances to grep through disjointed text logs. When you operate 10 separate VLANs and dozens of microservices, decentralized logging is an operational liability. I solved this by deploying a centralized, low-overhead syslog pipeline using Grafana Loki and Promtail.

Traditional log aggregation engines like Elasticsearch (ELK) or Splunk are notorious resource hogs that demand 16GB of RAM just to idle. In contrast, Grafana Loki takes a radically efficient approach: it doesn’t build expensive full-text inverted indexes on every word. Instead, it indexes only metadata labels (like hostname, facility, and severity) and compresses the raw log chunks. Here is how I set up Loki and Promtail on docker02, routed our UniFi hardware logs, and hooked up single sign-on via Authentik.

1. The Architecture: Promtail as the Universal Collector

Our centralized logging topology consists of three lightweight containerized services running on docker02 (IP 10.1.20.19):

1. Promtail (Port 514 UDP/TCP): Acts as the network syslog receiver. It listens for incoming RFC-3164 (legacy BSD) and RFC-5424 (modern IETF) syslog packets from our UniFi Dream Machine, 48-port PoE switches, and access points. It extracts structured labels and streams the logs into Loki.
2. Loki (Port 3100): The log database engine. It stores raw log streams compressed in filesystem chunks, indexing only the stream labels. Memory usage remains under 350MB even under heavy log bursts.
3. Grafana (Port 3000): The dashboard and query engine. It connects to Loki as an authoritative data source and allows rapid LogQL filtering and alerting.

2. Production Docker Compose & SSO Integration

Security is paramount. Rather than relying on local username/password accounts for Grafana, we integrated it directly with our centralized identity provider (Authentik) using OAuth 2.0 with PKCE:

# /opt/stacks/syslog/docker-compose.yml excerpt
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - loki-data:/loki
    restart: unless-stopped

  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./promtail-config.yaml:/etc/promtail/config.yml
      - /var/log:/var/log:ro
    command: -config.file=/etc/promtail/config.yml
    ports:
      - "514:1514/udp"
      - "514:1514/tcp"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
      - GF_SERVER_ROOT_URL=http://10.1.20.19:3000
      - GF_AUTH_GENERIC_OAUTH_ENABLED=true
      - GF_AUTH_GENERIC_OAUTH_NAME=IT Tech STL Sign-In
      - GF_AUTH_GENERIC_OAUTH_CLIENT_ID=${GRAFANA_OAUTH_ID}
      - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_SECRET}
      - GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile
      - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.ittechstl.com/application/o/authorize/
      - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.ittechstl.com/application/o/token/
      - GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.ittechstl.com/application/o/userinfo/
      - GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(groups[*], 'authentik Admins') && 'Admin' || 'Editor'
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped

3. Configuring Promtail for Network Syslog Ingest

To accept syslog traffic from our UniFi switches and gateways, Promtail must be configured with a syslog scrape job. The configuration file below listens on internal port 1514, strips timestamps, and dynamically assigns the origin IP as the hostname label:

# promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_jobs:
  # Ingest local Linux system logs
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlog
          host: docker02
          __path__: /var/log/*.log

  # Ingest remote syslog from UniFi hardware
  - job_name: unifi_syslog
    syslog:
      listen_address: 0.0.0.0:1514
      idle_timeout: 60s
      label_structured_data: yes
      labels:
        job: syslog
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'
      - source_labels: ['__syslog_message_severity']
        target_label: 'severity'

4. Directing UniFi to the Central Log Target

Routing telemetry from Ubiquiti hardware into our Docker container requires just two clicks in the UniFi Network Controller:

1. Navigate to Settings > System > Advanced > Remote Logging.
2. Enable Syslog, enter the IP address of docker02 (10.1.20.19), set the port to 514, and check Device Activities and Firewall Rule Triggers.

Within seconds, Promtail begins ingesting thousands of event streams—spanning everything from 802.1X Wi-Fi client handoffs to inter-VLAN firewall rule drops.

5. Real-World LogQL Queries That Save Hours

The Power of LogQL: Instead of opening terminal sessions and scrolling through hundreds of thousands of lines, you write stream queries directly in Grafana’s Explore tab.

Here are three LogQL queries I use weekly to diagnose network anomalies:

1. Isolating Dropped Firewall Traffic:

{job="syslog"} |= "UDM-Pro" |= "DROP" | pattern "<_>   kernel:  DROP: IN= OUT= MAC= SRC= DST= PROTO="

2. Detecting Flapping Ethernet Switch Ports:

{job="syslog"} |= "switch" |~ "(link down|link up|STP state changed)"

3. Tracking DHCP Exhaustion or Clashes:

{job="syslog"} |~ "dnsmasq-dhcp\\[.*\\]: (DHCPNAK|DHCPDECLINE)"

6. Retention & Compaction Strategy

To ensure logs don’t silently fill the system drive, we configured Loki’s compaction engine with a strict 30-day retention window. Older chunks are automatically pruned every 24 hours. Because Loki compresses log text using Snappy compression, 30 days of continuous network logs across 40+ endpoints occupies less than 8GB of disk space.

No more guessing why a Wi-Fi client dropped or whether an IoT sensor attempted to telephone abroad. Everything is indexed, searchable, and visualized in a single unified dashboard.