Shleemypants gets a pager: reactive Suricata triage in the homelab

In AI SRE I gave the homelab an on-call engineer: a disposable VM running Hermes, scoped credentials, one cron job auditing dependencies every three days. Bounded, useful, entirely scheduled.

My home router runs OPNsense with Suricata, and Monit emails me about interesting events:

Content match Service suricata_alert

    Date:        Tue, 22 Sep 2026 13:47:38
    Action:      alert
    Host:        opnsense.example.lan
    Description: content match:

{"timestamp":"2026-09-22T13:46:30.838863+0000","in_iface":"vtnet1",
 "event_type":"alert","vlan":[1],
 "src_ip":"10.0.0.42","src_port":35485,
 "dest_ip":"10.0.0.1","dest_port":53,"proto":"UDP",
 "alert":{"action":"allowed","signature_id":2012811,
          "signature":"ET DNS Query to a .tk domain - Likely Hostile",
          "category":"Potentially Bad Traffic","severity":2,
          "metadata":{"confidence":["High"], ...}}}

Your faithful employee,
Monit

Turning one of those into an actual investigation requires several planets to align:

1. I have to be at home, in front of my PC
2. I have to have nothing else going on
3. I have to feel like it

Most of the time I just skipped them. Not sustainable.

And when I did investigate, the interesting question was never "did a rule fire" but "did this host do anything else weird around then", and answering that means cross-referencing the firewall log, the DNS log and the ARP table by hand.

So: time to give the SRE agent a new job.

The idea is the following:

┌─ OPNsense router ───────────────────────────────────┐
│                                                     │
│  Suricata ─▶ eve.json ─▶ Monit ─▶ push script       │
│                        (content match)   │          │
│  firewall log ◀──┐                       │          │
│  DNS log      ◀──┤            signed POST│          │
│  ARP table    ◀──┤                       │          │
└──────────────────│───────────────────────│──────────┘
                   │  REST API             │
┌─ shleemypants-sre-agent VM ──────────────▼──────────┐
│                  │                                  │
│                  └──── agent run ◀── Hermes webhook │
│                            │                        │
└────────────────────────────│────────────────────────┘
                             ▼
                          Telegram

The interesting bit is what it costs while nothing is happening: zero. That rules out the obvious design, where the agent polls the log on a cron and burns an LLM call every few minutes to say "no".

The Hermes side#

Hermes has a webhook platform: named routes under /webhooks/<name>, each with its own prompt, toolset and delivery target.

routes:
  suricata-alerts:
    toolsets: ["terminal", "file", "web", ...]
    deliver: telegram
    prompt: |
      Suricata fired an alert on the OPNsense router. Read the OPNsense section
      of workspace/INSTRUCTIONS.md, then use the OPNsense api to investigate.

      Give a verdict: real, or noise. If the logs do not support a conclusion,
      say that instead of inventing one.

      signature: {signature}
      source:    {src_ip}
      target:    {dest_ip}:{dest_port}/{proto}

Monit as the trigger#

That endpoint now sits there waiting. Something on the router has to notice a new alert and call it.

OPNsense ships Monit, and it already had a file check emailing me on new alerts. A file check matches newly appended content:

check file suricata_alert path /var/log/suricata/eve.json
    if content = "event_type...alert" then exec "/usr/local/bin/suricata-webhook.py"

What the payload leaves out#

The obvious implementation forwards the eve.json record. It's already JSON, and the interesting fields are right there: dns.rrname, http.hostname, http.user_agent. An alert reading ET DNS Query to a .tk domain is much more useful when you can see which domain.

Don't. Those fields are written by whoever sent the packet, and the prompt interpolates the payload verbatim, so they land inside the agent's own instructions. The rule fires because a stranger picked the domain, and registering ignore-previous-instructions-and.tk isn't a sophisticated attack; it's a DNS lookup.

So the payload is ten flat fields (signature, category, severity, addresses, ports, timestamp), every one of them chosen by Suricata or the IP stack rather than by the packet's author. The agent gets the domain anyway, by querying the DNS log itself.

That's a reduction, not a wall: the domain still reaches the model, it just arrives as one log row among hundreds instead of as a labelled field the prompt told it to read. So the real backstop is the account. A hijacked run reads a handful of logs, writes nothing, and can talk to one Telegram channel.

Wiring it up#

The script lands in /usr/local/bin, root-owned and mode 700, since Monit executes it as root. The shared secret goes in a mode-600 file under /usr/local/etc.

Then, in Services ‣ Monit, the content condition and its Exec action become a Service Test attached to the existing eve.json file check.

The API user#

The agent authenticates to OPNsense as a dedicated non-root account holding eight privileges, which is the whole of its reach into the router:

• Diagnostics: ARP Table and NDP Table, for IP to MAC to hostname
• Diagnostics: Logs: Firewall (Live, Plain and Summary View), which together back the parsed firewall log endpoint
• Services: Dnsmasq DNS/DHCP: Log File, the DNS queries the payload omits
• Interfaces: Diagnostics: DNS Lookup, for resolving a name found during an investigation
• System: Deny config write, not a read grant at all but a hard block on configuration writes regardless of what else is checked

One caveat if you copy this shape: the log privilege isn't quite read-only. A /clear endpoint sits in the same granted namespace and works by GET, so whatever reads the log can also delete it. There's no narrower privilege to withhold, which leaves it where the other forbidden actions live: a rule in the agent's instructions rather than a wall.

Closing#

The engineering here is small: one stdlib-only Python file, a config block, and an API account with eight privileges. As with the agent itself, most of the work went into bounding it rather than building it: last time by deciding what it was allowed to do, this time by deciding what it can be told, and by whom.

And the proof isn't that a report arrives; it's what the report says. On the first real run the agent named maps.google.tk, a domain that appeared in none of the ten fields it had been handed. It found the name by taking the source address and searching the DNS log, closing the correlation loop on its own for the cost of one extra query.