Shleemypants gets a pager: reactive Suricata triage in the homelab

Andrea Straforini — Wed, 23 Sep 2026 00:00:00 +0000
In AI SRE</em></a> I gave the homelab an on-call engineer: a disposable VM running Hermes</a>, scoped credentials, one cron job auditing dependencies every three days. Bounded, useful, entirely scheduled</em>.</p>
My home router runs OPNsense with Suricata</a>, and Monit emails me about interesting events:</p>
Content match Service suricata_alert</span></span>
</span>
    Date:        Tue, 22 Sep 2026 13:47:38</span></span>
    Action:      alert</span></span>
    Host:        opnsense.example.lan</span></span>
    Description: content match:</span></span>
</span>
{"timestamp":"2026-09-22T13:46:30.838863+0000","in_iface":"vtnet1",</span></span>
 "event_type":"alert","vlan":[1],</span></span>
 "src_ip":"10.0.0.42","src_port":35485,</span></span>
 "dest_ip":"10.0.0.1","dest_port":53,"proto":"UDP",</span></span>
 "alert":{"action":"allowed","signature_id":2012811,</span></span>
          "signature":"ET DNS Query to a .tk domain - Likely Hostile",</span></span>
          "category":"Potentially Bad Traffic","severity":2,</span></span>
          "metadata":{"confidence":["High"], ...}}}</span></span>
</span>
Your faithful employee,</span></span>
Monit</span></span></code></pre>
Turning one of those into an actual investigation requires several planets to align:</p>

I have to be at home, in front of my PC</li>
I have to have nothing else going on</li>
I have to feel like it</li>
</ol>
Most of the time I just skipped them. Not sustainable.</p>

  
</figure>
And when I did investigate, the interesting question was never "did a rule fire" but "did this
host do anything else weird around then", and answering that means cross-referencing the
firewall log, the DNS log and the ARP table by hand.</p>
So: time to give the SRE agent a new job.</p>
The idea is the following:</p>
┌─ OPNsense router ───────────────────────────────────┐</span></span>
│                                                     │</span></span>
│  Suricata ─▶ eve.json ─▶ Monit ─▶ push script       │</span></span>
│                        (content match)   │          │</span></span>
│  firewall log ◀──┐                       │          │</span></span>
│  DNS log      ◀──┤            signed POST│          │</span></span>
│  ARP table    ◀──┤                       │          │</span></span>
└──────────────────│───────────────────────│──────────┘</span></span>
                   │  REST API             │</span></span>
┌─ shleemypants-sre-agent VM ──────────────▼──────────┐</span></span>
│                  │                                  │</span></span>
│                  └──── agent run ◀── Hermes webhook │</span></span>
│                            │                        │</span></span>
└────────────────────────────│────────────────────────┘</span></span>
                             ▼</span></span>
                          Telegram</span></span></code></pre>
The interesting bit is what it costs while nothing is happening: zero. That rules out the
obvious design, where the agent polls the log on a cron and burns an LLM call every few
minutes to say "no".</p>
The Hermes side#</span></a>
</h2>
Hermes has a webhook platform</a>: named routes under /webhooks/<name></code>, each with its own prompt, toolset and delivery target.</p>
routes</span>:</span></span>
  suricata-alerts</span>:</span></span>
    toolsets</span>: [</span>"terminal"</span>,</span> "file"</span>,</span> "web"</span>,</span> ...</span>]</span></span>
    deliver</span>:</span> telegram</span></span>
    prompt</span>:</span> |</span></span>
      Suricata fired an alert on the OPNsense router. Read the OPNsense section</span></span>
      of workspace/INSTRUCTIONS.md, then use the OPNsense api to investigate.</span></span>
</span>
      Give a verdict: real, or noise. If the logs do not support a conclusion,</span></span>
      say that instead of inventing one.</span></span>
</span>
      signature: {signature}</span></span>
      source:    {src_ip}</span></span>
      target:    {dest_ip}:{dest_port}/{proto}</span></span></code></pre>Monit as the trigger#</span></a>
</h2>
That endpoint now sits there waiting. Something on the router has to notice a new alert and
call it.</p>
OPNsense ships Monit, and it already had a file check emailing me on new alerts. A file check matches newly appended</em> content:</p>
check file suricata_alert path /var/log/suricata/eve.json</span></span>
    if content = "event_type...alert" then exec "/usr/local/bin/suricata-webhook.py"</span></span></code></pre>What the payload leaves out#</span></a>
</h2>
The obvious implementation forwards the eve.json record. It's already JSON, and the interesting
fields are right there: dns.rrname</code>, http.hostname</code>, http.user_agent</code>. An alert reading
ET DNS Query to a .tk domain</code> is much more useful when you can see which</em> domain.</p>
Don't. Those fields are written by whoever sent the packet, and the prompt interpolates the
payload verbatim, so they land inside the agent's own instructions. The rule fires because</em> a
stranger picked the domain, and registering ignore-previous-instructions-and.tk</code> isn't a
sophisticated attack; it's a DNS lookup.</p>
So the payload is ten flat fields (signature, category, severity, addresses, ports, timestamp),
every one of them chosen by Suricata or the IP stack rather than by the packet's author. The
agent gets the domain anyway, by querying the DNS log itself.</p>
That's a reduction, not a wall: the domain still reaches the model, it just arrives as one log
row among hundreds instead of as a labelled field the prompt told it to read. So the real
backstop is the account. A hijacked run reads a handful of logs, writes nothing, and can talk
to one Telegram channel.</p>
Wiring it up#</span></a>
</h2>
The script lands in /usr/local/bin</code>, root-owned and mode 700</code>, since Monit executes it as
root. The shared secret goes in a mode-600</code> file under /usr/local/etc</code>.</p>
Then, in Services ‣ Monit</strong>, the content condition and its Exec</code> action become a Service
Test attached to the existing eve.json</code> file check.</p>
The API user#</span></a>
</h3>
The agent authenticates to OPNsense as a dedicated non-root account holding eight privileges,
which is the whole of its reach into the router:</p>

Diagnostics: ARP Table</strong> and NDP Table</strong>, for IP to MAC to hostname</li>
Diagnostics: Logs: Firewall</strong> (Live, Plain and Summary View), which together back the
parsed firewall log endpoint</li>
Services: Dnsmasq DNS/DHCP: Log File</strong>, the DNS queries the payload omits</li>
Interfaces: Diagnostics: DNS Lookup</strong>, for resolving a name found during an investigation</li>
System: Deny config write</strong>, not a read grant at all but a hard block on configuration
writes regardless of what else is checked</li>
</ul>
One caveat if you copy this shape: the log privilege isn't quite read-only.</strong> A /clear</code>
endpoint sits in the same granted namespace and works by GET</code>, so whatever reads the log can
also delete it. There's no narrower privilege to withhold, which leaves it where the other
forbidden actions live: a rule in the agent's instructions rather than a wall.</p>
Closing#</span></a>
</h2>
The engineering here is small: one stdlib-only Python file, a config block, and an API account
with eight privileges. As with the agent itself, most of the work went into bounding it rather
than building it: last time by deciding what it was allowed to do, this time by deciding what
it can be told, and by whom.</p>
And the proof isn't that a report arrives; it's what the report says. On the first real run the
agent named maps.google.tk</code>, a domain that appeared in none of the ten fields it had been
handed. It found the name by taking the source address and searching the DNS log, closing the
correlation loop on its own for the cost of one extra query.</p>


    </div>
    
        License</strong>
        This article is licensed under the CC BY-SA 4.0 license</a>.</p>

    </div>
</section>
Strafo - suricata

Shleemypants gets a pager: reactive Suricata triage in the homelab
