Strafo - llm

Strafo - llm https://strafo.net Personal blog of a cat disguised as a human Zola en Sat, 08 Aug 2026 00:00:00 +0000 AI SRE: giving my homelab an SRE with an attitude problem Sat, 08 Aug 2026 00:00:00 +0000 Andrea Straforini https://strafo.net/blog/shleemypants-sre-agent/ https://strafo.net/blog/shleemypants-sre-agent/ <p>In <a rel="external" href="https://strafo.net/blog/iac-homelab/"><em>One-click homelab</em></a> I rebuilt my homelab so that the whole thing could be recreated from scratch in about an hour. That solved the "did I back that up" anxiety. It didn't solve the other anxiety: I'm the only person watching this stuff, and I don't always notice when something is on fire.</p> <p>So I gave the homelab its own on-call engineer. But that means tackling all the usual problems with agents: operating system rotting, tight permission scoping, prompt injection, and of course <strong>reproducibility</strong>, as always.</p> <p>This post walks through how I built it: a disposable Ubuntu VM running the Hermes agent connected to Telegram, with the necessary tools baked in so it can operate safely in the environment.</p> <h2 id="hermes-agent-in-a-nutshell">Hermes Agent in a nutshell<a class="post-anchor" href="#hermes-agent-in-a-nutshell" aria-label="Anchor link for: hermes-agent-in-a-nutshell"><span aria-hidden="true">#</span></a> </h2> <p>I used <a rel="external" href="https://github.com/NousResearch/hermes-agent">Hermes</a>, which turned out to be the right building block for a few reasons beyond "it can call MCP tools":</p> <ul> <li><strong>Runs as a container, not a framework you glue together.</strong> <code>hermes gateway run</code> is a supported foreground service mode: point it at a config, mount a volume for state, and it's a long-running Telegram-connected process. No custom bot loop to maintain.</li> <li><strong>Persistent conversation memory across restarts.</strong> Everything durable (conversation history, cron jobs, MCP secrets) lives under <code>HERMES_HOME</code> (<code>/opt/data</code> inside the container), which I bind-mount from the host. Restart the container, the agent doesn't forget who it's talking to.</li> <li><strong>A native cron subsystem.</strong> <code>hermes cron create</code> schedules a prompt to run unattended and deliver its output somewhere (Telegram, in my case). That's what runs the supply-chain audit below, with no external scheduler needed.</li> <li><strong>MCP servers are first-class config, not a bolt-on.</strong> <code>config.yaml</code> declares them directly; Hermes handles spawning <code>npx</code>/<code>uvx</code> and injecting <code>${VAR}</code>-style secrets from <code>.env</code> at connect time.</li> </ul> <h2 id="first-problem-operating-system-rotting">First problem: operating system rotting<a class="post-anchor" href="#first-problem-operating-system-rotting" aria-label="Anchor link for: first-problem-operating-system-rotting"><span aria-hidden="true">#</span></a> </h2> <p>By <em>operating system rotting</em> I mean the degradation that happens when an autonomous agent keeps modifying a system to fix immediate problems.</p> <p>Fortunately Hermes gives us a solution: <a rel="external" href="https://hermes-agent.nousresearch.com/docs/user-guide/docker/">running it in a container</a>. The image itself is immutable, read-only to the runtime user, and everything the agent could possibly mutate (config, sessions, memory, skills, logs) is confined to a single bind-mounted directory.</p> <h2 id="second-problem-giving-hermes-the-toolset">Second problem: giving Hermes the toolset<a class="post-anchor" href="#second-problem-giving-hermes-the-toolset" aria-label="Anchor link for: second-problem-giving-hermes-the-toolset"><span aria-hidden="true">#</span></a> </h2> <p>Now the agent needs to actually be able to touch the environment it's supposed to be watching. It gets tools for:</p> <ul> <li><strong>Proxmox</strong>: read cluster state, start/stop/reboot VMs</li> <li><strong>Kubernetes</strong>: cluster inspection, once a kubeconfig exists (it doesn't yet; more on that later)</li> <li><strong>GitLab and GitHub</strong>, where all the IaC lives: open issues, push to feature branches, comment on MRs</li> <li><strong><a rel="external" href="https://github.com/google/osv-scanner">osv-scanner</a></strong>: match dependencies against known vulnerability databases for the supply-chain audit</li> </ul> <p>Hermes' <a rel="external" href="https://hermes-agent.nousresearch.com/docs/user-guide/docker#installing-more-tools-in-the-container">own docs</a> offer a few ways to add tools: <code>npx</code>/<code>uvx</code> on demand, <code>apt-get install</code> at runtime (lost on restart), a derived image, or a sidecar.</p> <p>For tools needed on every boot, they recommend the derived image:</p> <details> <summary><code>sre-agent/Dockerfile</code> (click to expand)</summary> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="docker"><span class="giallo-l"><span style="color: #F92672;">ARG</span><span> HERMES_TAG=v2026.8.3</span></span> <span class="giallo-l"><span style="color: #F92672;">FROM</span><span> docker.io/nousresearch/hermes-agent:${HERMES_TAG}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span style="color: #F92672;">ARG</span><span> OSV_SCANNER_VERSION=1.9.2</span></span> <span class="giallo-l"><span style="color: #F92672;">ARG</span><span> OSV_SCANNER_SHA256=<checksum pinned here></span></span> <span class="giallo-l"></span> <span class="giallo-l"><span style="color: #F92672;">USER</span><span> root</span></span> <span class="giallo-l"><span style="color: #F92672;">RUN</span><span> apt-get update && apt-get install -y --no-install-recommends \</span></span> <span class="giallo-l"><span> ca-certificates curl gnupg gh glab \</span></span> <span class="giallo-l"><span style="color: #88846F;"> # kubectl from the official k8s apt repo, not Debian's, which only supports</span></span> <span class="giallo-l"><span style="color: #88846F;"> # ±1 minor from the API server</span></span> <span class="giallo-l"><span> && install -m 0755 -d /etc/apt/keyrings \</span></span> <span class="giallo-l"><span> && curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key \</span></span> <span class="giallo-l"><span> | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg \</span></span> <span class="giallo-l"><span> && echo </span><span style="color: #E6DB74;">"deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \</span></span> <span class="giallo-l"><span style="color: #E6DB74;"> https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /"</span><span> \</span></span> <span class="giallo-l"><span> > /etc/apt/sources.list.d/kubernetes.list \</span></span> <span class="giallo-l"><span> && apt-get update && apt-get install -y --no-install-recommends kubectl \</span></span> <span class="giallo-l"><span style="color: #88846F;"> # osv-scanner: downloaded, checksummed, installed to /usr/local/bin (not packaged)</span></span> <span class="giallo-l"><span> && curl -fsSLo /usr/local/bin/osv-scanner \</span></span> <span class="giallo-l"><span style="color: #E6DB74;"> "https://github.com/google/osv-scanner/releases/download/v${OSV_SCANNER_VERSION}/osv-scanner_linux_amd64"</span><span> \</span></span> <span class="giallo-l"><span> && echo </span><span style="color: #E6DB74;">"${OSV_SCANNER_SHA256} /usr/local/bin/osv-scanner"</span><span> | sha256sum -c - \</span></span> <span class="giallo-l"><span> && chmod +x /usr/local/bin/osv-scanner \</span></span> <span class="giallo-l"><span> && rm -rf /var/lib/apt/lists/*</span></span> <span class="giallo-l"><span style="color: #F92672;">USER</span><span> hermes</span></span></code></pre></details> <h2 id="third-problem-guardrailing-its-permissions">Third problem: guardrailing its permissions<a class="post-anchor" href="#third-problem-guardrailing-its-permissions" aria-label="Anchor link for: third-problem-guardrailing-its-permissions"><span aria-hidden="true">#</span></a> </h2> <p>Tools are only half the job. Each credential also needs the least standing power that still works, enforced by the platform, not by the model choosing to behave:</p> <ul> <li> <p><strong>Proxmox</strong>: a hand-built role, not <code>PVEAdmin</code>. <code>proxmox_virtual_environment_role</code> in OpenTofu lists exactly the privileges it gets: the built-in audit set plus <code>VM.PowerMgmt</code>, nothing that touches config, storage, or firewall rules:</p> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="hcl"><span class="giallo-l"><span style="color: #A6E22E;text-decoration: underline;">resource</span><span> "proxmox_virtual_environment_role" "sre_agent" {</span></span> <span class="giallo-l"><span> role_id</span><span style="color: #F92672;"> =</span><span style="color: #E6DB74;"> "SREAgentOperator"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> privileges</span><span style="color: #F92672;"> =</span><span> [</span></span> <span class="giallo-l"><span style="color: #E6DB74;"> "Sys.Audit"</span><span>,</span><span style="color: #E6DB74;"> "VM.Audit"</span><span>,</span><span style="color: #E6DB74;"> "VM.GuestAgent.Audit"</span><span>,</span></span> <span class="giallo-l"><span style="color: #E6DB74;"> "Datastore.Audit"</span><span>,</span><span style="color: #E6DB74;"> "Pool.Audit"</span><span>,</span><span style="color: #E6DB74;"> "SDN.Audit"</span><span>,</span><span style="color: #E6DB74;"> "Mapping.Audit"</span><span>,</span></span> <span class="giallo-l"><span style="color: #E6DB74;"> "VM.PowerMgmt"</span><span>,</span></span> <span class="giallo-l"><span> ]</span></span> <span class="giallo-l"><span>}</span></span></code></pre> <p>We could hand it more, like reading logs inside the VMs, but starting conservative is the point: every extra privilege is one more thing a prompt injection or a compromised dependency gets to use.</p> <p>For now, the old "Have you tried turning it off and on again?" is enough:</p> <figure> <img src="it-crowd-off-and-on.gif" alt="Roy from The IT Crowd asking "Have you tried turning it off and on again?""> <figcaption>The entire Proxmox permission model, honestly</figcaption> </figure> <p>The token itself is deliberately <em>not</em> rotated on every pipeline run:</p> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="hcl"><span class="giallo-l"><span style="color: #A6E22E;text-decoration: underline;">resource</span><span> "proxmox_user_token" "sre_agent" {</span></span> <span class="giallo-l"><span> user_id</span><span style="color: #F92672;"> =</span><span> proxmox_virtual_environment_user</span><span style="color: #F92672;">.</span><span>sre_agent</span><span style="color: #F92672;">.</span><span>user_id</span></span> <span class="giallo-l"><span> token_name</span><span style="color: #F92672;"> =</span><span style="color: #E6DB74;"> "token"</span></span> <span class="giallo-l"><span> privileges_separation</span><span style="color: #F92672;"> =</span><span style="color: #AE81FF;"> true</span></span> <span class="giallo-l"><span>}</span></span></code></pre> <p>The secret only ever reaches the VM at boot, via cloud-init. Rotate it on every pipeline run and the running agent keeps holding the old value after Proxmox has already killed it: 401s until the next rebuild. So it only rotates when the VM itself rebuilds.</p> </li> <li> <p><strong>GitHub</strong>: a personal access token scoped to just the repos it needs, consumed directly by <code>gh</code> via <code>GH_TOKEN</code>: no MCP wrapper, no OpenTofu-managed resource, just a secret handed to CI and templated into cloud-init like everything else.</p> </li> <li> <p><strong>GitLab</strong>: a <a rel="external" href="https://docs.gitlab.com/ee/user/group/service_accounts.html">group service account</a> with a rotating access token:</p> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="hcl"><span class="giallo-l"><span style="color: #A6E22E;text-decoration: underline;">resource</span><span> "gitlab_group_service_account_access_token" "sre_agent" {</span></span> <span class="giallo-l"><span> group</span><span style="color: #F92672;"> =</span><span> data</span><span style="color: #F92672;">.</span><span>gitlab_group</span><span style="color: #F92672;">.</span><span>strafohouse</span><span style="color: #F92672;">.</span><span>id</span></span> <span class="giallo-l"><span> user_id</span><span style="color: #F92672;"> =</span><span> var</span><span style="color: #F92672;">.</span><span>gitlab_sre_agent_service_account_id</span></span> <span class="giallo-l"><span> name</span><span style="color: #F92672;"> =</span><span style="color: #E6DB74;"> "shleemypants-mcp"</span></span> <span class="giallo-l"><span> scopes</span><span style="color: #F92672;"> =</span><span> [</span><span style="color: #E6DB74;">"api"</span><span>]</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> rotation_configuration</span><span style="color: #F92672;"> =</span><span> {</span></span> <span class="giallo-l"><span> expiration_days</span><span style="color: #F92672;"> =</span><span style="color: #AE81FF;"> 365</span></span> <span class="giallo-l"><span> rotate_before_days</span><span style="color: #F92672;"> =</span><span style="color: #AE81FF;"> 30</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span>}</span></span></code></pre> <p>The scope is <code>api</code>, which is broad on paper. But the account itself is only a <strong>Developer</strong> in the group, not Maintainer or Owner, so GitLab refuses anything that requires elevated permissions regardless of what the token can technically call: pushing to or merging <code>main</code>, reading CI/CD variables, touching protected branches. Scope decides which endpoints the token can call at all; role decides what it's allowed to do once it gets there. Neither alone would be enough.</p> <p>The service account is created once, by hand, in the GitLab UI, never by OpenTofu: GitLab soft-deletes and permanently reserves an SA username after deletion, so recreating one keeps 400ing with "username has already been taken" whenever the tofu state is reset and it tries to create the SA fresh.</p> <p>OpenTofu only mints and rotates the token against a pre-existing account, same idea as the Proxmox token: state can be destroyed and rebuilt without permanently burning a username.</p> </li> </ul> <h2 id="fourth-problem-prompt-injection">Fourth problem: prompt injection<a class="post-anchor" href="#fourth-problem-prompt-injection" aria-label="Anchor link for: fourth-problem-prompt-injection"><span aria-hidden="true">#</span></a> </h2> <p>Everything above scopes what the agent can do if I ask it something dumb, not what happens if a log line, an issue, or a fetched webpage carries instructions of its own.</p> <p>OpenRouter, which every model call already routes through, has a <a rel="external" href="https://openrouter.ai/docs/guides/features/guardrails/prompt-injection">guardrail</a> for that: OWASP-derived regex detection for instruction-override, fake system/developer-mode, prompt extraction, jailbreaks, and basic obfuscation (Base64, scrambled letters). Flag, redact, or block per match; Shleemypants is set to block.</p> <p>Hermes adds <a rel="external" href="https://hermes-agent.nousresearch.com/docs/user-guide/security">its own layers</a> on the VM side: context files are scanned before they reach the prompt (ignore-prior-instructions phrasing, hidden HTML comments, attempts to read <code>.env</code> or <code>.netrc</code>, credential exfil via <code>curl</code>, invisible Unicode), <code>tirith</code> scans commands before execution for homograph URLs and <code>curl | bash</code> patterns, terminal working directories are allowlisted, and destructive commands need explicit approval.</p> <p>The honest gap is <em>indirect</em> injection: a poisoned issue body, a webpage a tool fetched, an MCP response. Those land in context unfiltered. Regex misses novel phrasing, so none of this is the real backstop: the scoped credentials above are what cap the damage when something gets through.</p> <h2 id="giving-the-agent-a-personality">Giving the agent a personality<a class="post-anchor" href="#giving-the-agent-a-personality" aria-label="Anchor link for: giving-the-agent-a-personality"><span aria-hidden="true">#</span></a> </h2> <p><code>SOUL.md</code> sets the voice, deliberately not corporate, so uncertainty and annoyance stay visible instead of flattened into a neutral status line:</p> <blockquote> <p>"How you think you gonna push to prod while you're standin' in it, you dumb ass three-dimensional monkey ass dummy?"</p> </blockquote> <p><code>INSTRUCTIONS.md</code>, loaded at every conversation start, sets the rules: what it's forbidden from touching, and when to alert and wait instead of acting. Prompt text, not a sandbox; enforcement happens one layer down, in what tools it's physically capable of calling.</p> <figure> <img src="shleemypants-whats-up.gif" alt="Rick and Morty-style meeseeks-brained character at a desk captioned "Shleemypants here, what's up?""> </figure> <h2 id="how-the-pipeline-works">How the pipeline works<a class="post-anchor" href="#how-the-pipeline-works" aria-label="Anchor link for: how-the-pipeline-works"><span aria-hidden="true">#</span></a> </h2> <p>A <code>sre-agent/x.y.z</code> tag drives both stages: build the image, then replace the VM.</p> <p><strong>Build.</strong> <code>sre-agent/Dockerfile</code> (shown above, in the toolset section) extends the upstream <code>nousresearch/hermes-agent</code> image, pinned to a <code>HERMES_TAG</code>:</p> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="yaml"><span class="giallo-l"><span style="color: #F92672;">build-sre-agent-image</span><span>:</span></span> <span class="giallo-l"><span style="color: #F92672;"> script</span><span>:</span></span> <span class="giallo-l"><span> -</span><span style="color: #E6DB74;"> ref="${CI_COMMIT_TAG#sre-agent/}"</span></span> <span class="giallo-l"><span> -</span><span style="color: #E6DB74;"> docker build --pull -t "$IMAGE:$ref" sre-agent</span></span> <span class="giallo-l"><span> -</span><span style="color: #E6DB74;"> docker push "$IMAGE:$ref"</span></span></code></pre> <p><strong>Provision.</strong> The VM boots from a stock Ubuntu cloud image, fetched once and cached (<code>overwrite = false</code> skips re-fetching on a re-<code>apply</code>). Everything else (systemd units, <code>config.yaml</code>, <code>SOUL.md</code>, <code>INSTRUCTIONS.md</code>, every secret) gets templated into a cloud-init snippet:</p> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="hcl"><span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;"> 1</span><span style="color: #A6E22E;text-decoration: underline;">resource</span><span> "proxmox_virtual_environment_file" "cloud_config" {</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;"> 2</span><span> content_type</span><span style="color: #F92672;"> =</span><span style="color: #E6DB74;"> "snippets"</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;"> 3</span><span style="color: #000000;background-color: #FFFFFF;"> </span><span style="color: #A6E22E;text-decoration: underline;">source_raw</span><span> {</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;"> 4</span><span> file_name</span><span style="color: #F92672;"> =</span><span style="color: #E6DB74;"> "sre-agent.cloud-config.yaml"</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;"> 5</span><span> data</span><span style="color: #F92672;"> =</span><span style="color: #66D9EF;"> templatefile</span><span>(</span><span style="color: #E6DB74;">"</span><span style="color: #F92672;">${</span><span>path</span><span style="color: #F92672;">.</span><span>module</span><span style="color: #F92672;">}</span><span style="color: #E6DB74;">/tpl/cloud-config.yaml"</span><span>, {</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;"> 6</span><span> hermes_agent_service</span><span style="color: #F92672;"> =</span><span style="color: #66D9EF;"> file</span><span>(</span><span style="color: #E6DB74;">"</span><span style="color: #F92672;">${</span><span>path</span><span style="color: #F92672;">.</span><span>module</span><span style="color: #F92672;">}</span><span style="color: #E6DB74;">/../../sre-agent/hermes-agent.service"</span><span>)</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;"> 7</span><span> agent_config</span><span style="color: #F92672;"> =</span><span style="color: #66D9EF;"> file</span><span>(</span><span style="color: #E6DB74;">"</span><span style="color: #F92672;">${</span><span>path</span><span style="color: #F92672;">.</span><span>module</span><span style="color: #F92672;">}</span><span style="color: #E6DB74;">/../../sre-agent/config.yaml"</span><span>)</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;"> 8</span><span> agent_soul</span><span style="color: #F92672;"> =</span><span style="color: #66D9EF;"> file</span><span>(</span><span style="color: #E6DB74;">"</span><span style="color: #F92672;">${</span><span>path</span><span style="color: #F92672;">.</span><span>module</span><span style="color: #F92672;">}</span><span style="color: #E6DB74;">/../../sre-agent/SOUL.md"</span><span>)</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;"> 9</span><span> agent_instructions</span><span style="color: #F92672;"> =</span><span style="color: #66D9EF;"> file</span><span>(</span><span style="color: #E6DB74;">"</span><span style="color: #F92672;">${</span><span>path</span><span style="color: #F92672;">.</span><span>module</span><span style="color: #F92672;">}</span><span style="color: #E6DB74;">/../../sre-agent/INSTRUCTIONS.md"</span><span>)</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;">10</span><span> sre_agent_image</span><span style="color: #F92672;"> =</span><span> local.sre_agent_image</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;">11</span><span style="color: #88846F;"> # ...secrets: proxmox token, gitlab token, openrouter key, telegram</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;">12</span><span style="color: #88846F;"> # token, github PAT, context7 key, gatus token</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;">13</span><span> })</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;">14</span><span> }</span></span> <span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #90908A;">15</span><span>}</span></span></code></pre> <p>That snippet is a <code>#cloud-config</code> doc: it <code>write_files</code>s the systemd units and config into place, drops secrets into a root-owned <code>/var/lib/hermes/.env</code>, and its <code>runcmd</code> starts everything: <code>qemu-guest-agent</code>, <code>docker</code>, <code>hermes-agent</code>, <code>hermes-cron-seed</code>, <code>gatus-heartbeat.timer</code>.</p> <p>What forces a <em>new</em> VM rather than an in-place update is a <code>terraform_data</code> resource tracking the release tag, wired into the VM's <code>lifecycle</code>:</p> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="hcl"><span class="giallo-l"><span style="color: #A6E22E;text-decoration: underline;">resource</span><span> "terraform_data" "release" {</span></span> <span class="giallo-l"><span> input</span><span style="color: #F92672;"> =</span><span> local</span><span style="color: #F92672;">.</span><span>sre_agent_image</span></span> <span class="giallo-l"><span>}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span style="color: #A6E22E;text-decoration: underline;">resource</span><span> "proxmox_virtual_environment_vm" "sre_agent" {</span></span> <span class="giallo-l"><span style="color: #88846F;"> # ...</span></span> <span class="giallo-l"><span style="color: #000000;background-color: #FFFFFF;"> </span><span style="color: #A6E22E;text-decoration: underline;">lifecycle</span><span> {</span></span> <span class="giallo-l"><span> replace_triggered_by</span><span style="color: #F92672;"> =</span><span> [</span></span> <span class="giallo-l"><span> terraform_data</span><span style="color: #F92672;">.</span><span>release,</span></span> <span class="giallo-l"><span> proxmox_virtual_environment_file</span><span style="color: #F92672;">.</span><span>cloud_config,</span></span> <span class="giallo-l"><span> ]</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span>}</span></span></code></pre> <p>Bump the tag, or edit anything templated into cloud-init (an edited systemd unit, a new <code>config.yaml</code>) and the VM gets replaced, not reconfigured. Recreating the VM <em>is</em> the deploy.</p> <p>That's a deliberate trade: replace-not-reconfigure means zero in-place state to reason about, at the cost of wiping conversational memory on every release. A smarter setup could diff the cloud-init payload and only bounce the systemd unit when it actually changed, keeping the VM, and the agent's memory, alive across releases. For now that's more machinery than the problem deserves: one VM, one container, releases that aren't exactly frequent. Operational simplicity over state management: KISS, until it isn't.</p> <h2 id="what-it-s-actually-allowed-to-touch">What it's actually allowed to touch<a class="post-anchor" href="#what-it-s-actually-allowed-to-touch" aria-label="Anchor link for: what-it-s-actually-allowed-to-touch"><span aria-hidden="true">#</span></a> </h2> <p>Not every credential turns into an MCP server. <code>config.yaml</code> wires up MCP for the things that benefit from structured tool calls:</p> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="yaml"><span class="giallo-l"><span style="color: #F92672;">mcp_servers</span><span>:</span></span> <span class="giallo-l"><span style="color: #F92672;"> proxmox</span><span>:</span></span> <span class="giallo-l"><span style="color: #F92672;"> command</span><span>:</span><span style="color: #E6DB74;"> npx</span></span> <span class="giallo-l"><span style="color: #F92672;"> args</span><span>: [</span><span style="color: #E6DB74;">"-y"</span><span>,</span><span style="color: #E6DB74;"> "proxmox-mcp-server@0.2.0"</span><span>]</span></span> <span class="giallo-l"><span style="color: #F92672;"> env</span><span>:</span></span> <span class="giallo-l"><span style="color: #F92672;"> PVE_READONLY</span><span>:</span><span style="color: #E6DB74;"> "false"</span></span> <span class="giallo-l"><span style="color: #F92672;"> PROXMOX_TOKEN_SECRET</span><span>:</span><span style="color: #E6DB74;"> ${PROXMOX_MCP_TOKEN_SECRET}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span style="color: #F92672;"> context7</span><span>:</span></span> <span class="giallo-l"><span style="color: #F92672;"> command</span><span>:</span><span style="color: #E6DB74;"> npx</span></span> <span class="giallo-l"><span style="color: #F92672;"> args</span><span>: [</span><span style="color: #E6DB74;">"-y"</span><span>,</span><span style="color: #E6DB74;"> "@upstash/context7-mcp@3.1.0"</span><span>]</span></span> <span class="giallo-l"><span style="color: #F92672;"> env</span><span>:</span></span> <span class="giallo-l"><span style="color: #F92672;"> CONTEXT7_API_KEY</span><span>:</span><span style="color: #E6DB74;"> ${CONTEXT7_API_KEY}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span style="color: #F92672;"> serena</span><span>:</span></span> <span class="giallo-l"><span style="color: #F92672;"> command</span><span>:</span><span style="color: #E6DB74;"> uvx</span></span> <span class="giallo-l"><span style="color: #F92672;"> args</span><span>: [</span><span style="color: #E6DB74;">--from</span><span>,</span><span style="color: #E6DB74;"> serena-agent==1.6.1</span><span>,</span><span style="color: #E6DB74;"> serena</span><span>,</span><span style="color: #E6DB74;"> start-mcp-server</span><span>,</span></span> <span class="giallo-l"><span style="color: #E6DB74;"> --context</span><span>,</span><span style="color: #E6DB74;"> ide-assistant</span><span>,</span><span style="color: #E6DB74;"> --project</span><span>,</span><span style="color: #E6DB74;"> /opt/data/workspace/repos</span><span>]</span></span></code></pre> <p>Proxmox for infra state and power actions, context7 for library docs, <a rel="external" href="https://github.com/oraios/serena">serena</a> for LSP-backed code search over its cloned workspace. <code>${VAR}</code> placeholders resolve from <code>/opt/data/.env</code> at connect time, so <code>config.yaml</code> only ever holds the <em>shape</em> of the config; the real values live in a root-owned, mode-<code>0600</code> env file.</p> <p>GitHub, GitLab, and Kubernetes deliberately <em>aren't</em> MCP servers, just CLIs (<code>gh</code>, <code>glab</code>, <code>kubectl</code>) in the image, reading <code>GH_TOKEN</code>/<code>GITLAB_TOKEN</code> natively. <code>INSTRUCTIONS.md</code> tells the agent to ask for <code>--json</code>/<code>-o json</code> and filter it itself: cheaper in context than a bespoke MCP wrapper around a CLI that's already JSON-first.</p> <p>Kubernetes is the exception: <code>kubectl</code> is in the image, but wiring up a properly scoped kubeconfig is a bigger job than the other three, so I haven't done it yet. Coming later.</p> <h2 id="running-it-the-systemd-unit">Running it: the systemd unit<a class="post-anchor" href="#running-it-the-systemd-unit" aria-label="Anchor link for: running-it-the-systemd-unit"><span aria-hidden="true">#</span></a> </h2> <p>Docker gives the container; systemd keeps it alive.</p> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="ini"><span class="giallo-l"><span style="color: #88846F;"># sre-agent/hermes-agent.service</span></span> <span class="giallo-l"><span>[Unit]</span></span> <span class="giallo-l"><span style="color: #F92672;">StartLimitIntervalSec</span><span>=0</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>[Service]</span></span> <span class="giallo-l"><span style="color: #F92672;">ExecStartPre</span><span>=-/usr/bin/docker rm -f shleemypants</span></span> <span class="giallo-l"><span style="color: #F92672;">ExecStartPre</span><span>=/usr/bin/docker pull ${SRE_AGENT_IMAGE}</span></span> <span class="giallo-l"><span style="color: #F92672;">ExecStart</span><span>=/usr/bin/docker run --rm --name shleemypants \</span></span> <span class="giallo-l"><span> --network host \</span></span> <span class="giallo-l"><span> --volume /var/lib/hermes:/opt/data \</span></span> <span class="giallo-l"><span> ${SRE_AGENT_IMAGE} gateway run</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span style="color: #F92672;">Restart</span><span>=always</span></span> <span class="giallo-l"><span style="color: #F92672;">RestartSec</span><span>=10</span></span> <span class="giallo-l"><span style="color: #F92672;">TimeoutStartSec</span><span>=0</span></span></code></pre> <p>Three deliberate choices:</p> <ul> <li><strong><code>--rm</code></strong>: every start is a fresh container off the same known-good image. A bad run can't leave a half-provisioned writable layer for the next one to inherit.</li> <li><strong><code>StartLimitIntervalSec=0</code></strong>: retry forever instead of parking the unit as <code>failed</code>. A fresh VM has to pull the image over the network first, and that can flake.</li> <li><strong><code>TimeoutStartSec=0</code></strong>: that pull takes minutes on a cold VM; the 90s default would kill it mid-download and restart-loop forever.</li> </ul> <h2 id="standing-watch-the-supply-chain-cron">Standing watch: the supply-chain cron<a class="post-anchor" href="#standing-watch-the-supply-chain-cron" aria-label="Anchor link for: standing-watch-the-supply-chain-cron"><span aria-hidden="true">#</span></a> </h2> <p>One scheduled job: a supply-chain audit across every repo, every three days. <a rel="external" href="https://github.com/google/osv-scanner">OSV-Scanner</a> does the advisory matching; the agent adds the heuristics it misses: unpinned inputs on a moving branch, <code>:latest</code> tags, unpinned CI includes. Critical/high findings go to Telegram. It never patches.</p> <p>Hermes has no declarative cron config, so the job lives in runtime state (<code>/opt/data/cron/jobs.json</code>), created imperatively by a systemd oneshot at boot.</p> <h2 id="shipping-it-the-deploy-pipeline">Shipping it: the deploy pipeline<a class="post-anchor" href="#shipping-it-the-deploy-pipeline" aria-label="Anchor link for: shipping-it-the-deploy-pipeline"><span aria-hidden="true">#</span></a> </h2> <p>Deploys trigger on tags matching <code>sre-agent/x.y.z</code>. Recreating the VM <em>is</em> the deploy, so CI's job is just to build the image and hand OpenTofu the secrets:</p> <pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="yaml"><span class="giallo-l"><span style="color: #F92672;">opentofu-apply-production-sre-agent</span><span>:</span></span> <span class="giallo-l"><span style="color: #F92672;"> variables</span><span>:</span></span> <span class="giallo-l"><span style="color: #F92672;"> TF_VAR_sre_agent_release_tag</span><span>:</span><span style="color: #E6DB74;"> "$CI_COMMIT_TAG"</span></span> <span class="giallo-l"><span style="color: #F92672;"> TF_VAR_sre_agent_telegram_allowed_users</span><span>:</span><span style="color: #E6DB74;"> "$TELEGRAM_SHLEEMYPANTS_ALLOWED_USERS"</span></span> <span class="giallo-l"><span style="color: #88846F;"> # ...openrouter key, github PAT, gitlab token, context7 key, gatus token</span></span> <span class="giallo-l"><span style="color: #F92672;"> rules</span><span>:</span></span> <span class="giallo-l"><span> -</span><span style="color: #F92672;"> if</span><span>:</span><span style="color: #E6DB74;"> '$CI_COMMIT_TAG =~ /^sre-agent\/\d+\.\d+\.\d+$/'</span></span></code></pre> <p>Every secret lives as a masked GitLab CI variable and travels into the cloud-init snippet as an OpenTofu template value, so nothing lands in a plaintext file on the runner. It works, but using GitLab CI variables as the root of trust is the weakest link in the whole setup, and a proper secret store is on the list.</p> <h2 id="what-s-still-open">What's still open<a class="post-anchor" href="#what-s-still-open" aria-label="Anchor link for: what-s-still-open"><span aria-hidden="true">#</span></a> </h2> <p>A few things I'd still call unfinished, honestly:</p> <ul> <li><strong>Secret injection.</strong> Everything reaches the VM by being templated into the cloud-init snippet, which means the secrets also sit in OpenTofu state and in a snippet file on the Proxmox host. It works and it's mode-<code>0600</code> at rest, but the right answer is fetching them at boot from an actual secret store, along the lines of what Hermes' <a rel="external" href="https://hermes-agent.nousresearch.com/docs/user-guide/security#overview">security guide</a> describes for credential handling.</li> <li><strong>Kubernetes.</strong> As above: <code>kubectl</code> ships in the image, but scoping an RBAC role tightly enough to hand over is its own project.</li> <li><strong>More permissions.</strong> Right now the agent can read state and power-cycle VMs, nothing more. Reading guest logs and restarting individual services are the obvious next steps.</li> </ul> <h2 id="closing-thoughts">Closing thoughts<a class="post-anchor" href="#closing-thoughts" aria-label="Anchor link for: closing-thoughts"><span aria-hidden="true">#</span></a> </h2> <p>The goal was narrow on purpose: a first integration of an AI agent doing real SRE work, secure enough to actually leave running. Not an autonomous operator; a bounded one.</p> <p>The interesting problem turned out to be <em>where</em> to enforce the bounds. A prompt saying "alert before acting" is a suggestion. A Proxmox role without <code>VM.Config.*</code>, a GitLab account capped at Developer, an immutable image, a VM rebuilt rather than patched: those hold whether or not the model cooperates. Getting that layering right was the actual design work.</p> <p>What's left open is the honest measure of <em>first</em>: better secret delivery, Kubernetes, a wider permission set. Each widens the blast radius, so each lands on its own.</p> <p>For now it reads state, power-cycles VMs, audits dependencies every three days, and complains about my commit messages. Right amount of autonomy to start with.</p> <hr /> <section class="alert note" role="note" aria-labelledby="Yo8JWS1N"> <div class="alert-icon alert-icon-info"></div> <div class="alert-content" role="presentation"> <strong id="Yo8JWS1N" class="alert-title" aria-hidden="true">License</strong> <p>This article is licensed under the <a rel="external" href="https://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0 license</a>.</p> </div> </section>