AI SRE: giving my homelab an SRE with an attitude problem

{} {} :: # # # # # #

In One-click homelab I rebuilt my homelab so that the whole thing could be recreated from scratch in about an hour. That solved the "did I back that up" anxiety. It didn't solve the other anxiety: I'm the only person watching this stuff, and I don't always notice when something is on fire.

So I gave the homelab its own on-call engineer. But that means tackling all the usual problems with agents: operating system rotting, tight permission scoping, prompt injection, and of course reproducibility, as always.

This post walks through how I built it: a disposable Ubuntu VM running the Hermes agent connected to Telegram, with the necessary tools baked in so it can operate safely in the environment.

Hermes Agent in a nutshell

I used Hermes, which turned out to be the right building block for a few reasons beyond "it can call MCP tools":

First problem: operating system rotting

By operating system rotting I mean the degradation that happens when an autonomous agent keeps modifying a system to fix immediate problems.

Fortunately Hermes gives us a solution: running it in a container. The image itself is immutable, read-only to the runtime user, and everything the agent could possibly mutate (config, sessions, memory, skills, logs) is confined to a single bind-mounted directory.

Second problem: giving Hermes the toolset

Now the agent needs to actually be able to touch the environment it's supposed to be watching. It gets tools for:

Hermes' own docs offer a few ways to add tools: npx/uvx on demand, apt-get install at runtime (lost on restart), a derived image, or a sidecar.

For tools needed on every boot, they recommend the derived image:

sre-agent/Dockerfile (click to expand) 
ARG HERMES_TAG=v2026.8.3
FROM docker.io/nousresearch/hermes-agent:${HERMES_TAG}

ARG OSV_SCANNER_VERSION=1.9.2
ARG OSV_SCANNER_SHA256=<checksum pinned here>

USER root
RUN apt-get update && apt-get install -y --no-install-recommends \
      ca-certificates curl gnupg gh glab \
    # kubectl from the official k8s apt repo, not Debian's, which only supports
    # ±1 minor from the API server
    && install -m 0755 -d /etc/apt/keyrings \
    && curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key \
         | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg \
    && echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
         https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" \
         > /etc/apt/sources.list.d/kubernetes.list \
    && apt-get update && apt-get install -y --no-install-recommends kubectl \
    # osv-scanner: downloaded, checksummed, installed to /usr/local/bin (not packaged)
    && curl -fsSLo /usr/local/bin/osv-scanner \
         "https://github.com/google/osv-scanner/releases/download/v${OSV_SCANNER_VERSION}/osv-scanner_linux_amd64" \
    && echo "${OSV_SCANNER_SHA256}  /usr/local/bin/osv-scanner" | sha256sum -c - \
    && chmod +x /usr/local/bin/osv-scanner \
    && rm -rf /var/lib/apt/lists/*
USER hermes

Third problem: guardrailing its permissions

Tools are only half the job. Each credential also needs the least standing power that still works, enforced by the platform, not by the model choosing to behave:

Fourth problem: prompt injection

Everything above scopes what the agent can do if I ask it something dumb, not what happens if a log line, an issue, or a fetched webpage carries instructions of its own.

OpenRouter, which every model call already routes through, has a guardrail for that: OWASP-derived regex detection for instruction-override, fake system/developer-mode, prompt extraction, jailbreaks, and basic obfuscation (Base64, scrambled letters). Flag, redact, or block per match; Shleemypants is set to block.

Hermes adds its own layers on the VM side: context files are scanned before they reach the prompt (ignore-prior-instructions phrasing, hidden HTML comments, attempts to read .env or .netrc, credential exfil via curl, invisible Unicode), tirith scans commands before execution for homograph URLs and curl | bash patterns, terminal working directories are allowlisted, and destructive commands need explicit approval.

The honest gap is indirect injection: a poisoned issue body, a webpage a tool fetched, an MCP response. Those land in context unfiltered. Regex misses novel phrasing, so none of this is the real backstop: the scoped credentials above are what cap the damage when something gets through.

Giving the agent a personality

SOUL.md sets the voice, deliberately not corporate, so uncertainty and annoyance stay visible instead of flattened into a neutral status line:

"How you think you gonna push to prod while you're standin' in it, you dumb ass three-dimensional monkey ass dummy?"

INSTRUCTIONS.md, loaded at every conversation start, sets the rules: what it's forbidden from touching, and when to alert and wait instead of acting. Prompt text, not a sandbox; enforcement happens one layer down, in what tools it's physically capable of calling.

Rick and Morty-style meeseeks-brained character at a desk captioned "Shleemypants here, what's up?"

How the pipeline works

A sre-agent/x.y.z tag drives both stages: build the image, then replace the VM.

Build. sre-agent/Dockerfile (shown above, in the toolset section) extends the upstream nousresearch/hermes-agent image, pinned to a HERMES_TAG:

build-sre-agent-image:
  script:
    - ref="${CI_COMMIT_TAG#sre-agent/}"
    - docker build --pull -t "$IMAGE:$ref" sre-agent
    - docker push "$IMAGE:$ref"

Provision. The VM boots from a stock Ubuntu cloud image, fetched once and cached (overwrite = false skips re-fetching on a re-apply). Everything else (systemd units, config.yaml, SOUL.md, INSTRUCTIONS.md, every secret) gets templated into a cloud-init snippet:

resource "proxmox_virtual_environment_file" "cloud_config" {
  content_type = "snippets"
  source_raw {
    file_name = "sre-agent.cloud-config.yaml"
    data = templatefile("${path.module}/tpl/cloud-config.yaml", {
      hermes_agent_service = file("${path.module}/../../sre-agent/hermes-agent.service")
      agent_config         = file("${path.module}/../../sre-agent/config.yaml")
      agent_soul           = file("${path.module}/../../sre-agent/SOUL.md")
      agent_instructions   = file("${path.module}/../../sre-agent/INSTRUCTIONS.md")
      sre_agent_image      = local.sre_agent_image
      # ...secrets: proxmox token, gitlab token, openrouter key, telegram
      # token, github PAT, context7 key, gatus token
    })
  }
}

That snippet is a #cloud-config doc: it write_filess the systemd units and config into place, drops secrets into a root-owned /var/lib/hermes/.env, and its runcmd starts everything: qemu-guest-agent, docker, hermes-agent, hermes-cron-seed, gatus-heartbeat.timer.

What forces a new VM rather than an in-place update is a terraform_data resource tracking the release tag, wired into the VM's lifecycle:

resource "terraform_data" "release" {
  input = local.sre_agent_image
}

resource "proxmox_virtual_environment_vm" "sre_agent" {
  # ...
  lifecycle {
    replace_triggered_by = [
      terraform_data.release,
      proxmox_virtual_environment_file.cloud_config,
    ]
  }
}

Bump the tag, or edit anything templated into cloud-init (an edited systemd unit, a new config.yaml) and the VM gets replaced, not reconfigured. Recreating the VM is the deploy.

That's a deliberate trade: replace-not-reconfigure means zero in-place state to reason about, at the cost of wiping conversational memory on every release. A smarter setup could diff the cloud-init payload and only bounce the systemd unit when it actually changed, keeping the VM, and the agent's memory, alive across releases. For now that's more machinery than the problem deserves: one VM, one container, releases that aren't exactly frequent. Operational simplicity over state management: KISS, until it isn't.

What it's actually allowed to touch

Not every credential turns into an MCP server. config.yaml wires up MCP for the things that benefit from structured tool calls:

mcp_servers:
  proxmox:
    command: npx
    args: ["-y", "proxmox-mcp-server@0.2.0"]
    env:
      PVE_READONLY: "false"
      PROXMOX_TOKEN_SECRET: ${PROXMOX_MCP_TOKEN_SECRET}

  context7:
    command: npx
    args: ["-y", "@upstash/context7-mcp@3.1.0"]
    env:
      CONTEXT7_API_KEY: ${CONTEXT7_API_KEY}

  serena:
    command: uvx
    args: [--from, serena-agent==1.6.1, serena, start-mcp-server,
           --context, ide-assistant, --project, /opt/data/workspace/repos]

Proxmox for infra state and power actions, context7 for library docs, serena for LSP-backed code search over its cloned workspace. ${VAR} placeholders resolve from /opt/data/.env at connect time, so config.yaml only ever holds the shape of the config; the real values live in a root-owned, mode-0600 env file.

GitHub, GitLab, and Kubernetes deliberately aren't MCP servers, just CLIs (gh, glab, kubectl) in the image, reading GH_TOKEN/GITLAB_TOKEN natively. INSTRUCTIONS.md tells the agent to ask for --json/-o json and filter it itself: cheaper in context than a bespoke MCP wrapper around a CLI that's already JSON-first.

Kubernetes is the exception: kubectl is in the image, but wiring up a properly scoped kubeconfig is a bigger job than the other three, so I haven't done it yet. Coming later.

Running it: the systemd unit

Docker gives the container; systemd keeps it alive.

# sre-agent/hermes-agent.service
[Unit]
StartLimitIntervalSec=0

[Service]
ExecStartPre=-/usr/bin/docker rm -f shleemypants
ExecStartPre=/usr/bin/docker pull ${SRE_AGENT_IMAGE}
ExecStart=/usr/bin/docker run --rm --name shleemypants \
    --network host \
    --volume /var/lib/hermes:/opt/data \
    ${SRE_AGENT_IMAGE} gateway run

Restart=always
RestartSec=10
TimeoutStartSec=0

Three deliberate choices:

Standing watch: the supply-chain cron

One scheduled job: a supply-chain audit across every repo, every three days. OSV-Scanner does the advisory matching; the agent adds the heuristics it misses: unpinned inputs on a moving branch, :latest tags, unpinned CI includes. Critical/high findings go to Telegram. It never patches.

Hermes has no declarative cron config, so the job lives in runtime state (/opt/data/cron/jobs.json), created imperatively by a systemd oneshot at boot.

Shipping it: the deploy pipeline

Deploys trigger on tags matching sre-agent/x.y.z. Recreating the VM is the deploy, so CI's job is just to build the image and hand OpenTofu the secrets:

opentofu-apply-production-sre-agent:
  variables:
    TF_VAR_sre_agent_release_tag: "$CI_COMMIT_TAG"
    TF_VAR_sre_agent_telegram_allowed_users: "$TELEGRAM_SHLEEMYPANTS_ALLOWED_USERS"
    # ...openrouter key, github PAT, gitlab token, context7 key, gatus token
  rules:
    - if: '$CI_COMMIT_TAG =~ /^sre-agent\/\d+\.\d+\.\d+$/'

Every secret lives as a masked GitLab CI variable and travels into the cloud-init snippet as an OpenTofu template value, so nothing lands in a plaintext file on the runner. It works, but using GitLab CI variables as the root of trust is the weakest link in the whole setup, and a proper secret store is on the list.

What's still open

A few things I'd still call unfinished, honestly:

Closing thoughts

The goal was narrow on purpose: a first integration of an AI agent doing real SRE work, secure enough to actually leave running. Not an autonomous operator; a bounded one.

The interesting problem turned out to be where to enforce the bounds. A prompt saying "alert before acting" is a suggestion. A Proxmox role without VM.Config.*, a GitLab account capped at Developer, an immutable image, a VM rebuilt rather than patched: those hold whether or not the model cooperates. Getting that layering right was the actual design work.

What's left open is the honest measure of first: better secret delivery, Kubernetes, a wider permission set. Each widens the blast radius, so each lands on its own.

For now it reads state, power-cycles VMs, audits dependencies every three days, and complains about my commit messages. Right amount of autonomy to start with.