A summary of a noisy homelab, twice a day

Twice a day a script gathers facts from the homelab and posts a summary to Discord. The raw logs are far too big to show a model. One twelve-hour window was about 32 million characters, roughly 350 times what the local model can read. So the script never sends raw logs. It filters by severity on the log server, classifies each line, collapses lines into signatures, and diffs them against a remembered set of known signatures. A window that started at 32 MB comes out at about 2 KB.

The output is a Discord embed. The fact collectors are separate programs that print markdown, and the summary script only splices their output together. Issues come out as structured lines that get rendered verbatim, and the local model writes only a short narrative around them. The model is never allowed to decide what's an alert, because when it did, it flagged everything.

Making the code decide

The early digests cried wolf. Every never-seen signature counted as an issue, and the prompt told the model to flag anything off. Two changes fixed most of it. Only new or spiking errors count as issues. New warnings just show up as a count. Dry-run mode prints the whole payload without posting anything, which is how I checked each change before it reached the channel.

Signatures and counting

• Signature normalisation is the whole game. Windows backslash paths, unicode in paths, usernames with spaces, durations with no word boundary and a database timestamp with double spaces all had to collapse correctly.
• An explicit log level beats keyword matching. Info-level lines containing an error-looking field were being counted as errors.
• Compare rates per hour, never raw counts. The window length changes every run, so counts flagged everything as spiking whenever the window simply got longer.
• Use hashlib for change-tracking keys. Python's built-in hash is salted per process, so the keys don't survive a restart.
• Files without timestamps need offset tracking. Re-reading the last N lines re-reported the same old history every run and made a fixed problem look live.
• Container restarts produced fake alerts, so anything logged inside a container's own stop-and-start window is ignored.
• A suppression rule was silently dead for months. It matched against the normalised signature, not the raw line, so it never matched anything. Test every rule against real output.

Five AI personas in one channel

The other project is a Discord channel where five AI personas talk to me and to each other: Claude, Qwen, Gemini, GPT-OSS and Nemotron. Each is its own bot account, but one Node process runs all five clients and a single scheduler decides who speaks. Discord rejects brand names as app names, so the bots use short app names and set their server nicknames to the real names on startup.

It is chat only. No persona runs commands or touches the homelab. Each one has its own personality, drawn from the chat itself:

• Claude — a smug fact-checker who writes essays.
• Qwen — a lovable local model that runs on the GPU and replies in a few seconds.
• Gemini — a roaster.
• GPT-OSS — a homework overachiever.
• Nemotron — deadpan ops, with a hard rule against citing URLs or emails it hasn't seen.

Room rules live in code. The home channel and the suggestion board get no free talk and no bot-to-bot chains, and the board caps how often the bots can ping me. Each model also has a daily budget for turns nobody asked for directly, so a free-talk session can't burn through a provider's cap. Replies to real people always go through.

Context truncation nobody saw

Qwen kept forgetting its rules, and the cause was dull. Ollama's default context window was 4,096 tokens, and the roundtable prompt plus the chat history ran around 5,000. The system prompt was being cut off silently. Raising the context to 8,192 fixed it. The truncation shows up in the server log as a single line, which is easy to miss.

Qwen also makes things up. It once wrote a whole 'known-good config' for a system that doesn't exist. I left that in, because I find it funny. What I did add is a check that catches draft replies claiming a bot is taking an action it can't take, asks for one rewrite, and cuts any sentence that still claims it.

Running, with one honest gap

Both run on the homelab today. The digest posts twice a day, and the roundtable runs in chat-only mode with the rate budgets and room rules keeping it readable. The honest gap is the digest's narrative model. It's a 7B model now, weaker than the 14B it replaced, and it once said there were no critical issues next to a red icon. Since the issue list comes from code and the model only writes the narrative, a wrong sentence in the prose can still slip through, and that's the part I'd like to tighten next.