K8gentS — Kubernetes RCA MCP Server

K8gentS ☸️🤖

A Human-in-the-Loop RCA Agent for Kubernetes — LLM Diagnosis, OPA-Bounded Remediation

📖 Overview

K8gentS is a Kubernetes diagnostic agent built around a single premise: an LLM is well-suited to *reason* about cluster failures, but unfit to *act* on them unsupervised. It continuously watches the cluster event stream, runs Warning-class events through a Gemini-powered root cause analysis (RCA) pipeline, and surfaces the top hypotheses with resolution steps to Slack. Every remediation is human-in-the-loop (HITL) — a reviewer approves or rejects via interactive buttons before anything mutates. Approved fixes execute inside an ephemeral Job whose blast radius is bounded by an OPA Gatekeeper admission policy at the API server, independent of RBAC. The reasoning is non-deterministic by design; the controls around it are not.

The goal: reduce MTTR on diagnosable failures without handing an LLM a kubectl apply.

🧠 The Hard Problem

Building the Kubernetes side of this is straightforward. The real challenge is making a diagnostic layer trustworthy when the engine behind it is fundamentally non-deterministic.

Traditional observability is built on guarantees. Alerts fire on known thresholds. Dashboards show reproducible numbers. Logs return consistent answers to the same query. SRE success depends on that predictability — it's what makes incident response repeatable and on-call sustainable.

An LLM-driven diagnostic layer breaks that contract. The same pod failure can produce three different plausible explanations across three different runs. Each may be coherent. Each may even be correct under different assumptions. But "plausible" is not the same as "right," and for infrastructure, the gap between them is where outages live.

Some of the specific problems I've been working through while building K8gentS:

1. Confidence scoring with an unbounded output space. The agent returns top-3 root causes with confidence metrics, but confidence in what, exactly? The model is not selecting from a fixed set of known failure modes — it's generating free-form hypotheses. A calibrated confidence score needs a reference distribution, and the distribution here is whatever the model happened to produce this run.

2. When to trust reasoning vs. fall back to deterministic checks. - THE ART Some failures (CrashLoopBackOff, OOMKilled) have well-traveled diagnostic paths and a deterministic check will be right every time. Others benefit from the model's ability to interpolate across signals. Drawing that line — and doing it at runtime — is non-trivial and where the art really lies.