digline MCP server

digline

Regression testing for LLM applications: sample your reference, declare a floor on the checks that need one, and a score drop is read against how much that score moves on its own. A flip from pass to fail is always a regression.

_{CI is the merge queue's latest run — the gate a change passes on its way to main; a red can mean the queue turned a pull request away while main did not move.}

*The agent above is the one editing your code, not one running in production. digline runs in your pipeline over the suite you committed: it reads no traces.*

Your prompt worked on Tuesday. On Thursday it works a little less — not enough to break, enough for a user to notice in two weeks. No ordinary test catches it, because there is no correct output to compare against, only a better or a worse one.

digline gives you an approved reference — the baseline — and on every change tells you whether you are below it: which case, which check, by how much. The baseline is a JSON file in your repository, so it goes through code review and it rolls back with git. No server, no account, no network call you have not configured yourself.

When a check uses a model to grade, that grader does not always agree with itself. Ask digline to sample and it records how much the score moved on its own. Declare floor_standard_errors on that check and that measurement becomes a floor: a drop outside it is reported with the interval on its line, and a drop inside it is not a regression — the headline counts it, and explain prints the interval. Declare nothing and the samples are still recorded, but the tolerance decides alone.

$ digline compare --suite suite.py --run latest
2 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

how-do-i-return · llm_rubric · Score fell from 1.000000 to 0.700000.
how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).

Why digline

Most evaluation tools tell you whether an output is below a threshold. digline also tells you whether it is *worse than it was* — the drift from 0.91 to 0.78 that trips no threshold and is the first thing a user feels.

The suite is Python, not YAML: a judge is an object, a target is a function, and what may leave a perimeter is declared in code — none of which a configuration file expresses without reinventing a language. Where a suite *is* plain data it can be TOML instead, and the two forms build the same objects — what TOML cannot express, it refuses by name rather than half-supporting. Built for teams shipping LLM features for someone else, who have to show a customer what was tested, when, under which commit, and who approved it.

Wondering how digline differs from promptfoo, DeepEval, or observability platforms? See How digline compares.

Quickstart