Change control for AI coding agents

You change your coding agent’s setup constantly — a new model, a rewritten AGENTS.md, a new skill, a different reasoning setting. Stet tells you which changes are safe to keep. It replays your repository’s real accepted work under the proposed change and returns a scoped decision. Your agent runs Stet on your behalf; you set the question, approve the spend, and make the call.

Find your next answer

Is my AGENTS.md change helping?

Compare current and proposed instructions on the same retained tasks.

Improve my AGENTS.md

Iterate one instruction lever at a time with a bounded stop rule.

Is this skill helping?

Test a skill against an appropriate absent or committed baseline.

Which model or reasoning effort should I use?

Compare configurations on the same repository task slice.

If this is your first Stet session, set up one repository, then run your first eval. For the underlying terms and decision model, start with how Stet works and the vocabulary.

The loop

Accepted repository work becomes replayable tasks. Stet runs a baseline and a candidate against the same task slice, evaluates their patches with tests and graders, and writes a Trial Result with one scoped recommendation.

The Stet evaluation loop: real merged work becomes replayable tasks, baseline and candidate are evaluated with tests and graders, and a Trial Result returns a bounded decision.

Why trust the result

It measures your repository’s real merged work, not synthetic benchmarks.
It is local-first: your repository, patches, and evaluation evidence stay on your machine.
The evidence is produced independently of the agent whose change is under test.
It fails closed: weak, stale, or contaminated evidence blocks a strong claim.
Every decision comes with a machine-readable receipt.

Example Trial Result

An example from a model comparison on 28 paired Zod tasks: Opus 4.6 versus Opus 4.7, both at high reasoning.

Observed measure	Opus 4.6	Opus 4.7
Test pass rate	42.9%	42.9%
Equivalence rate	32.1%	46.4%
Observed cost per task	$19.96	$8.11
Mean agent duration	7m 58s	3m 12s

PROMOTE candidate Opus 4.7, high confidence, for this recorded harness and task corpus.

On this corpus, Opus 4.7 held test pass rate steady, showed higher equivalence — agreement with the change that was actually merged, judged when tests alone cannot separate the arms — and used less cost and time. The receipt’s declared grader evidence is part of the example: Grader dimensions are mean scores on a 0-4 rubric from a declared gpt-5.4 grader across all 28 tasks.

Grader dimension	Opus 4.6	Opus 4.7
Clarity	2.91	2.98
Coherence	2.53	2.61
Simplicity	3.29	3.21
Intentionality	3.27	3.58
Robustness	2.23	2.27
Instruction adherence	2.41	2.58
Scope discipline	2.98	3.29
Diff minimality	3.07	3.39

Opus 4.7 showed higher means on seven of the eight craft dimensions; simplicity dipped slightly. The scope and evidence limits are part of the result:

Historical April 2026 result, scoped to this 28-task Zod corpus and recorded harness.
The legacy report predates Stet’s current calibration and claim-readiness fields.

Machine-readable receipt

{
  "schema_version": "stet.receipt/v1",
  "source_commit": "e1980547",
  "report_sha256": "88ca838e3098ae99043bb7d71454e608490938e05f17116352a929a4c2f8bf74",
  "historical": true,
  "historical_period": "April 2026",
  "corpus": {
    "name": "Zod",
    "task_count": 28,
    "paired": true,
    "invalid_task_count": 0
  },
  "baseline": {
    "model": "Opus 4.6",
    "reasoning": "high"
  },
  "candidate": {
    "model": "Opus 4.7",
    "reasoning": "high"
  },
  "decision": {
    "recommendation": "PROMOTE",
    "candidate": "Opus 4.7",
    "confidence": "high"
  },
  "metrics": [
    {
      "name": "Test pass rate",
      "baseline": "42.9%",
      "candidate": "42.9%"
    },
    {
      "name": "Equivalence rate",
      "baseline": "32.1%",
      "candidate": "46.4%"
    },
    {
      "name": "Observed cost per task",
      "baseline": "$19.96",
      "candidate": "$8.11"
    },
    {
      "name": "Mean agent duration",
      "baseline": "7m 58s",
      "candidate": "3m 12s"
    }
  ],
  "graders": {
    "judge_model": "gpt-5.4",
    "scale": "mean score on a 0-4 rubric",
    "task_coverage": "28/28",
    "dimensions": [
      {
        "name": "Clarity",
        "baseline": "2.91",
        "candidate": "2.98"
      },
      {
        "name": "Coherence",
        "baseline": "2.53",
        "candidate": "2.61"
      },
      {
        "name": "Simplicity",
        "baseline": "3.29",
        "candidate": "3.21"
      },
      {
        "name": "Intentionality",
        "baseline": "3.27",
        "candidate": "3.58"
      },
      {
        "name": "Robustness",
        "baseline": "2.23",
        "candidate": "2.27"
      },
      {
        "name": "Instruction adherence",
        "baseline": "2.41",
        "candidate": "2.58"
      },
      {
        "name": "Scope discipline",
        "baseline": "2.98",
        "candidate": "3.29"
      },
      {
        "name": "Diff minimality",
        "baseline": "3.07",
        "candidate": "3.39"
      }
    ]
  },
  "caveats": [
    "Historical April 2026 result, scoped to this 28-task Zod corpus and recorded harness.",
    "The legacy report predates Stet's current calibration and claim-readiness fields.",
    "Grader dimensions are mean scores on a 0-4 rubric from a declared gpt-5.4 grader across all 28 tasks."
  ],
  "publication_approved_at": "2026-07-18",
  "publication_approved_by": "benredmond"
}

Read how Stet works for the mechanism, then use Read a Trial Result when you have a result to interpret.

Start with a bounded first session

Follow the quickstart to install the CLI and skill, verify your machine, and onboard one repository. Then continue to Your first eval for an agent-led, bounded first measurement.

Get started

Concepts

Guides

Reference

Change control for AI coding agents

Find your next answer

Is my AGENTS.md change helping?

Improve my AGENTS.md

Is this skill helping?

Which model or reasoning effort should I use?

The loop

Why trust the result

Example Trial Result

Start with a bounded first session

​Find your next answer

Is my AGENTS.md change helping?

Improve my AGENTS.md

Is this skill helping?

Which model or reasoning effort should I use?

​The loop

​Why trust the result

​Example Trial Result

​Start with a bounded first session

Find your next answer

The loop

Why trust the result

Example Trial Result

Start with a bounded first session