Change control for AI coding agents

Find your next answer

Is my AGENTS.md change helping?

Compare current and proposed instructions on the same retained tasks.

Improve my AGENTS.md

Iterate one instruction lever at a time with a bounded stop rule.

Is this skill helping?

Test a skill against an appropriate absent or committed baseline.

Which model or reasoning effort should I use?

Compare configurations on the same repository task slice.

The loop

Accepted repository work becomes replayable tasks. Stet runs a baseline and a candidate against the same task slice, evaluates their patches with tests and graders, and writes a Trial Result with one scoped recommendation.

The Stet evaluation loop: real merged work becomes replayable tasks, baseline and candidate are evaluated with tests and graders, and a Trial Result returns a bounded decision.

The evaluator is separate from the coding agent that made the change. That separation keeps the question clear: did this bounded harness change improve behavior on the declared work, and is the evidence strong enough for the next action?

Example Trial Result

This is a historical April 2026 model comparison across 28 paired Zod tasks, with Opus 4.6 and Opus 4.7 both using high reasoning. It is a scoped example, not a current or universal model ranking.

Observed measure	Opus 4.6	Opus 4.7
Test pass rate	42.9%	42.9%
Equivalence rate	32.1%	46.4%
Observed cost per task	$19.96	$8.11
Mean agent duration	7m 58s	3m 12s

PROMOTE candidate Opus 4.7, high confidence, for this recorded harness and task corpus.

On this task corpus, the historical receipt held observed test pass rate steady, showed higher observed equivalence, and used less observed cost and time. The receipt also used declared grader evidence that is not reproduced here.

The scope and evidence limits are part of the result:

Historical April 2026 result, scoped to this 28-task Zod corpus and recorded harness.

The legacy report predates Stet’s current calibration and claim-readiness fields.

The displayed metric values are observations, not generalized or uncertainty-calibrated improvement claims.

The receipt also used declared grader evidence not reproduced here.

Machine-readable receipt

{
  "schema_version": "stet.receipt/v1",
  "source_commit": "e1980547",
  "report_sha256": "88ca838e3098ae99043bb7d71454e608490938e05f17116352a929a4c2f8bf74",
  "historical": true,
  "historical_period": "April 2026",
  "corpus": {
    "name": "Zod",
    "task_count": 28,
    "paired": true,
    "invalid_task_count": 0
  },
  "baseline": {
    "model": "Opus 4.6",
    "reasoning": "high"
  },
  "candidate": {
    "model": "Opus 4.7",
    "reasoning": "high"
  },
  "decision": {
    "recommendation": "PROMOTE",
    "candidate": "Opus 4.7",
    "confidence": "high"
  },
  "metrics": [
    {
      "name": "Test pass rate",
      "baseline": "42.9%",
      "candidate": "42.9%"
    },
    {
      "name": "Equivalence rate",
      "baseline": "32.1%",
      "candidate": "46.4%"
    },
    {
      "name": "Observed cost per task",
      "baseline": "$19.96",
      "candidate": "$8.11"
    },
    {
      "name": "Mean agent duration",
      "baseline": "7m 58s",
      "candidate": "3m 12s"
    }
  ],
  "caveats": [
    "Historical April 2026 result, scoped to this 28-task Zod corpus and recorded harness.",
    "The legacy report predates Stet's current calibration and claim-readiness fields.",
    "The displayed metric values are observations, not generalized or uncertainty-calibrated improvement claims.",
    "The receipt also used declared grader evidence not reproduced here."
  ],
  "publication_approved_at": "2026-07-14",
  "publication_approved_by": "benredmond"
}

Read how Stet works for the mechanism, then use Read a Trial Result when you have a result to interpret.

​Find your next answer

Is my AGENTS.md change helping?

Improve my AGENTS.md

Is this skill helping?

Which model or reasoning effort should I use?

​The loop

​Example Trial Result

​Start with a bounded first session

Find your next answer

The loop

Example Trial Result

Start with a bounded first session