Skip to content

Methodology

The first AI coding agents we tried would happily write code that referenced functions that didn't exist, or invent the third argument to a method when only two were defined. The fix everyone reached for was "give the LLM more context" — bigger window, RAG over the codebase, retrieval-augmented this and that. We got tired of that approach because the failure mode was the same: a confident wrong answer, dressed up better.

The way Agent Smith works is different. Every change has to come from evidence in the codebase, and every claim the agent makes can be challenged by a different role before any code lands. That's the methodology. The rest of this page is what it looks like in practice.

The evidence contract

Every claim a role makes is a typed observation, not free text. Each observation carries a Concern, a Confidence (0–100), a Blocking flag, and an EvidenceMode. Evidence mode is the key idea:

  • AnalyzedFromSource — the observation is backed by something the role actually read in the codebase (a file path plus a line number).
  • Potential — the observation is the role thinking out loud about something that might be true but hasn't been verified.

Speculation surfaces; it doesn't get to pass as fact. And the framework enforces the reading part: a finding that claims source analysis of a file the master never actually read gets downgraded to Potential.

How a coding run flows

There is one coding pipeline, code. Its shape is simple to say: the ticket becomes a specification, and the specification is worked spec by spec, each spec checked before the next one starts.

The spec is the plan. After the code has been read, DeriveSpec cuts the ticket into an ordered set of specs on the ticket branch — goal, steps, "done when" per spec. Before it writes, the derivation looks at the repository through read-only tools, and the cut it posts to the ticket separates facts (each backed by a look it took) from assumptions (stated without one). A fresh reviewer objects to the cut before it is kept. When several specs share names, data or files, the series carries a contract that says who owns what, and a cut that collides is sent back. There is no second planning step and no approval gate: the master opens on the spec, and the spec is what gets verified. Details in Expectations.

Attack each spec before building it. Before the master starts, fresh reviewers open the code and answer BUILD, BUILD WITH CUTS or REFUSE for every spec a person hasn't approved word for word. Cuts are applied and the next round attacks the revised text, for up to four rounds, inside a share of the run's budget. A REFUSE that survives is asked once on the ticket. See the code pipeline.

Hand it back instead of guessing. A ticket that reads two ways, contradicts the repository, or can't be built as written is handed back at derivation, before any code. A ticket that asks for something that must not be done is refused even earlier, by the scope call, before a sandbox exists.

Check the premises before each spec. A spec states what it rests on. Before its work starts, a fresh instance checks those premises against the repositories as they are now — it can run one of the verify stages the repository declares, by its label. A premise that no longer holds hands the spec back with what was looked at. The check reports; it never rewrites the specification.

One master, real commands. The coding-agent-master skill does the spec's work: it edits the code and runs the repo's own build and tests via real commands, visible in the run timeline.

Verification the master doesn't grade. VerifySpec runs the verify stages each repository declares, then takes the delivery account: every criterion checked against the real branch, with citations that have to resolve, by a reader that never saw the master's reasoning. An outstanding criterion earns one repair pass. The run's verdict comes from these per-spec accounts and from nothing else.

A review after green. A fresh reviewer reads the spec's diff against it and the repository's principles. Its findings get one fix pass; a fix that doesn't verify is undone, and what the review still finds goes into the spec record, the spec_review artifact and the PR. The review never fails a spec that verified.

A run that stops half way still delivers honestly: verified specs ship on a ready PR with a "Not delivered" section, unverified work is reset, and a run that verified nothing leaves a draft PR with a failure banner. See Lifecycle.

Scan pipelines: master plus roles

The findings pipelines (security-scan, api-security-scan, legal-analysis) run a master that orchestrates specialist roles; every role emits typed observations under the evidence contract, and the delivered result is the master's curated triage backed by the deterministic scanner facts. The scan master works on a read-only tool surface — it can read, list and grep, and that's it. Details per pipeline under Reference → Pipelines.

Spec-first

The methodology works on two levels. The runs follow it: every change starts as a spec with a "done when" list, and ends as a spec record committed to the target repository under .agentsmith/specs/done/, with a one-line entry in that repository's context. The skills follow it too. Every skill in the agent-smith-skills catalog is YAML frontmatter plus a Markdown body, pinned by version, so the judgement a master exercises can be tuned without a release. See the Skills catalog.

What lands in the run directory

Every run leaves a record in the repository, committed with the change. The run directory is .agentsmith/runs/{run-id}/:

result.md — what got done. YAML frontmatter with the result, the duration, token counts and the cost block, then the changed files, the delivery account, declined criteria, decisions and the execution trail. A failed run leads with why. If pricing is configured for the agent, dollar costs are accurate; otherwise you get token counts.

plan.md — the master's own working plan for the run, when it wrote one.

Decisions — non-obvious choices the agent made during the run. "Picked 400 BadRequest over 404 Not Found because the OpenAPI spec already documents 400 for malformed input". The agent records these with the log_decision tool when something would surprise a future reader; they land in .agentsmith/decisions/<run-id>.yaml and in the run's event stream.

The specification itself lives next to the code: one YAML spec and one Markdown companion per spec under .agentsmith/specs/planned/, moved to .agentsmith/specs/done/ once it ran, and one manifest per series at .agentsmith/series/{base}.yaml, which also records where every part of the ticket went. The run directories are what the knowledge-base feature compiles when it builds the wiki across all your runs.

What about hallucinations

The evidence contract is the first mitigation. A finding without a file:line citation is a Potential observation by definition, and a claimed citation of a file nobody read is downgraded. The agent can still be wrong about the file:line it points at — but you can check, which is a much weaker class of failure than "the agent invented a function name".

The second is role separation. The instance that judges the work is never the one that did it: the cut reviewer, the premise check, the delivery account and the spec review each start fresh, with no channel to the master's reasoning. Asking the author whether it is done is persuasion; looking at the branch is not.

Why this order

Spec before code: agree on the WHAT while it's still cheap to disagree. Every correction you make to the cut is an argument you don't have on the PR.

Premises before work: a spec can go stale between the day it was cut and the day its spec runs. Checking costs one look; building on a false premise costs the spec.

Verification after every spec, not once at the end: each spec ends at its own done-list, so stopping is structural rather than a judgement the model has to reach, and a run that stops half way knows exactly which specs are through.

Review after green: the review reads verified work, so what it objects to is quality, not whether the thing works.

Next