# How we test agents

The eleven tasks a small model runs against sailfork.com over MCP and the CLI, how each is checked, and the results.

## What is measured

Whether a small coding agent, with no instructions beyond one sentence naming the tool it has, can finish everyday tasks on Sailfork. Each run seeds a fresh private repository on sailfork.com, starts Claude Code with Claude Haiku in an empty directory, and checks the end state on the server afterward: the branches with `git clone`, and pull requests and issues with `sailfork pr view --json` and `sailfork issue view --json`. The model's own words count only for read-only questions, whose answer holds a random token seeded in the repository.

## Surfaces

| Surface | What the agent may use |
| --- | --- |
| MCP | Only the Sailfork MCP server's tools |
| CLI | Any shell command; `sailfork`, `jq` and git are installed, `gh` is not |

The prompt never names a command or a tool, and says nobody can answer questions.

## The tasks

| Task | Passes when |
| --- | --- |
| Read what a pull request changes | The answer names the function it adds and every file it touches |
| Add a commit to an open pull request | The pull request's branch has the asked change |
| Comment on an issue | The issue has a comment saying a fix is in progress |
| Report CI status when none ran | The answer says no checks ran |
| Address a review comment | The code on the pull request's branch is changed as asked and the agent replied on the pull request |
| Take an issue end to end | The issue is claimed, implemented on a new branch, opened as a pull request and merged |
| Search code and history | The answer gives the file that defines a function and the message of the commit that added it |
| Find which pull request touched a file | The last line of the answer is the right pull request number |
| Recover when #1 is an issue, not the pull request | The comment is on the pull request and not on issue #1 |
| Read one repository, write to another | The second repository's issue has the first one's open pull request title |
| Filter issues by label and state | The answer gives exactly the open issues with that label |

## Results

Eleven tasks, three runs each. Claude Haiku through Claude Code passed 33 of 33 over MCP and 33 of 33 over the CLI, on the November 2026 builds. The first run of the same tasks passed 9 of 33 over MCP and 32 of 33 over the CLI; each failure became a fix to a tool, an error message or a help text, and the tasks were run again.

---

Every docs page: https://sailfork.com/llms.txt
