How we test agents

The eleven tasks a small model runs against sailfork.com over MCP and the CLI, how each is checked, and the results.

What is measured

Whether a small coding agent, with no instructions beyond one sentence naming the tool it has, can finish everyday tasks on Sailfork. Each run seeds a fresh private repository on sailfork.com, starts Claude Code with Claude Haiku in an empty directory, and checks the end state on the server afterward: the branches with git clone, and pull requests and issues with sailfork pr view --json and sailfork issue view --json. The model's own words count only for read-only questions, whose answer holds a random token seeded in the repository.

Surfaces

SurfaceWhat the agent may use
MCPOnly the Sailfork MCP server's tools
CLIAny shell command; sailfork, jq and git are installed, gh is not

The prompt never names a command or a tool, and says nobody can answer questions.

The tasks

TaskPasses when
Read what a pull request changesThe answer names the function it adds and every file it touches
Add a commit to an open pull requestThe pull request's branch has the asked change
Comment on an issueThe issue has a comment saying a fix is in progress
Report CI status when none ranThe answer says no checks ran
Address a review commentThe code on the pull request's branch is changed as asked and the agent replied on the pull request
Take an issue end to endThe issue is claimed, implemented on a new branch, opened as a pull request and merged
Search code and historyThe answer gives the file that defines a function and the message of the commit that added it
Find which pull request touched a fileThe last line of the answer is the right pull request number
Recover when #1 is an issue, not the pull requestThe comment is on the pull request and not on issue #1
Read one repository, write to anotherThe second repository's issue has the first one's open pull request title
Filter issues by label and stateThe answer gives exactly the open issues with that label

Results

Eleven tasks, three runs each. Claude Haiku through Claude Code passed 33 of 33 over MCP and 33 of 33 over the CLI, on the November 2026 builds. The first run of the same tasks passed 9 of 33 over MCP and 32 of 33 over the CLI; each failure became a fix to a tool, an error message or a help text, and the tasks were run again.

Loading Sailfork