The eleven tasks a small model runs against sailfork.com over MCP and the CLI, how each is checked, and the results.
What is measured
Whether a small coding agent, with no instructions beyond one sentence naming the tool it has, can finish everyday tasks on Sailfork. Each run seeds a fresh private repository on sailfork.com, starts Claude Code with Claude Haiku in an empty directory, and checks the end state on the server afterward: the branches with git clone, and pull requests and issues with sailfork pr view --json and sailfork issue view --json. The model's own words count only for read-only questions, whose answer holds a random token seeded in the repository.
Surfaces
Surface
What the agent may use
MCP
Only the Sailfork MCP server's tools
CLI
Any shell command; sailfork, jq and git are installed, gh is not
The prompt never names a command or a tool, and says nobody can answer questions.
The tasks
Task
Passes when
Read what a pull request changes
The answer names the function it adds and every file it touches
Add a commit to an open pull request
The pull request's branch has the asked change
Comment on an issue
The issue has a comment saying a fix is in progress
Report CI status when none ran
The answer says no checks ran
Address a review comment
The code on the pull request's branch is changed as asked and the agent replied on the pull request
Take an issue end to end
The issue is claimed, implemented on a new branch, opened as a pull request and merged
Search code and history
The answer gives the file that defines a function and the message of the commit that added it
Find which pull request touched a file
The last line of the answer is the right pull request number
Recover when #1 is an issue, not the pull request
The comment is on the pull request and not on issue #1
Read one repository, write to another
The second repository's issue has the first one's open pull request title
Filter issues by label and state
The answer gives exactly the open issues with that label
Results
Eleven tasks, three runs each. Claude Haiku through Claude Code passed 33 of 33 over MCP and 33 of 33 over the CLI, on the November 2026 builds. The first run of the same tasks passed 9 of 33 over MCP and 32 of 33 over the CLI; each failure became a fix to a tool, an error message or a help text, and the tasks were run again.