Open source Apache-2.0 Local-first Built on Pi
Sidebet is an open-source harness around your AI coding agent. It keeps your rules and memory consistent across sessions, runs the same task on different models side by side in isolated Git worktrees, and shows you the real diffs, test results and token costs before you keep one. Over time, your own history becomes evidence about which model works for your projects.
Illustrative output. The numbers come from a demo run, not a benchmark, and the models are examples.
The problem
If you use an AI coding agent every day, with Claude Code, Codex, Cursor, Pi or anything else, you have probably felt all three of these.
Conventions, slow test suites, files that must not change: they live in your head and in whichever chat you had open. Every new session starts from zero.
Rules and memory are injected into every session. You can see exactly what the model received, and why each item was included or left out.
Trying another model means redoing the task by hand and juggling branches. So you don't, and you never learn what you missed.
The same task runs on two or more models at once, each in its own Git worktree. Real diffs, real tests, real token counts, side by side.
Which model passes your tests more often? Which one do you actually prefer once you read the diff? Today that is a feeling.
Every run is recorded. sidebet odds shows pass rates, your preferences
and medians per model, with sample sizes, never blended into a score.
What it is
Pi runs the agent. Sidebet owns everything around it: context, memory, policy, isolation, review and history. Pi is never forked or patched; Sidebet uses its public SDK and the credentials it already holds.
Saying this plainly is the point.
sidebet 
AGENTS.md Pi already has.
Nothing to re-enter.
How it works
Everything below is implemented and tested today. The output shown is illustrative; the models are examples.
Sidebet inspects an existing repository and reports what it found: language,
package manager, test command, an existing AGENTS.md, and how many
models Pi has credentials for. It writes exactly one small file, never overwrites
it, and is idempotent.
sidebet init Sidebet initialized Project acme-shop Runtime pi 1.1.0 Language TypeScript Package mgr pnpm Rules AGENTS.md detected Models 7 available (default anthropic/claude-sonnet-5-5) Tests pnpm test Wrote .sidebet/config.json (commit it to share the project id) You're ready. sidebet run "Explain this repository" sidebet compare "Improve the search handler"
Rules are always injected. Memory is advisory, searched per task, and never
overrides a rule. --dry-run shows the whole plan, every context item
and the reason it was included, without calling a model. --readonly
removes write tools in code, not in the prompt, and verifies the tree afterwards.
sidebet memory add "Search must stay case-insensitive; never run regexes on user input" Saved mem_fa704c29 (project). Included in relevant future runs; explicit rules still take precedence. sidebet run "Make search case-insensitive" --dry-run Dry run: nothing will be executed, no model will be called, nothing is saved. Task Make search case-insensitive Mode run in your working tree Models anthropic/claude-sonnet-5-5 Tools read, bash, edit, write Policy pol_3a11f4fdbe16 Why each item native agents AGENTS.md ~410t - loaded by the runtime itself injected rules .sidebet/rules/conventions.md ~260t - always injected injected memory mem_fa704c29 ~21t - included: matched task Policy decisions protected paths: .env, .env.*, **/*.pem, **/*.key, **/id_rsa*, .git/** approval required (bash): rm -rf, git push, git reset --hard, git clean, sudo, curl | sh Estimated context: 691 tokens (281 injected by Sidebet, 410 loaded by the runtime, budget 12,000)
Two or more models run the same task concurrently, each in its own Git worktree
created from a snapshot of your working tree, uncommitted changes included. Your
files are never touched. Each attempt gets your test suite. Afterwards a metrics
table puts them side by side; ◂ marks the better value for that one
metric, and nothing is combined into a score.
sidebet compare "Make search case-insensitive and add a test for it" --models sonnet,gpt5 Comparing anthropic/claude-sonnet-5-5 vs openai/gpt-5.5 Attempts start from a snapshot of your working tree (2 uncommitted changes included). Your files are untouched. … Metrics
| A claude-sonnet-5-5 | B gpt-5.5 | |
|---|---|---|
| Status | completed | completed |
| Total time | 2m 58s ◂ | 3m 31s |
| Agent time | 2m 31s ◂ | 3m 02s |
| Time to first token | 1.4s ◂ | 2.1s |
| Time to first edit | 41s ◂ | 1m 12s |
| Turns | 6 | 9 |
| Tool calls | 14 (read 6, edit 3, bash 3, grep 1, write 1) | 21 (read 8, edit 5, bash 4, find 2, grep 1, write 1) |
| Tool errors / blocked | 0 / 0 | 0 / 1 |
| Input tokens | 45.1k ◂ | 57.9k |
| Output tokens | 3.1k ◂ | 3.8k |
| Total tokens | 48.2k ◂ | 61.7k |
| Cost | $0.21 | not reported |
| Files changed | 2 | 3 |
| Lines +/- | +38 / -12 | +71 / -19 |
| Tests | passed (14 passed) | passed (14 passed) |
◂ marks the better value for that single metric; metrics are not combined into a score. Cost is what the runtime reported; subscription logins often report none, and nothing is estimated. next sidebet review run_7f3a9c21 your working tree was not modified
Diffs open in a full-screen viewer with Pi's own rendering. You record which result you prefer and why. Applying previews the files, warns about your local edits, refuses on conflict, saves a backup ref first, changes only the working tree, and prints the command to undo it. Nothing is ever merged automatically.
sidebet review run_7f3a9c21 · A claude-sonnet-5-5 vs B gpt-5.5 ▸ View A diff View B diff Compare A and B Select preferred result Record review accepted or not, with a reason Apply a result to this repository Check out an attempt in a worktree Exit Recorded rev_48d693c9 · preferred A · accepted · "smaller diff, kept the existing test structure" · your call ◉ matched Apply A (anthropic/claude-sonnet-5-5) to /Users/you/acme-shop src/search.ts test/search.test.ts Applies to the working tree only (no commit, index untouched). A backup ref of your current tree is saved first. Apply these changes? [y/N] y Applied 2 files. Attempt kept at: refs/sidebet/runs/run_7f3a9c21/A To undo: git checkout refs/sidebet/backups/run_7f3a9c21 -- "src/search.ts" (for files that existed before)
Per model: runs, test pass rate (objective), how often you preferred it (human),
acceptance, median time, tokens and cost. Sample sizes are always shown. The
head-to-head table only uses runs where both models attempted the same task.
sidebet bet A records your prediction before you read the diffs;
odds reports how often your calls matched, a measure of your
judgement, not of the models.
sidebet odds ▗▟▀▙▖ sidebet v0.1.0 odds · acme-shop ▝▜▄▛▘ 31 finished runs · 31 compare · 26 with a recorded preference Model Runs Tests passed (objective) Preferred (human) Accepted Median time Median tokens Median cost anthropic/claude-sonnet-5-5 31 27/31 (87%) 19/26 (73%) 22/26 (85%) 3m 04s 52.1k $0.24 openai/gpt-5.5 31 24/31 (77%) 7/26 (27%) 18/26 (69%) 3m 48s 64.9k n/a Head-to-head (same task, both attempted) claude-sonnet-5-5 vs gpt-5.5: 31 runs; preferred 19-7; tests: 22 both passed, 5 only sonnet, 2 only gpt-5.5, 2 both failed (n=31) Your calls ◉◉◎◉◉◉◎◉◉◉◉◉ 21/26 matched your later preference A call is a `sidebet bet` placed before reviewing; it measures your prediction, not the models. • Test pass rate and preference are different evidence: tests are objective but only as good as your suite; preference is your judgement. They are shown separately and never blended. • Cost is shown only where the runtime reported one; missing costs are not estimated.
sidebet ui serves a read-only web view of the same records: every
run, attempts side by side, metrics, per-file diffs, tool calls, test output, the
exact context the model received, odds and memory. It binds to
127.0.0.1, has no write endpoints, and adds no dependencies.
sidebet ui ▗▟▀▙▖ sidebet v0.1.0 ui · acme-shop ▝▜▄▛▘ http://127.0.0.1:4242/ ctrl+c to stop Read-only. Serves this project's runs, attempts, diffs, context snapshots, odds and memory from ~/.sidebet/sidebet.db to your browser. Nothing leaves your machine. sidebet runs ID STATUS KIND MODELS WHEN TASK run_7f3a9c21 completed compare sonnet-5-5 vs gpt-5.5 2m ago Make search case-insensitive and add a test for it run_c41e08d2 completed compare sonnet-5-5 vs gpt-5.5 3h ago Add input validation to the checkout endpoint run_9be0a7f3 completed run sonnet-5-5 1d ago Explain the auth flow read-only run_2d7c5e10 timeout compare sonnet-5-5 vs gpt-5.5 2d ago Migrate the search index to the new schema
Everything around the agent
Each of these exists today and is covered by tests that run without paid model calls.
sidebet init
Git, package manager, test command, AGENTS.md, Pi models and skills. Writes one small config file. Idempotent.
sidebet run
A Pi-backed session with your rules and memory injected, tool activity streamed, usage and the exact change set recorded.
--readonly · --dry-run
Read-only exposes only read, grep, find and ls, blocks writes in code and verifies the tree afterwards. Dry run prints the whole plan and calls nothing.
sidebet compare
Several models, one task, each in its own worktree from a snapshot of your tree. Tests per attempt. A live dashboard while it runs.
sidebet review
Column-by-column metrics, full-screen diffs, a recorded review, and an apply that previews, backs up, refuses conflicts and prints the undo.
sidebet odds · bet
Pass rates, preferences and medians per model, sample sizes always shown. Place a call before reviewing and learn how good your instincts are.
sidebet runs · rerun
Every run, attempt and diff. Rerun with the original context snapshot or the current one, stated explicitly. Inputs are reproduced, not outputs.
sidebet context
What the agent would receive now, or the stored snapshot of any past run: each item injected, native or omitted, its tokens, and why.
sidebet memory
Project and global memory in SQLite with full-text search. Advisory: rules always win. Suggestions are shown, never saved without you.
sidebet policy check
Provider and model allow-lists, tool deny lists, protected paths and approval-required commands. The matrix says what is enforced, partial or unsupported.
sidebet ui · doctor
A read-only browser view of runs, diffs, context, odds and memory on 127.0.0.1. Doctor explains what Sidebet sees and why something was refused.
pi install ./extensions/pi
/sidebet-compare, /sidebet-review and /sidebet-odds inside Pi, plus your rules and relevant memory added to plain Pi sessions.
Trust
Sidebet separates instructions to a model from enforcement in code. A model can ignore an instruction. It cannot ignore a hook that blocks the tool call before it runs. Here is which is which.
policy.strict removes bash entirely.~/.sidebet/sidebet.db. Prompts go to the providers you chose, exactly as with Pi.
The complete list is in
docs/security.md.
sidebet policy check prints the same matrix for your own configuration.
Roadmap
Open core. The CLI, the core library and the Pi adapter are Apache-2.0 and stay that way. Everything in the first column exists; the rest is intent, not a promise.
Everything on this page runs on your machine today.
sidebet uiWhat we intend to build next, in roughly this order.
odds.env handling for worktrees so more test suites run unmodifiedsidebet export and import of run recordsNot built. The local CLI stays complete without it.
AgentRuntime interfaceNon-goals for now: accounts, cloud sync, a vector store, a plugin marketplace, automatic memory promotion, automatic merging.
Shared memory, team policy and hosted history are the cloud product we have not built. If you want it, say so. Interest decides what gets built next.
Get started
Sidebet reuses your Pi login and your AGENTS.md. There are no accounts
and nothing to re-enter. The walkthrough below ends with you comparing two models on a
real change and choosing which result to keep.
git clone https://github.com/jamesray/sidebet.git
cd sidebet
pnpm install && pnpm build
cd apps/cli && pnpm link --global # puts `sidebet` on your PATH
sidebet doctor
cd your-project
sidebet init
sidebet models # the models you are logged into in Pi
sidebet compare "Make search case-insensitive" --models <a>,<b>
sidebet review
pi install ./extensions/pi # adds /sidebet, /sidebet-compare, /sidebet-review, /sidebet-odds
Run the task on both. Read the diffs. Keep the receipts.