Multi-Agent Experimentation is a testbed where AI agents (Claude, GPT, a small local model) play Minecraft, and the question is what they manage and what it costs. Every decision an agent makes is written to a log file as one line of JSON, and every number the project publishes is worked out from those files by code, never typed in by hand. This is how that works, why, and what it costs.
There's no database and no dashboard here: just files and scripts on one machine. The logs aren't public either, so readers can check the code that counts, not the raw data. The numbers below are from the project's final 605 logs (October 23 to November 14, 2026).
What JSONL is
JSONL (JSON Lines) is a text file with one complete JSON object on each line:
{"step": 1, "skill": "gather", "result": "success"}
{"step": 2, "skill": "craft", "result": "failed"}
{"step": 3, "skill": "flee", "result": "success"}
That's the whole format. It suits logs better than one big JSON array for three reasons:
- Adding a line doesn't touch the rest. A JSON array has to be closed with
], so adding to it means rewriting the end of the file. A JSONL file is only ever appended to. - A crash loses almost nothing. If the program dies mid-run, every finished line is still valid JSON. A half-written array is unreadable until it's repaired.
- It can be read one line at a time, so a script never has to hold a huge file in memory, and
ordinary tools like
grep,headandwc -lwork on it.
How the testbed uses it
An agent's run is a loop: look at the world, ask the model what to do, do it, repeat. Each pass is a step, and each step becomes one line in that run's file. The code that writes it is a single call:
write(step: LoggedStep): void {
appendFileSync(this.path, JSON.stringify(step) + "\n");
}
Here is a real step from October 31, 2026, shortened and spread over several lines to read (in the file it's one line):
{
"step": 6,
"observation": {
"health": 20, "food": 18, "isDay": true,
"inventory": { "oak_log": 4, "dark_oak_log": 30 },
"hostiles": [{ "name": "creeper", "distance": 16 }]
},
"decision": {
"skill": "hunt",
"args": { "target": "sheep", "count": 5 },
"rationale": "Collect wool for the team’s beds; several sheep are nearby and beds are the assigned need."
},
"result": { "status": "success" },
"calls": [{
"model": "gpt-6-luna",
"inputTokens": 5622, "outputTokens": 260, "cacheReadTokens": 4716,
"text": "…", "request": "…"
}],
"system": "… (19,163 characters)",
"config": { "episode": "team-tluna11-1", "memory": "plan", "budgetUSD": 0.5, "…": "…" }
}
Each line holds what the agent saw (observation), what it chose and why (decision), what
happened (result), and the model call behind it (calls), including the exact request and the raw
reply. The run's settings (config) and system prompt (system) are on every line too, so any line
on its own says which model and setup produced it. The last line of a run records why it stopped:
goal met, time budget, cost budget and a few others. A file with no such line is a run that
crashed.
Why log this way
Every result can be rebuilt. Success rates, costs and which skills got used are all computed from the logs by scripts. Each published report is stamped with the date and the commit of the code that built it:
stamp="*Built from the run logs on $(date -u +%Y-%m-%d) by the report code at $(git rev-parse --short HEAD) ..."
So when the counting code is fixed, the reports are rebuilt rather than patched by hand.
flowchart LR run[Agent run] -->|one line per step| log[(runs/*.jsonl)] log --> scripts[Report scripts] price[Price table] --> scripts scripts --> reports[Reports<br/>stamped with a commit]
Tokens, not dollars. Each call records token counts, not a cost. Cost is worked out later from one price table. That paid off on November 10, 2026: the table had charged GPT-6's cached input at the full price, so one batch was estimated at $1.45 when $0.53 was actually billed. Fixing the table corrected every report at once (that batch now prices at $0.69), with no run repeated.
The log records facts; judgements live in code. Fields added over time are optional, and the report code falls back when an older log doesn't have them. For example, before October 19 the agent could log "goal met" on the survive-the-night goal even after dying that night. The report doesn't trust that line in older logs: it replays the nights from the observations instead. Had the wrong judgement been the only thing stored, it would be permanent.
What it costs on disk, and the fix
Repeating the prompt and settings on every line is expensive. The 605 logs total 2.5 GB, and 55% of that is the system prompt written again on every step. There are only 126 distinct prompts in all of it, 2.4 MB together.
The obvious fix is to store the prompt once per run. Compression does nearly the same job, but only the right compression:
| Compressor | Size | Ratio | Time |
|---|---|---|---|
| None | 2,513 MB | 1× | |
| gzip (default level) | 593 MB | 4× | 59 s |
| zstd (default level) | 57.4 MB | 44× | 0.6 s |
Both work by spotting text they've seen before and pointing back to it. gzip only looks back 32 KB, and a line here averages about 37 KB, so by the next copy of the prompt the last one is out of reach. zstd looks back megabytes, finds the copy one line up, and stores each repeat in a few bytes. Any JSONL where big blocks repeat from line to line (prompts, schemas, settings) behaves the same way. The logs here are still kept uncompressed, because 2.5 GB isn't a problem on this machine.
What this doesn't fix
- The logs aren't public. They hold every prompt and reply, so they stay out of git, with a copy in a private S3 bucket since November 12, 2026. A batch can be rerun from its commit, but the models will answer differently, so outsiders can check the method, not the exact runs.
- Logs from an earlier machine are gone. Their published reports remain, but they can't be rebuilt again.
- No database. A question across all runs means writing a script, and rebuilding every report means parsing all 2.5 GB.
- A report is only as right as its code. That's why each one names the commit that built it.
