In progress
What do AI agents do when they're given a goal and a set of tools, then left alone?
TypeScript · Node.js · Claude · OpenAI · Ollama · Docker · Mineflayer
This is a testbed for running autonomous agents hands-off. Each agent gets an outcome to reach and a fixed set of tools, and nothing else: no step-by-step instructions, and no human stepping in when it gets stuck. Several agents can share one world, each driven by a different model: Claude, GPT, or a free model running locally.
The project measures whether they reach the goal, how they plan, how they recover when something fails, whether they work together, and what it all costs. Every decision is logged with the exact prompt and answer, so any run can be replayed and costed afterwards.
Minecraft is the environment because it's convenient. It's free to run, can be watched live, needs real planning, and a fixed seed makes runs repeatable. The game isn't the subject. The agents, tools, logs and reports would be the same if the environment were a codebase, a ticket queue or a data pipeline.
The world moves on in real time, and a model takes seconds to answer, so the model is never in the control loop. That keeps an agent fast and cheap: it can work through a whole in-game day on a few dozen decisions.
01
The agent's body sums up its situation in a short briefing: what it holds, what's nearby, its health, the time, and who else is around.
02
The model picks one goal-level tool, such as gather_wood(10) or build_shelter(). It never steers step by step.
03
Deterministic code carries the goal out. It finds its own targets, retries, gives up cleanly, and reports back in plain words.
04
The model is asked again only when the goal succeeds, fails, or something important happens.
These results come from logged runs, mostly with a small local model (Qwen3 4B), plus team runs with Claude Haiku, Sonnet and GPT.
3/9 → 8/9
On the same small local model and the same goal, fixing the skill code took success from 3 of 9 runs to 8 of 9. Better targeting, clearer failure messages and hazard avoidance did more than any change to the model or the prompt.
134×
One agent holding 12 logs asked for “one more log” 134 times, though it was told each time that it already had them. In team runs the local model changed course after about one failure in ten. The Claude models changed course after about half or more.
13/24
Carrying its own plan between decisions made the small model worse (13 of 24 runs) than full memory (21 of 24) or no memory at all (12 of 12). It followed stale plans into loops, whatever the game told it.
“Done”
Small models regularly said they were finished when the goal wasn't met. The harness now refuses a premature “done” and says what is still missing. Each false claim is counted as a result.
0
Agents drowned while waiting for the model to decide what to do. Once the code kept them out of the water and swam them out before asking the model anything, no agent drowned in the next two team runs.
$0.28
Four agents on four different models met a shared goal in under five minutes of game time for $0.28. One made every tool and handed them out, and another built the shelter. Getting there took explicit rules: agents had to be told their own names, and repeated messages were stopped from interrupting each other.
Businesses are starting to hand work to AI agents: support tickets, invoice reconciliation, data pipelines, overnight alert triage. In real systems an agent's mistakes are buried in logs and surface weeks later. Here every decision is visible, and the goal is either met or it isn't. The environment is a game, but the questions are the ones a company faces.
An agent is only as reliable as the systems it works through. A clear error such as “invoice not found: check the supplier ID” is worth more than a cleverer model guessing.
An agent that improvises every click is slow, expensive and fragile. Have the AI choose what to do, and let proven code do it.
Never take an agent's word that a job is finished. Check the result, for example that the refund was actually issued, before anyone relies on it.
Set limits on time and spend per task, and watch for an agent doing the same thing over and over.
Price per decision ranged from nothing (a local model) to about $0.02 (Claude Sonnet). A cheap model that finishes can beat an expensive one that doesn't.
Anything that must never go wrong, such as spending limits, data deletion or access to customer records, should be enforced by code, not left to the agent to remember.
When something goes wrong, you need to see what the agent knew and why it acted, not just the end result.
In short, autonomous agents can do useful work unattended. How reliable they are depends far more on the systems around them than on which model sits in the middle: good tools, independent checks, firm limits and complete records.
These are early results. Most conditions have three runs, most runs used one local model, and the paid models have been tried but not yet compared in repeated batches. They point to where to look; they aren't proof yet. The repo documents what has and hasn't been tested.