Visualizations - How to turn a game into an RL environment: the technical intuition

#1 Chart

Geoguesser Space

Play a round: look around, walk, drop a pin, commit. Runs on a shared Space, so a cold start takes a minute. Open it in a new tab for more room.

#2 Chart

One episode

Gather evidence, test a candidate on the map, then commit. Every action costs a little reward, and the pin never tells you whether you are close.

View in article →

1 Look around as many times as you like, each action costs a little reward

look Turn the camera. Signage, road markings, which side traffic drives on.

zoom Narrow the view to read something distant. 90° rarely resolves text.

move Walk along the road to reach a sign you cannot read from here.

2 Test a guess on the map as often as you like, and back to looking whenever

place_pin Drop a coordinate on the world map. It reports what is there: the country, the nearest city, the distance from your last pin.

It never tells you whether you are close to the answer. If it did, the best strategy would be binary search, and the game would be about bisection instead of geography.

3 Commit terminal, one guess per episode

submit_guess Your latitude and longitude. Scored on kilometres of error, and running out of turns without guessing scores zero.

how the guess is scored

0 km, reward 1.0 1,500 km, reward 0.37 far, reward 0

#3 Table

look around	walk	bulk cache	licence	verdict
OSV-5M	no	no	yes	CC BY-SA 4.0
Google Street View	yes	yes	no	ToS-restricted
Mapillary	yes	yes	yes	CC BY-SA 4.0

#4 Chart

Banner

View in article →

#5 Chart

Fifteen models, before any training

Reward, accuracy at two thresholds, turn count, and the two failure modes that the reward column hides.

View in article →

#6 Table

tool	what it does
`look`	Turn the camera to an absolute heading, 0 = true north. This is how it reads signage, road markings, vegetation, architecture.
`zoom`	Narrow the field of view without turning. 30° reads a distant sign; 90° almost never resolves text.
`move`	Walk forward or back along the road. Reports how far it actually travelled, which is not what you asked for.
`place_pin`	Drop a candidate on the world map. Returns what is at that coordinate: country, subregion, nearest city with distance and bearing, and how far it is from your earlier pins.
`submit_guess`	Commit. Terminal, scored on distance.

#7 Chart

One episode, tool call by tool call

A real recorded episode replayed from its trace. Every frame is the reprojection the model was actually served, verified against the hash in the record. The left panel is the observation, the right is the call that produced it and the reply the policy wrote.

View in article →

what the environment returned

one episode, twelve actions allowed

the policy's own words on this turn

#8 Chart

One class in, two API surfaces out

Orange is what create_app() derives from the environment class. The trainer drives the simulation over HTTP, the policy only ever sees tools over MCP, and in production mode the simulation routes are never registered at all.

View in article →

what we wrote ~5k lines, all of it about GeoGuessr

GeoGuesserEnvironment reset(), step(), state, eleven MCP tools

PanoramaBackend frames, reprojection, movement graph

reward and renderers distance curve, action cost, view, minimap

openenv.yaml entrypoint, port, Space variables

create_app(env, Action, Observation)

what OpenEnv generates from it none of this is in our repository

simulation the trainer drives it

POST /resetsplit, index, seed
POST /steptyped action in, observation out
GET /stateepisode_id, step_count

Registered only in simulation mode.

agent the policy sees only this

POST /mcpJSON-RPC, tools only
WS /mcpone session per connection

the two calls everything the policy can do

tools/listthe eleven schemas, straight from FastMCP
tools/callone action, one observation back

Reserved names: a tool may not be called reset, step, state or close.

discovery from the Pydantic models

GET /schemaaction, observation, state
GET /metadata
GET /health

task API under /geoguesser_env

/splits /num_tasks /task /task_range

Declared on the class, or the routes return 501.

runtime lifecycle and concurrency

a fresh environment per session, up to max_concurrent_envs
an idle-session reaper
CAPACITY_REACHED rather than a silent queue
sync step() in a thread pool, so one slow rollout blocks nothing
a container and a Space manifest, one image for both

HTTP for the trainer, MCP for the policy

who reaches what

TRL GRPOTrainer eight sessions, one per rollout in the group

/reset /step /state task API

the policy inside the rollout, over MCP

eleven tools /reset does not exist here

The boundary is the server's, not the prompt's. In production mode the simulation routes are never registered, so a policy cannot reroll a task it does not like.

#9 Chart

Where each piece lives, and what talks to what

Storage in a Bucket and a Dataset, the running environment on a Space with the bucket mounted read-only, GPUs rented per run through Jobs, and one HTTP URL that the trainer, the eval harness and a browser all reach.

View in article →

storage, one repo type per kind of thing

Storage Bucket 22 GB

geoguesser-panos 86,395 panoramas. Mutable object storage, so a harvest can top it up without a commit.

Dataset 8 MB

geoguesser-tasks The task indexes alone. Versioned, so a change to a frozen benchmark shows up in history.

Models 2 × 40 MB

…-4b-grpo, …-grpo-v3 LoRA adapters, not merged weights. Small enough that anyone can pull one and reproduce a score.

compute

Space CPU

geoguesser-env The environment itself, always on. Serves the browser game, the MCP tools and the HTTP simulation routes from one image.

8 concurrent sessions ALLOW_FETCH=0

Jobs 4 × A100

training and serving Rented per run, not kept. The same CLI serves a checkpoint behind vLLM for an eval sweep and tears it down after.

10 h, about $100

Space CPU

geoguesser-trackio All four training runs in one dashboard, so the runs can be read against each other rather than one at a time.

who connects, and from where

the trainer Eight sessions, one per rollout in a GRPO group.

the eval harness The same URL, the same splits. Nothing to install, no imagery, no GPU.

anyone with a browser Plays a round, or points their own model at the tools.

The property worth designing for is the middle row reading the top row the same way everywhere. Locally the indexes and imagery sit in the checkout; on the Space they arrive through the mount. Splits, step budget, street labels and offline enforcement are identical, and checked: the same task returns the same image checksum, reward and distance from either.

#10 Table

what	where it lives	why there
22 GB of panoramas	Storage Bucket	Space disk is ephemeral and capped well below 22 GB. A bucket is mutable object storage, which suits something a harvest tops up over time, and it mounts read-only into the Space at `/data`. The cache can keep growing without a single commit.
the task indexes	dataset repo	8 MB of metadata, and the part that has to be frozen. A bucket is not versioned, so a benchmark living in one can change under you silently; in a dataset repo a change to a split appears in history where a reviewer can see it.
the environment	Space	One image serves the browser game, the MCP tools and the HTTP simulation routes. Locally the indexes and imagery come from the checkout, on the Space they arrive through the mount, and nothing but environment variables differs.
the adapters	model repos	Both are LoRA adapters instead of merged weights, so they are tens of megabytes. Anyone can pull one, point it at the same Space, and check our numbers.
the GPUs	Jobs	Rented per run instead of keeping them: ten hours on four A100s, about $100. The same CLI stands up a vLLM endpoint for an eval sweep and tears it down afterwards.
the curves	Trackio Space	Every run in one dashboard, which is what makes them readable against each other instead of one at a time.

#11 Chart

What a guess is worth, and why the two curves disagree

Pick an episode or move the sliders. The dashed line is the game's curve, the solid one is what the trainer optimises, and the shaded band is where the game curve has already flattened to zero.

View in article →

what the trainer optimises the game's own curve where the game curve is flat at zero x: kilometres off, log scale

how far off the guess is

what it did to get there

training reward

game reward

#12 Table

bug	what it looked like
two turn budgets disagreeing	0 of 6 episodes ever reached an answer
`done` read from the observation instead of the step result	every episode unterminated, no score captured
a tool omitting a parameter the environment supports	a `TypeError`, a wasted turn, a capability hidden from the model

#13 Chart

Trainer to environment

Where the group is formed, where the reward is computed, and which side holds the websocket.

View in article →

the trainer, on the GPU node

the dataset one row per task, just split and index

one row

the group NUM_GENERATIONS=8 rollouts of the same row

8 concurrent sessions

vLLM, colocated generation shares the same four cards as training

8 rewards come back

the update the advantage is the spread inside the group, so a group whose rollouts all score alike teaches nothing

HTTP one session per rollout, open for the whole episode

the environment, on a Space

reset(**row) the whole dataset row lands here, which is what puts all eight rollouts on one task

the first view

eleven tools, over MCP public methods become tools, schemas come from the docstrings, and a tool result may carry an image

up to 12 turns

get_reward() distance and action cost, turned into a reward by the training script rather than read off the environment

Where the boundary sits. The trainer owns the loop, the group and the update. The environment owns the task, the tools and the truth. Neither one imports the other, which is why the same URL serves the eval harness unchanged.

#14 Chart

Run 1, from the untrained baseline

Per-step training reward with a 50-step moving average, and the held-out eval score of each checkpoint on the same axis. Both start from the untrained baseline of 0.4825.

View in article →

training reward, smoothed one task per step, raw held-out eval, mean of 4 every other model, same 200 tasks

optimizer step0

training reward·

eval reward·

vs baseline·

turns per episode·

entropy·

models passed·

#15 Table

value
model
hardware
steps
rollouts
episodes
turns
image
optimiser
wall clock
cost

#16 Chart

Behaviour across checkpoints

Turns, output tokens and zero-scoring episodes all fall together while accuracy improves. Score plateaus by step 200.

View in article →

#17 Table

checkpoint	eval score	turns	looks	pins	output tokens	scored zero	median error
base	0.4825	6.7	2.2	2.1	1062	29.5%	1226 km
step 50	0.4968	4.8	1.5	1.7	602	25.4%	1308 km
step 100	0.5600	2.6	0.6	0.9	186	18.5%	964 km
step 150	0.6113	1.3	0.0	0.2	50	5.2%	714 km
step 200	0.6393	1.1	0.0	0.1	46	1.9%	675 km
step 500	0.6350	1.1	0.0	0.1	86	2.1%	709 km
step 1000	0.6445	1.1	0.0	0.1	66	0.5%	662 km

#18 Chart

Turns against score, every model on the board

Across the nine models we did not train, more turns go with a lower score, r = −0.75. Marker size is the share of episodes that scored zero.

View in article →

Every point is one model on the same 200 held-out tasks, scored mean-of-4. Marker size is the share of its episodes that scored exactly zero, which on the game curve means a guess beyond about 3,300 km. The two arms we trained are marked, and the line is fitted to the nine models we did not.

#19 Chart

The finished board

Mean-of-4 on 200 held-out tasks, 800 episodes per arm. The trained 4B is second of eleven.

View in article →

#20 Table

model	mean-of-4	median error
claude-sonnet-5	0.6952	324 km
run 1, step 1000 (Qwen3.5-4B + LoRA)	0.6445	662 km
gpt-5.4-mini	0.5732	753 km
claude-haiku-4.5	0.5374	939 km
Qwen3.5-122B-A10B	0.5338	767 km
Qwen3.5-4B, untrained	0.4825	1226 km
Qwen3.5-9B	0.4776	1203 km
Qwen3.5-35B-A3B	0.4483	1485 km
Qwen3.5-27B	0.4478	1289 km
Qwen3.5-397B-A17B	0.4466	1420 km
gpt-5.4-nano	0.3748	2541 km

#21 Chart

Three runs, one knob at a time

Paired gain against each run's own base, with 95% confidence intervals. Run 2 damped the instability and lost most of the gain.

View in article →

Paired per task against each run's own base arm, measured in the same sweep, so the four numbers are comparable to each other and not to a difference of two means. Whiskers are 95% confidence intervals; the bar is the estimate.

#22 Table

run 1	run 2 · 4B	run 2 · 2B	run 3
`scale_rewards`	`group`	`none`	`none`
`beta`	0	0.02	0.02
tasks per step	1	2	2
action cost scale	1.0	0.2	0.2
paired gain	+0.1620	+0.0326	+0.0663
95% CI	±0.0137	±0.0090	±0.0091

#23 Chart

Why one task per step matters

At one task per step the group standard deviation is the within-task spread, and that collapses. At two it pools between-task variance, which never does.

View in article →

left: spread inside the group · right: what GRPO multiplies the advantage by · shaded: 20x or more

GRPO divides each advantage by its own group's standard deviation, at grpo_trainer.py:2809. With one task per optimizer step, that spread is just how much eight rollouts of the same task disagree, and it collapses as the policy makes up its mind. With two tasks per step it also contains the difference between the tasks, which never collapses. Same line of code, two entirely different training dynamics.

#24 Table

bug	what it looked like	what it was
Wrong base served	2B checkpoints scoring 0.469–0.479	vLLM accepted 2B LoRAs on a 4B base and served anyway. All four scores invalid
Dead tunnel	ckpt75 regressing to 0.4475	13% of requests got an HTML 404 recorded as an empty reply, so the episode burned its turns and scored 0
Two reward scales	run 2 deltas incomparable to run 1's	the stored reward is the environment's, while reports recompute through the pinned curve. The same guess scored 0.0107 and 0.135
No base arm	deltas read across sweeps	the same frozen base scored 0.465–0.500 between sweeps, wider than most differences being claimed
Comparing across `k`	run 1 tying Sonnet 5	baselines read at pass@1 gave Sonnet 0.6798, at pass@4 0.6952. Mean-of-k is unbiased in `k`, so that gap was single-pass noise rather than a `k` artefact. Either way, compare arms measured with the same number of passes
Concurrent sweeps	`k = 1.8` with `PASSES=1`	two sweeps writing one directory. Arithmetically impossible, and the only tell

#25 Chart

The dashboard, in full

All four runs and every metric they logged, live. This is the database behind every training figure in this article.

View in article →

The whole dashboard, live: all four runs, every metric each of them logged, and the sidebar to filter and group them. Every training figure in this article is built from this same database. One thing to know if you go exploring: the x axis defaults to the logging step, which ticks twice per optimizer step, so switch it to train/global_step to line it up with the checkpoint numbers. Open it in a new tab for more room.