Castlevania

A brain made with Cadence plays the NES game Castlevania from what its eye sees on the screen. The page plays recordings of its games beside its brain, and can run the brain itself in your browser.

What it shows about CadenceA Cadence brain has real memory and a real state, and runs in real time, like the brain of a living being. This one keeps a memory of the moments of the game it is playing, carries its state from one decision to the next, and decides five times a second from what its eye sees. With a look-ahead on the emulator checking its moves near enemies, it reaches stage 3 on all nine starts of a measured set and loses 5 lives where it alone lost 36. The giant bat at stage 3 is not beaten.

What you see

The page has two recordings; the switch at the top picks one.

Brain + look-ahead plays highlights of two games: three stretches, each played as recorded, with a card at each cut. Every frame shows the game, the brain's choice and its brain:

1. The game. The red box is the eye's sharp window ahead of Simon, the blue box the one behind him.
2. The brain's choice. One row per slot of the pad: left or right, up or down, whip, jump. The bars are its probabilities; the red one is its choice. An amber outline marks the move the look-ahead played instead.
3. The settle. Each decision the whole brain settles, sweep by sweep, until nothing changes; the line is the largest change in a sweep.
4. Motor strip. Ten motor neurons in the four slots.
5. Association. 128 neurons, the hub: every other region reads it and writes to it.
6. Module 0. 256 neurons that read the eye's maps and the body sense.
7. Visual cortex: the eye. Four maps of what the eye sees: the whole screen, what moved since the last decision, and the windows ahead of and behind Simon. Beside them the body-sense strip: position, health, lives, facing, stage, block and floor.
8. Prefrontal: working memory. 128 neurons that carry a trace of the last moments from one decision to the next.
9. The choice. Each slot's choice, read from its motor neurons (the role of the striatum). Amber ring: the look-ahead overruled the brain here; teal: route recovery.
10. Hippocampus: recall memory. 128 slots, each holding a moment of this game. Gold slots are the ones this decision reads back.
11. What it recalls. The game frame of the moment the brightest slot holds, and the scene parts of what the two read heads recall and expect next.
12. What was played. The brain's move, or the look-ahead's (amber), or route recovery's (teal), and the running shares of the three.

Brain alone plays one whole game of the brain by itself: the game video on the left with the stage, health and lives under it, the pad the brain presses, and its brain on the right, drawn from the recorded state of the decision the video shows. Every part with a small ? explains itself when the pointer rests on it.

Why it matters

Real memory. The brain keeps a recall memory of the game it is playing: every decision it writes the moment it is in, and reads back the moments that match it. The memory starts empty when a game starts and fills as the game goes; the hippocampus line counts the moments held.

A real state. Each decision the brain settles starting from the state the last decision left, and its prefrontal region carries a working trace of the last moments. It is a continuing brain: what it did a moment ago shapes what it does next.

Real time. It decides five times a second, at the game's own speed, from 827 readings of its eye. In your browser, Live runs the same brain on the recorded eye inputs and shows how long each decision takes; the recorded check took about a tenth of a second a decision, where the game needs one every fifth of a second.

What the look-ahead adds. The brain alone loses all its lives on every start, most of them in stage 2. A look-ahead on the emulator, which tries the brain's move and a few safer ones a short way ahead, overrules about 4 decisions in 100; together they reach stage 3 on every start and lose few lives. That result belongs to the brain and the look-ahead together; 93 of every 100 moves played are the brain's own.

What you can do

• Play the highlights with the video's controls: play, pause, seek, full screen. Rest the pointer on a part of the frame to see what it is.
• Switch to Brain alone to watch the brain's own whole game. Play, pause, seek or change the speed: the brain beside the game follows the decision the video shows, settling sweep by sweep.
• Press Live (in Brain alone) to run the brain itself in your browser. The first time, Python, NumPy, the released Cadence library and the 37 MB brain load into the tab. From the game's start the brain settles on the recorded eye inputs, decision by decision, and the line beside the button counts how often it presses the recorded buttons and how long a decision takes. The brain view then shows the state your browser computed.
• Run the live game on your own machine: the game is not included, so the emulator runs locally with your own copy of the cartridge (see the README in the repository).

How it works

Senses. Every 12 frames the eye reads the screen: the whole picture pooled to 12 by 16 cells, what moved since the last decision, and a sharp window 64 by 48 pixels ahead of Simon and another behind him. Simon's position, facing and body state, and the stage, block and floor, come from the console's memory. 827 numbers in all.

Thinking. The whole brain, 4,697 neurons and 724,736 synapses, settles on those numbers, starting from the state the last decision left: module 0, the association hub, the prefrontal working trace, the recall memory's two read heads and the motor strip agree.

Acting. The strongest motor neuron of each slot is pressed, and held for 12 frames: five decisions a second.

Memory. The synapses hold what it learned, and are frozen here. The working trace and the recall memory are written at every decision, read by the next ones, and emptied when a new game starts.

Learning. Offline, the brain imitated a scripted teacher that reads the console's memory and knows each stage's route: about 425,000 decisions of practice, the last 100,160 of them driven by the brain itself while the teacher labelled every state it reached, so it learned from its own mistakes. The brain never reads the console's memory beyond its eye's position readings. Nothing is learned while it plays here.

The look-ahead. Near enemies, and at two known trouble spots, the look-ahead saves the emulator's state, plays the brain's proposed move and a few safer moves 12 decisions ahead, and restores the state. It plays the brain's move unless another loses clearly less health. Where the brain makes no progress along the route for 40 decisions, the route's move stands in for 30 decisions: route recovery. The look-ahead is ordinary code over the emulator, not part of the brain.

The numbers

Selection starts (0, 3, 7, 10, 14, 21, 28, 35, 42 idle decisions before Simon starts) were used to pick the brain among its checkpoints; fresh starts (1, 5, 11, 17, 25, 31, 38, 45, 49) were read once, after the pick. Where the files are, in the castlevania/ folder of the repository: both tables' comparisons, evidence/lookahead_receipt.json; the brain and its readings, evidence/receipt.json; the uncut games, evidence/castlevania-run.json and evidence/castlevania-brain-alone-run.json; live mode, player/evidence/player_check.json (cloud CPU) and evidence/promise_test-20261113.json (Apple M4); this page's promise, played on the page, evidence/promise_test-20261113.json.

Limits

• The look-ahead is not the brain. Every overruled move and every route-recovery move comes from a search over the emulator's saved states and the teacher's route. The brain's own record is Brain alone: it loses all four lives on every start, and reaches stage 3 on 3 and 2 of 9.
• The giant bat is not beaten, by the brain alone or with the look-ahead. With the look-ahead, five of the nine selection starts end in a standoff at the bat; the brain alone took at most 36 of its 64 health.
• Frozen. The brain learned by imitating a teacher, offline; it does not learn while it plays, and the game's score or reward played no part.
• Part of the eye reads the console's memory. Simon's position, facing and body state, and the stage, block and floor, come from memory; everything inside the eye's windows is pixels.
• Recordings. The game is not included, so the site shows recorded games. Live mode runs the real brain on the recorded eye inputs; it does not play the game.
• One cartridge. Raised and checked on one printing of the cartridge, on one emulator core.