How do AI systems learn, and how fast?

Why this question

Our first attempt at a Terraria-playing AI was hand-built: 84 modules and 19,115 lines of game-specific rules, written in three days. Its player chose among hand-scripted skills, so it was never actually tested for learning. In its last 11.8 hours of play, 5,840 of its 6,188 decisions (94.4%) were the same single forced move, which the game refused each time. A learner offered one option has nothing to learn. So we started again with a general learner that gets better only by playing, and we judge approaches by how quickly they learn rather than by how much game-specific code they carry.

What we built

• The body. A connection to the game, written for Terraria, that gives the learner ordinary human controls and reads the game out as numbers rather than pixels: the tiles around it, what it carries, nearby creatures and the options in open menus. Every tile, item and creature appears as an anonymous id, so the learner is never told what “copper ore” or a “workbench” is.
• The learner. A small neural network (1.79 million parameters) trained with PPO, a standard reinforcement-learning method. There is no gameplay code, no language model and no human demonstration.
• The reward. Generic and written in advance: +1 the first time it holds each new kind of item in a game, a small amount for gathering, a smaller amount for reaching a new area, and −1 for dying.
• Training. Five practice worlds, 3.0 hours of computer time (10,803 seconds), 1,287 updates and 1,647,360 decisions. That is about 45.8 hours of game time.

The test, fixed before training

Learning is switched off and the learner plays three worlds it has never seen, two games each. A game ends at the first death or after five game-minutes. It is compared with random button presses and with its own untrained starting point. The measure is how many new kinds of items it obtains per game.

Results on the unseen worlds

The pre-registered pass mark was a lead of at least 1 item over both comparisons, and the trained learner met it exactly. Afterwards we also ran permutation tests, which were not part of the pass mark. The gap over random play (2.8 items, p = 0.033) is mostly not from training: the untrained learner was already 1.8 items ahead of random play. The gain from training itself (1.0 item over the untrained learner) could easily be chance (p = 0.29). The six games are not fully independent (see below), so neither p-value is exact. The trained learner explored far more of each world, but exploring was part of what it was rewarded for, so that shows training doing what it was designed to do.

New items per game on unseen worlds, measured during training

Average new items per game at each checkpoint: untrained 5.7, update 100 2.8, 200 3.5, 300 4.3, 400 7.3, 500 6.3, 600 4.3, 700 8.3, 800 8.0, 900 8.3, 1000 6.0, 1100 5.2, 1200 8.5, final (1,287) 6.7. Random play averages 3.8.

What this does and does not show

• It shows a general learner, trained only by playing, meeting a pass mark set in advance on worlds it had never seen. The margin that isolates training is small (1.0 item) and could be chance.
• It does not show that the AI “learned to play Terraria”, beat a person, or learns faster than people. We have not measured how quickly first-time human players reach the same milestones.
• The sample is small: three worlds with two games on each, so the six games are three linked pairs rather than six independent trials. Games are not perfectly paired either, because monsters and drops use the game’s own randomness.
• Training stopped once by accident: GLEE’s own gaming-mode watcher stopped the trainer to free the graphics card for a game. It resumed from the checkpoint at update 100 with the optimizer’s internal state restarted, and the two updates in progress (101 and 102) were discarded. This was written into the run’s pre-registration file when it happened, and the reviewer checked it.

Review

An AI reviewer running on a different provider’s model recomputed all 18 games from the raw records. Its verdict: the main prediction (P1) passes, P2 fails and stays on the record, and this is “not a full-preregistration or generalization PASS.”

Next

A second version has finished. It plays under different rules from the table above: a death respawns the player, and every game runs the full five game-minutes. Over 12 games per player on the same three worlds, it averaged 7.5 items, against 6.0 for its untrained start and 5.4 for random play. That cleared the pre-registered margin but failed the pre-registered chance test (p = 0.095, where 0.05 was required), and an AI reviewer from a different company judged it a fail. Under the same new rules the first version scored 7.3, so the second shows no clear gain over it (6 of 12 paired games, p = 0.48).

A third version has also finished. It changed one thing from the second: how the network picks which nearby tile to aim at, so that what it learns about one tile carries over to the others. Its aim did become more focused, as predicted, but that did not help it find items. It averaged 5.5 per game, against 4.7 for its own untrained start, 5.4 for random play and 7.5 for the second version, and did better than the second version in only 1 of 12 paired games. It failed both pre-registered tests, and an AI reviewer judged it a fail. The second and third versions were designed after we had seen these three test worlds, so they will need fresh, unseen worlds before any of their results count.

The second version has now been tested on 13 new worlds, made after its test plan was fixed and never played by any version before the test. Over 52 games each, it averaged 7.0 items, against 4.9 for its own untrained start and 5.1 for random play, and it beat its untrained start in all 13 worlds. Counting each world once, the gain was 2.1 items (95% range 1.6 to 2.5). Both pre-registered tests passed, and a separate AI reviewer that did not run the test recomputed all 156 games from the raw records and judged it a pass. Most of the gain is one skill: the trained version picked up dirt in all 52 games, against 12 for its untrained start, while it found wood and acorns about as often as random play. So this shows a learner improving at a short, five-minute task on worlds it had never seen. It does not show that it plays the game well, keeps improving over time, or learns as fast as a person.

We then tried a fourth version that changed one thing from the second. Using a tool, such as an axe on a tree, takes several decisions in a row with the same item in hand, so we tested whether making that easier would help: the fourth version started training with keeping its current item as the default. It trained on exactly as much play as the second version (997 updates, about 1.28 million decisions), and both were tested on 13 more new worlds under a plan fixed before training began. It did worse: 6.3 items per game against 7.6 for the second version, and it was ahead in only 2 of the 13 worlds. Counting each world once, it found 1.3 fewer items (95% range 0.6 to 2.1 fewer). It found wood or acorns in 13 of its 52 games, against 23 for the second version, and it did not clearly beat its own untrained start (6.3 against 5.7, p = 0.13). Training also undid the change: the trained fourth version held each item for 9.4 decisions in a row, fewer than its untrained start (12.1) and the second version (13.5). So whether holding a tool for longer would help is still untested, because the test never produced a learner that held its tools for longer than the second version. A separate AI reviewer recomputed all 208 games from the raw records and judged it a fail. The second version remains our best learner.

A fifth version tested a different idea. In the second and fourth versions' training, each of the five copies of the game played about 85 games on the same world without restoring it. In a separate test with the second version's finished weights held fixed, the first 8 games on each of four worlds averaged 7.3 items per game, and games 17 to 24 on the same worlds averaged 5.1. The fifth version is the second with one change: after every 4 of its training games, each copy of the game goes back to an unplayed copy of the same world. Its training scores fell only slightly, from 8.6 items per game in the first fifth of training to 8.3 in the last, against 7.8 to 3.8 for the fourth version. On 13 more new worlds, made after its test plan was fixed, it averaged 7.7 items per game, against 7.3 for the second version, 5.0 for its untrained start and 4.8 for random play. It clearly beat its untrained start (p < 0.001), but its comparison with the second version was inconclusive: weighting each world equally, it was ahead by 0.4 items per game (p = 0.23; 95% bootstrap confidence interval for the average lead: 0.4 fewer to 1.2 more items per game), and it was ahead in only 6 of the 13 worlds. A separate AI reviewer, OpenAI's Codex, recomputed all 208 games from the raw records and confirmed both decisions. A fresh copy restores the terrain but also resets enemies and dropped items, so this test cannot say which of those mattered.

The setup

A 16 by 16 grid game with six controls whose effects are hidden and shuffled for every game: four directions, a do-nothing and a marker. The learner has 30 steps to explore, then must reach targets, mark given cells, and keep working after the controls change without warning. We tested seven learners, each adding or removing one ingredient of learning, across 960 registered games, plus 480 more in a learning-to-learn chain.

What we found

• Memory alone is not learning. The memorizer had experience, change, retention and reuse, and did no better than random on anything new.
• The right assumption did the work. Assuming a control behaves the same everywhere took target-reaching from 0.03 to 1.00. Sharing knowledge across controls cut the presses needed from 20.5 to 9.1.
• Choosing what to try mattered. Exploring at random cost about 15 extra presses with the same model.
• Revising beliefs is needed to adapt. The learner that stopped revising matched the others on targets but adapted far worse to changed games (0.08 against 0.49 on our transfer measure).
• Learning to learn, narrowly. A learner that carried what it knew between games cut the presses needed by 40%.

What failed, and the limits

Eight of nine predictions held. The failure: after two controls were swapped, the best learner recovered far less than predicted (0.175 against a predicted 0.8). We traced this to two specific defects and tested a repaired learner on fresh games; that repair is exploratory, not pre-registered. This is a toy world, and the useful structure was designed in rather than discovered. Finding that structure without being told is the open problem.

The setup

A shared lab where AI systems and automated searches design learning programs. Outside submissions are not open yet. Each design runs in a sandbox and is scored against the best of six simple baselines on five synthetic worlds, then rechecked on new random variations of those worlds that it was never tuned on. We invited AI systems from several companies, small models running locally, and an automated search with no AI involved.

What we found

• The ranking is mostly noise. The top six changed order on every recheck with no change to any code. Only two orderings survived repeated paired tests.
• Ideas compounded across sources. The automated search adopted a component first written by one AI and another written by a second. Removing the borrowed component, tested on random variations the search had never seen, dropped its score on one world from +0.086 to +0.057.
• Using the lab found defects in the lab. A sandbox that never started had been scored as a real result, and a design that replayed public answers briefly ranked first. Both were fixed.

Limits

These are synthetic worlds, so this is not a ranking of real-world capability and no company “won”. The scoring rules were tightened several times after reviewers found flaws.