Reference · Updated 2552.07.26.14.59
Autoresearch — Opportunities for SystemLink & MenuKit
Findings doc, 2026-07-18. What "autoresearch" is on GitHub and concrete ways we could use it for the game, the dev workflow, and the MenuKit Fab product. Written for future-Claude
Findings doc, 2026-07-18. What "autoresearch" is on GitHub and concrete ways we could use it for the
game, the dev workflow, and the MenuKit Fab product. Written for future-Claude and Beepers.
#TL;DR
Autoresearch = an autonomous agent loop: make one change → run a mechanical metric → keep it if the number improved, git-revert it if it didn't → repeat, unattended, for dozens/hundreds of iterations. Karpathy's original ran it on ML training; the idea generalizes to anything with a fast, objective, scriptable metric.
- It is not a general "make the game better" button. It only works where success is a **number a shell
command can print in seconds/minutes**. Most game feel fails that test; a surprising amount of our tech, tooling, and the MenuKit generator passes it.
- The single highest-value fit for us is the MenuKit AI-native manifest→generator (
MenuKitExtractionPlan.md
§11): a generator's output is mechanically checkable (does it compile? does it match a golden spec? how many manual steps remain?), which is exactly what an autoresearch loop needs.
- Actionable path: install the Claude Autoresearch skill (
uditgoenka/autoresearch) since we already live
in Claude Code, and start with one narrow, well-instrumented pilot.
#What's actually on GitHub (three different things)
| Project | What it is | Relevance to us |
|---|---|---|
karpathy/autoresearch | The origin. Agent edits one train.py, trains for exactly 5 min, reads one metric (val_bpb), keeps/discards. ~12 experiments/hr, ~100 overnight. | Conceptual template. The discipline (fixed budget, single metric, single file, reviewable diffs) is the takeaway. |
uditgoenka/autoresearch (also leo-lilinxiao/codex-autoresearch, davebcn87/pi-autoresearch) | The concept as a reusable Claude Code / Codex / pi skill. You give it Goal + Metric + Verify command + Scope; it loops modify→verify→keep/discard, logs TSV, auto-reverts failures via git. Has guardrails (blocks rm -rf/force-push/.env, LOC cap, .ckignore). | The directly usable one for us. Drop-in for Claude Code. |
AutoResearch/autora (Brown University) | Unrelated. A scientific "Automated Research Assistant" for closed-loop empirical science (theorist → experimentalist → runner). Not Karpathy-derived. | Not relevant to game dev. Noted so we don't confuse the names. |
Proven non-ML wins from the community lists (evidence the loop generalizes): Shopify Liquid engine — 53% faster parse+render / 61% fewer allocations over 93 auto-commits; a CUDA kernel 18→187 TFLOPS; pi-autoresearch optimizing "test speed, bundle size, build times, Lighthouse scores." Pattern: perf/size/throughput metrics.
#The gate: does a task qualify?
An autoresearch loop is worth setting up only if all of these are true:
- Objective metric — one number, higher-or-lower-is-better, no human judgement.
- Scriptable verify — a shell command prints that number (ideally in < a few minutes).
- Bounded scope — a small, safe set of files/params it may edit.
- Cheap revert — git makes a bad change free to throw away.
- Fast-ish iteration — enough cycles to compound (minutes, not hours).
If a task needs "does this feel right / look right," it fails gate 1 — keep humans in that loop. The trick for a game project is finding the measurable sub-problems hiding inside subjective goals.
#Ways we could use it
#A. MenuKit / the menu product ⭐ best fit
The Fab product's AI-native wedge is a text manifest → generated WBP widget tree (plan §11). Generators are the ideal autoresearch target because their output is mechanically judged.
| Loop | Metric | Verify (sketch) |
|---|---|---|
| Harden the manifest→WBP generator | # of golden manifests that generate + compile cleanly (pass/total) | Python-via-bridge: generate each, compile_blueprint, count successes |
| Drive "manual steps" to zero | # of human touch-ups needed vs. a golden target tree | Tree-diff generated vs. golden; count mismatched nodes |
| Beat the root/reparent footguns | # of widget archetypes buildable end-to-end via script (we just hit the protected-RootWidget wall) | Attempt build of each archetype; count that render a valid root |
| Widget runtime cost | Tick/paint time or invalidation count for a sample menu | UE stat slate / automation profiler dumped to a number |
| Manifest schema coverage | % of a feature checklist the manifest can express | Parse checklist → assert generator supports each |
Why this matters commercially: "menus your users' AI can operate" is the differentiator no Fab menu competitor has. Using autoresearch to build the generator that builds the menus is on-brand and self-demonstrating — the loop's TSV log literally becomes marketing proof ("generator hardened over N automated iterations").
#B. Engine / gameplay tech with real numbers
Feel is subjective; the plumbing under it often isn't.
| Loop | Metric | Notes |
|---|---|---|
| Packaged build size | MB of the cooked build | Great fit; pure number, slow-ish but clear |
| Shader permutation count / cook time | count or seconds | Substrate/VSM/SM6 configs generate a lot |
| Frame time on a fixed benchmark flythrough | ms from stat unit / CSV profiler | Needs a deterministic automated PIE run (see caveats) |
| Load / level-open time | seconds to TestMap ready | Scriptable via automation |
Memory (LLM/stat memory) for a scene | MB | Objective |
| Netcode: bandwidth/RPC count for a fixed scripted match | bytes/RPCs | We have replicated weapon/grenade/health systems to trim |
| Automated functional-test pass rate | passing/total | Only if we build a UE automation-test suite first (we mostly don't have one yet) |
Poor fit (don't): weapon sway/lag/ADS "feel," grenade arc satisfaction, animation blends, reticle look. These are the feedback_commonui_verify_one_at_a_time / "PIE-test by eye" tasks — no mechanical metric.
#C. Dev workflow, build, docs
| Loop | Metric |
|---|---|
Compile time of SystemLinkCore | seconds (UBT timing) |
| Warning count from a build | count → drive to zero |
Docs/ link integrity / freshness | # of broken [[links]] or stale refs |
| Footgun-doc coverage | # of BugTracker RCAs without a Footguns.md line |
#D. Fab go-to-market (weaker fit — be honest)
Listing conversion / sales are slow, noisy, real-world metrics — they violate gate 5 (fast iteration). Autoresearch is bad at those. Where it can help: mechanical proxies —
- Docs readability score (Flesch/word-count targets) over the product README/manual.
- Asset-listing completeness against a Fab-submission checklist (count required fields present).
- Example-menu generation breadth (how many demo menus the generator can emit for the store gallery).
#Candidate first pilots (ranked)
- Generator hardening (A) — most on-strategy, self-contained, and the output is trivially scriptable via
the Unreal bridge. Metric: golden-manifests-that-compile. Start here once the generator spike is real.
- Packaged build size (B) — dead-simple metric, no new infra, immediately useful, safe scope (config +
cook settings). Good "learn the tool" pilot while the generator matures.
SystemLinkCorecompile time / warning count (C) — zero-risk, pure dev-quality win.
#UE-specific caveats & guardrails
- Getting a metric out of Unreal is the hard part. Most of our verification today is human-in-PIE. Autoresearch
needs a headless/automation path: -ExecCmds, Automation tests, commandlets, or the systemlink-unreal bridge dumping a number. Budget setup time for the harness before expecting overnight runs.
- Determinism. Frame-time / gameplay metrics need a fixed, scripted scenario or the number is noise. Fixed-seed,
fixed-camera, fixed-input.
- Editor automation footguns we already know apply if the loop touches assets: protected
RootWidget(can't
script-set a widget root), mid-session reparent → World Leak crash, Control-Rig member-var crash. Keep the loop's scope on code/config/data, not risky asset surgery, until those are solved. See Docs/Footguns.md.
- Cost & safety. Long unattended agent loops burn tokens and edit the repo. Use the skill's guardrails (git
auto-revert, LOC cap, blocked commands, Guard: command that must stay green), run on a branch, and cap Iterations: until trusted.
- One change at a time is already our house rule (
feedback_commonui_verify_one_at_a_time) — autoresearch
enforces exactly that, which is a nice cultural fit.
#Setup (when we pilot)
# Claude Code skill (recommended path)
npx skills add uditgoenka/autoresearch
# then restart Claude Code; use /autoresearch:plan to validate Goal/Metric/Verify before looping
Define per run: Goal (plain language), Metric (the number), Verify (shell cmd that prints it), Scope (editable files), optional Guard (must-stay-green cmd), Iterations (bounded first).
#Bottom line
Autoresearch is a force-multiplier for the measurable, mechanical corners of SystemLink — build/perf/size, dev-quality, and above all the MenuKit generator, where "an agent iterates the tool that generates menus" is both a real productivity win and a marketable story. It is not for tuning game feel. Recommended: start with the build-size or compile-time pilot to learn the tool cheaply, then point it at the manifest→generator once that spike is production-shaped.