If you run Godot headless in CI, or let an AI agent “verify” a change, it usually runs the engine, sees exit code 0, and calls it a pass. In Godot 4 that proves very little. I measured these on 4.7.2, Windows and Linux. Every one exits 0:
--import on a project with a hard syntax error: no output at all
a scene whose own script doesn’t parse: SCRIPT ERROR, then the scene runs as a bare node
a test harness that forgot to await a slow check: prints PASS, and the failing check never prints
running with --user-data-dir (Godot 4 has no such flag): empty output, the scene never ran
a fresh clone run before --import: a false Could not find type "<ClassName>"
a healthy scene that prints nothing: byte-identical log to the “never ran” case
And a false alarm the other way: --check-only --script fails every correct script that references an autoload.
Multiplayer is worse:
An RPC to a function without @rpc logs its error only in the sender’s log.
An RPC to a node path the other side doesn’t have logs only in the receiver’s.
Both processes exit 0 either way.
A client with no server gets connection_failed after ~31 s.
A client whose server died hears about it after ~10 s.
Every one of these is reproduced by a script (tests/measure_traps.sh) on a small demo project. If a claim doesn’t reproduce, it’s not in the repo.
I turned this into godot-claude-kit, an MIT-licensed plugin for Claude Code. It gives the agent:
A runner. It isolates the engine from your editor settings, imports when needed, and judges a run by banner + error tokens + exit code.
A parse-gate scene that actually checks every script compiles, with autoloads live.
A harness template that counts its checks, so an un-awaited one can’t pass silently.
A multiplayer harness. A real headless server and client as separate processes, every log judged, plus a static check that every RPC you send has an @rpc to land on.
One ci.sh plus a generated GitHub Actions workflow.
The GIF in the README is a real session: I broke the server-side hit validation on purpose, asked “make sure multiplayer still works”. Claude ran a real server and client, got FAIL from both sides, and pointed at the line.
The scripts are plain bash. The gate, the harnesses and ci.sh work without any AI.
I’d love to hear which headless/CI traps you have hit, and whether these reproduce on other Godot versions. So far I’ve only measured 4.7.2.
These reproduce on 4.4.1, 4.5.2 and 4.6.3 as well, and measuring across versions adds one more trap to your list: the same gate passes on the version you test and hides a total script failure on the version you don’t.
Three we hit while building a save-file inspector/editor (a tool whose whole point is that an agent can read and change one value without reserializing the file). Measured on 4.4.1 / 4.5.2 / 4.6.3 / 4.7.2, Windows.
A plugin or tool script that fails to compile still leaves the editor gate at exit 0.godot --headless --editor --quit on a project whose add-on calls ResourceUID.uid_to_path() (static only from 4.5) exits 0 on 4.7.2; on 4.4.1 it also exits 0 while every script of the add-on failed to load. The only signal is SCRIPT ERROR in the log. Our gate therefore reads the log, not the exit code: fail on SCRIPT ERROR / ERROR / Parse Error, and require the add-on’s own init lines to be present — a log with nothing in it is exactly your “never ran” case. An exit-code-only gate would have shipped a build that cannot be installed on 4.4.
Your own command-line flags are an exit-0 trap too. A --script tool that reads arguments in order and ignores what it doesn’t recognise: typo --jso instead of --json → exit 0 plus the human-readable report, so a CI step piping stdout into a JSON parser receives nothing, and a step that only looks at the exit code sees a pass. The fix belongs in the tool: read all arguments first, reject unknown ones with a non-zero code, and put the machine-readable result on the last line of stdout so the human text in front can’t be mistaken for data. Then validate the shape (required fields, an explicit ok/error field) — “json parsed” is not a pass.
Fresh clone: silent, and in the dangerous direction. With no .godot/ folder (a CI checkout before the first import), ResourceLoader.get_resource_uid() returns empty even for scripts that ship a .uid file. A reference checker then reports no references for a file that is actually referenced from preload("uid://…") — clean exit 0, wrong answer, in the direction that gets files deleted. Reading the .uid sidecar, or the uid= attribute at the top of a scene or resource file, as a fallback makes the fresh-clone output identical to the imported one (byte-identical across the four versions).
One addition to your false-alarm list: --check-only --script also fails on a script that uses a class_name which hasn’t been imported yet, so on a fresh clone the false alarm and the false pass above cancel out and you learn nothing either way. Importing once before any check is the only fix we found.
Is the version matrix part of your runner? Running the same gate on the oldest version you claim to support is cheap and it is the only way we found case 1.
Thanks, this was very useful, and it made me do the measuring. No, there was no version matrix: everything was measured on 4.7.2 only, and the README said “4.x”. I’ve now run the kit’s whole trap suite on 4.4.1, 4.5.2, 4.6.3 and 4.7.2, on Windows and Linux. The exit-0 traps reproduce on all four. The differences:
--import with a syntax error is silent on 4.5+, but 4.4.1 does print the SCRIPT ERROR (still exit 0).
On 4.4.1, --debug prints GDScript warnings as ERROR: '<msg>' lines with no file or line. The missing_await warning doesn’t fire at all on 4.4.1 or 4.5.2, even when enabled.
GUT 9.7.1 doesn’t compile on 4.4–4.6. On 4.5 and 4.6 the tests still run, and GUT says “All tests passed!” with exit 0 while two of its own scripts failed to load. gdUnit4 6.2.2 needs 4.5+, and on 4.4.1 both frameworks’ CLIs hang until something kills them.
Your case 1 reproduces: on 4.4.1, an add-on calling ResourceUID.uid_to_path() leaves --editor --quit at exit 0. It also exposed a gap in the kit: the parse gate skips res://addons, so it never checked an add-on you are writing. There’s now gate.sh --include res://addons/<name>, which checks that one directory while third-party add-ons stay skipped. A broken plugin script fails it, and a correct EditorPlugin passes, on all four versions. Running it on the oldest version you support catches exactly your case.
Case 2 caught the kit too. The bash wrappers rejected unknown flags, but three GDScript tools behind them silently ignored a typo, and PARSE_GATE_MIN=abc quietly became 0. They all fail now, each with a test case.
The README now lists exactly which versions and OSes were measured.