CI/CD pipeline builds break sporadically

Godot Version

4.2.1.stable.official.b09f793f5

But also witnessed in previous version.

The docker image trying to build the project:

Docker file running to build the project
FROM ubuntu:23.10

ENV DEBIAN_FRONTEND=noninteractive
ENV GODOT_VERSION "4.2.1"
ENV GODOT_ARCH "x86_64"

USER root

RUN apt-get update && \
    apt-get install -y --no-install-recommends ca-certificates fontconfig unzip wget zip && \
    rm -rf /var/lib/apt/lists/*

RUN wget https://downloads.tuxfamily.org/godotengine/${GODOT_VERSION}/Godot_v${GODOT_VERSION}-stable_linux.${GODOT_ARCH}.zip && \
    wget https://downloads.tuxfamily.org/godotengine/${GODOT_VERSION}/Godot_v${GODOT_VERSION}-stable_export_templates.tpz && \
    mkdir ~/.cache && \
    mkdir -p ~/.config/godot && \
    mkdir -p ~/.local/share/godot/export_templates/${GODOT_VERSION}.stable && \
    unzip Godot_v${GODOT_VERSION}-stable_linux.${GODOT_ARCH}.zip && \
    mv Godot_v${GODOT_VERSION}-stable_linux.${GODOT_ARCH} /usr/local/bin/godot && \
    unzip Godot_v${GODOT_VERSION}-stable_export_templates.tpz && \
    mv templates/* ~/.local/share/godot/export_templates/${GODOT_VERSION}.stable && \
    rm -f Godot_v${GODOT_VERSION}-stable_export_templates.tpz Godot_v${GODOT_VERSION}-stable_linux.${GODOT_ARCH}.zip

Question

Hi there,

I am trying build my project in a headless pipeline but run into several issues.

The pipeline is acting weird and I don’t have any ideas left what I could try to stabilize it.
Namely:

  • breaking sporadically (restart finishes the pipeline just fine. most of the time at least)
  • not properly exporting assets like textures (as far as I can tell).

Is it currently just not possible to automate the export without an installed graphics-card/screen-server in a pipeline-runner?

Or am I just doing something completely wrong?

From what I’ve seen the build with --headless is not able to generate the .godot-directory properly… but including that directory in the VCS does hardly seem right.

The pipeline is breaking sporadically

$ cd game && godot --export-release --headless "Linux" ../export/linux/$EXPORT_NAME.x86_64
Godot Engine v4.2.1.stable.official.b09f793f5 - https://godotengine.org
 
WARNING: Custom cursor shape not supported by this display server.
     at: cursor_set_custom_image (servers/display_server.cpp:505)
ERROR:  Config File "res://.gut_editor_config.json" does not exist.
GUT config loaded
savepack: begin: Packing steps: 102
	savepack: step 2: Storing File: res://.godot/imported/finite_state_machine.svg-3f20a362149aa0d11e1dd7d3143fd2e1.ctex
...
	savepack: step 46: Storing File: res://.godot/imported/compute_relax.glsl-b06f9e60cda7719b78bde9673f2501b7.res
ERROR: Cannot erase nonexistent section "params".
   at: erase_section (core/io/config_file.cpp:120)
	savepack: step 46: Storing File: res://addons/proton_scatter/src/modifiers/compute_shaders/compute_relax.glsl.import
...
	savepack: step 102: Storing File: res://project.binary
savepack: end
ERROR: Parameter "data.tree" is null.
   at: get_tree (scene/main/node.h:413)
================================================================
handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.2.1.stable.official (b09f793f564a6c95dc76acc654b390e68441bd01)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
ERROR: FATAL: Index p_index = 1 is out of bounds (size() = 0).
   at: get (./core/templates/cowdata.h:158)
/usr/bin/bash: line 168:    15 Illegal instruction     (core dumped) godot --export-release --headless "Linux" ../export/linux/$EXPORT_NAME.x86_64
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

The error is not specific to linux-export but sporadically also happens on windows and another linux (server) build.

Assets are not exported properly

Textures are mapped wrongly (distorted) onto models or are missing entirely.

Edit

Building the exact same project locally with --headless results in the same issues (breaks sometimes, and assets glitched or missing).

Building it without the --headless flag builds it reliably and also has no corrupted assets.

Initializing .godot-directory without --headless and running the build with that .godot-directory fixes the issues regarding the assets but is still prone to crashes.

In my experience, the headless export should run fine, unless you’re doing a very specific build config ( maybe, no idea ). I’ve used it with Github actions for PC and Android builds, granted, with the excellent chickensoft action. I’m doing pretty much what you’re doing, here’s the workflow for reference.

Where are you running this image? Do you get the same issue with debug builds?

The content of one of the exports that is randomly failing would be:

Export Preset
[preset.1]

name="Linux"
platform="Linux/X11"
runnable=true
dedicated_server=false
custom_features="client"
export_filter="all_resources"
include_filter=""
exclude_filter=""
export_path="../export/linux/taog.x86_64"
encryption_include_filters=""
encryption_exclude_filters=""
encrypt_pck=false
encrypt_directory=false

[preset.1.options]

custom_template/debug=""
custom_template/release=""
debug/export_console_wrapper=1
binary_format/embed_pck=false
texture_format/bptc=true
texture_format/s3tc=true
texture_format/etc=false
texture_format/etc2=false
binary_format/architecture="x86_64"
ssh_remote_deploy/enabled=false
ssh_remote_deploy/host="user@host_ip"
ssh_remote_deploy/port="22"
ssh_remote_deploy/extra_args_ssh=""
ssh_remote_deploy/extra_args_scp=""
ssh_remote_deploy/run_script="#!/usr/bin/env bash
export DISPLAY=:0
unzip -o -q \"{temp_dir}/{archive_name}\" -d \"{temp_dir}\"
\"{temp_dir}/{exe_name}\" {cmd_args}"
ssh_remote_deploy/cleanup_script="#!/usr/bin/env bash
kill $(pgrep -x -f \"{temp_dir}/{exe_name} {cmd_args}\")
rm -rf \"{temp_dir}\""

Github actions are a bit too incomprehensible for me at the moment… will try to figure out what they use (the download seems the same, so it might be either the base-image that is used or my preset).

The image is being run on the gitlab-runners I host myself (also as self-hosted gitlab).

Export for the previously mentioned preset would be the following execution (the root of the project is inside a sub-directory):

cd game && godot --export-release --headless "Linux" ../export/linux/$EXPORT_NAME.x86_64

That’s pretty much what the build.sh you linked chickensoft project (just inside bash-script kinda roundabout).

Might be something inside my project preventing a smooth build. Will try a clean project tomorrow and add the addons piece by piece to see if it starts to break at some point.

Didn’t checkout the links to check what type of projects those are. But I think it might be an issue with either one of my addons or the project being 3d.