Godot inconsistent loading times driving me mad - ResourceLoader.load_threaded_request

[jesterrrrrr]

Godot Version

4.2.2

Question

I’ve been fighting this for months now and am completely lost. When using ResourceLoader.load_threaded_request to load scenes loading times are completely inconsistent, spiking randomly.

The inconsistency makes it hard to test. In game with visuals it is worse, reaching 5+ seconds, often after the loading time rises it never goes back down. Taking 5 seconds to load 3mb is outright killing my attempt to stream in my scenes.

I have absolutely no idea where to go form here and any help would be greatly appreciated.

Example code

I have created a simple script to demonstrate. The scenes loading are named t0x0, t1x0, t2x0… and so on up to 6x6. It just loads one after another:

extends Node3D

var path = "res://Terrain/"
var finalPath
var loading = false
var start
var p = 1
var x = 0
var y = 0

# Called when the node enters the scene tree for the first time.
func _ready():
	print("Beginning loading time test")

# Called every frame. 'delta' is the elapsed time since the previous frame.
func _process(delta):
	if !loading:
		print("starting a load, x: "+str(x)+", y: "+str(y))
		start = Time.get_ticks_msec()
		finalPath = path+"t"+str(x)+"x"+str(y)+".tscn"
		print("path: "+finalPath)
		ResourceLoader.load_threaded_request(finalPath,"",false,0)
		loading = true
	else:
		if ResourceLoader.load_threaded_get_status(finalPath) == 3:
			var loadingTime = Time.get_ticks_msec() - start
			print("loading time: "+str(loadingTime))
			loading = false
			x+=1
			var res = ResourceLoader.load_threaded_get(finalPath)
			#trying to make 100% sure it's not cached and get deleted
			res.take_over_path(str(p))
			res = null
			p+=1
			if x > 6:
				x = 0
				y+=1
				if y > 6:
					x = 0
					y = 0

(Scenes consist only of MeshInstance3Ds. No scripts, nothing else. Each scene has an identical amount of data - 3.22mb split over 661 unique .mesh files)

Here is some output:

starting a load, x: 4, y: 2
path: res://Terrain/t4x2.tscn
loading time: 467
starting a load, x: 5, y: 2
path: res://Terrain/t5x2.tscn
loading time: 450
starting a load, x: 6, y: 2
path: res://Terrain/t6x2.tscn
loading time: 2734

All load times where around ~450 until it got to that point, then it went back to ~450 again.

Running a second time we get more inconsistent load times on different scenes:

starting a load, x: 5, y: 0
path: res://Terrain/t5x0.tscn
loading time: 3484
starting a load, x: 6, y: 0
path: res://Terrain/t6x0.tscn
loading time: 2545
starting a load, x: 0, y: 1
path: res://Terrain/t0x1.tscn
loading time: 466
starting a load, x: 1, y: 1
path: res://Terrain/t1x1.tscn
loading time: 465
starting a load, x: 2, y: 1
path: res://Terrain/t2x1.tscn
loading time: 467
starting a load, x: 3, y: 1
path: res://Terrain/t3x1.tscn
loading time: 450
starting a load, x: 4, y: 1
path: res://Terrain/t4x1.tscn
loading time: 483
starting a load, x: 5, y: 1
path: res://Terrain/t5x1.tscn
loading time: 3600
starting a load, x: 6, y: 1
path: res://Terrain/t6x1.tscn
loading time: 2195

What I have tried:
I’ve tried creating my own Thread object and loading normally on that. I’ve made sure nothing is running in the background on my PC that could mess with it. I’ve tried exporting a .exe and running that. I’ve tried moving the .exe to a different HDD, testing both on a mechanical and on my steam library NVME SSD. Tried using subthreads. I’ve also tried saving resources locally to scene since I figured 1 big file is better than 661 small files, but this more than doubles the total size and made loading even slower. I’ve checked task manager and Memory, Disk, and CPU usage are all very low. I’ve tried checking the progress that ResourceLoader offers but it’s just 0,0, 0.5, 0.5, …, 1.0.

One final note: Sometimes I can run this code without a problem for 10 minutes+. When I instance these scenes and add them to my scene (I make sure not to include these operations in the timings) the problem happens within a minute pretty much every time.

[pennyloafers]

I.thinknyou may need to go deeper into Godot.

Using C++ profilers

[jesterrrrrr]

Damn, was really hoping to avoid that. Having never even built anything or touched C++ I’m not sure I’d be able to make anything of it.

I did notice Godot 4.3 beta has a bunch of thread related upadtes:

Multiple fixes to improve thread safety during resource loading. These make the multithreaded resource loader safer to use with fewer edge cases that could result in deadlocks

So I gave that a go. Average load time has doubled, and my new maximum is 11 seconds.

[jesterrrrrr]

I haven’t been able to solve the issue, but have found a work around, which I’ll post for any potential future readers suffering from the same problem.

Saving scenes as .glb using Godot 4’s own .glb exporter (using GDScript in the editor, no manual work per scene) results in smaller file sizes (~7mb vs ~12) and massively reduced loading times:

-.tscn 700ms - 11,000ms (4.3 beta1, was less in 4.2.2)
-.glb 50ms - 500ms

Maybe for tscn it was generating LODs or something, I’m unsure and can’t even figure out how to test it, no LODs appear when I view it with wireframe. Regardless the spiking loading times still exists, but is no longer big enough to cause me any problems. I find it strange that Godot’s own formats are significantly larger files.

[indicainkwell]

tscn is a text format designed to be version controlled and human readable. It’s not the fastest to parse. When you export your project tscn is converted to scn the binary format. scn is parsed much faster.

glb files are also converted to scn.

You can always save your scene as binary like t0x0.scn.

Fun fact: “scn” used to be the main (only?) scene format. “tscn” and “tres” were added later for better version control and team collaboration.

[jesterrrrrr]

Oh wow I had absolutely no idea about that. Thank you! Back to testing to see what’s fastest I guess.

[jesterrrrrr]

Well, I gave it a go and while the .scn files are significantly smaller than .tscn as you’d expect, they are still slower than glb though for me. ~200vs ~50, but they are also causing massive stutters if assets are local to scene (if they aren’t then its 800ms loading time similar to tscn).

@indicainkwell

glb files are also converted to scn.

Could you tell me a little more about this? When does this happen, would all my .glb files become .scn upon deployment? Given my results that line has me worried now!

[indicainkwell]

You mind sharing your test scene and the glb file? I’m not able to reproduce your results. A 35 mb scn file made up of mesh data ran with a dev build of godot loads in about 50ms for me.

[jesterrrrrr]

Sorry was just editing that post!

Are you loading on a thread? Really appreciate your offer to try it out, if I’m still dying from this I’ll put together as simple as possible a test scene.

[indicainkwell]

I tested load_threaded_request() and load(). Both had similar results

[indicainkwell]

Here’s docs on the 3d formats: Available 3D formats — Godot Engine (stable) documentation in English

GLTF files are converted to “scn” and saved to the “.godot” folder when imported in the editor.

[comradecid]

dumb question:

you noted that the terrain chunks have an identical amount of data… is the data itself also identical? is it invariant, or are there marked differences between each?

given the latter discussion regarding conversion to binary format, and assuming that there might be some measure of compression/decompression occurring during load efforts, i’m wondering if there was something to the data itself that might have played a factor, such as aspects that are more time-consuming to decompress or parse, etc.

also: suspecting that whatever method godot is employing doesn’t necessary guarantee thread sovereignty/prioritisation is there anything else that could be competing with thread resources? what’s happening in the overall game? system processes?

[jesterrrrrr]

Nothing. Meshes with verts, no UVs, no vert colors, no weight data, nothing. The mesh data was all created in an identical fashion too, with just a seed value as the difference maker. It’s a subdivided grid with normals exported, and it’s bumpy enough such that it’s basically identical file size.

I did my best to check for PC issues, and couldn’t see anything. There was no indication in the task manager that my PC was under any stress, GPU, CPU, or disk. I closed everything I could and the problem would still occur without any obvious cause, seemingly random. The game itself is doing almost nothing - it only moves the camera based on WASD keys and is rendering the terrain.

One thing I’ve noted is that when I created a test that didn’t render the scenes, just loaded scenes then set them to null, it happened significantly less frequently, and I mean really significantly. So it would seem just rendering stuff is having an affect?

[comradecid]

interesting. loading itself is one job; actual instantiation, rendering, etc. is another; i wonder what the interrelation might be.

in possibly-related-and-hopefully-relevant news, i very recently came across a short reddit thread discussing load times (https://www.reddit.com/r/godot/comments/1bu8ni8/incredible_long_initial_loading_time/), featuring this response:

Shader compilation or also a common reason for slow loading is when assets are stored as text resources.

Especially for 3D assets using .tscn or .tres files is performance seppuku. If you do not store your Meshes in a binary format with .mesh or .res you are doing it wrong.

i’m curious to see what effect this approach might have. clearly, ‘loading’ entails loading files from the local system, parsing them , and then performing some level of pre-processing on the data. i’m wondering how this pipeline differs when it is instead managing more granular subcomponents, instead of a larger formal scene…