Background Loading freezing main thread

abitrolly · April 9, 2024, 3:37pm

What NVIDIA logo is doing here? It looks like something injects into Godot process and that can cause all sorts of problems.

The only advice I have is to try it on other platform, where tools are less invasive.

midrule · April 9, 2024, 6:08pm

As mentioned above this happens on multiple different PCs, one of which is integrated graphics so there’s no nvidia. It seems that there’s a pr that will fix this and another issue I was having in 4.4 though, so it’s a godot issue that’s being heavily improved in the future

abitrolly · April 9, 2024, 11:16pm

Is there a link to the PR?

midrule · April 9, 2024, 11:29pm

github.com/godotengine/godot

Ubershaders and pipeline pre-compilation (and dedicated transfer queues).

godotengine:master ← DarioSamo:transfer_and_pipelines

opened 05:40PM - 08 Apr 24 UTC

DarioSamo

+4250 -1812

This is a big PR with quite a bit of history that should be evaluated very thoro…ughly to evaluate where do we want to make some concessions and try to mitigate the side effects as much as possible. However, the benefits are essential to shipping games with the engine and making the final experience for users much better. To read further on @reduz's notes about the topic, you can check out these documents ([Part 1](https://gist.github.com/reduz/97634e58f4700793d32a5591b5f4fcee) and [Part 2](https://gist.github.com/reduz/2c180c264c0d5a14e72ac3839ba2f304)). Due to the complexity of this PR and how 4.3 is currently in feature freeze, I'd definitely not consider this PR until 4.3 is out. **If you want the TL;DR: skip ahead to the two videos with the TPS demo to see the immediate difference**. NOTE: These improvements will only affect the _Forward+_ and _Mobile_ renderers. No changes are expected for _Compatibility_. ## Transfer queues First of all, this PR supersedes the [transfer queues PR](https://github.com/godotengine/godot/pull/87590) and effectively uses it as its base. The reliance on needing to unlock parts of the behavior of RenderingDevice to make it multithread-friendly to reap the benefits was far too much to keep both PRs separate. As mentioned in that previous PR, merging it as is will cause a small performance regression unless https://github.com/godotengine/godot/pull/86333 is merged first. ## Pipeline compilation Modern APIs like Vulkan and D3D12 have made rendering pipeline management very explicit: their creation is no longer hidden behind the current rendering state and handled on demand by the driver. Instead, the developer must create the entire pipeline ahead of time and wait on a blocking operation that can take a significant amount of time depending on the complexity of the shader and the speed of the hardware. This has seen some improvements recently with the introduction of new extensions like [VK_EXT_graphics_pipeline_library](https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VK_EXT_graphics_pipeline_library.html), but as always, Godot must engineer solutions aimed towards resolving the problem for as much hardware as possible and use such features optionally for optimization in the future. Godot has the responsibility to perform as fast as possible for the end user, which leaves it no choice but to generate pipelines with the least amount of code and requirements as possible. The engine achieves this through the use of shader compilation macros (shader variants) and the use of specialization constants to optimize code for a particular pipeline (pipeline variants). While Godot resolves shader variant compilation and can even ship the shader cache to skip the step altogether, it coudn't resolve pipeline variant compilation ahead of time before this PR at all. If you're familiar with the "stutters when playing the game for the first time" phenomenon that has plagued all games shipped with Godot 4's RD-based renderers, **this is pretty much the entire root of the problem**. This is not a problem exclusive to Godot as it's been very evident in lots of commercial releases that include very extensive shader pre-compilation steps the first time a game starts or a driver update happens. [The issue is so prevalent even Digital Foundry points it out as the #1 problem plaguing PC game releases in this article](https://www.eurogamer.net/digitalfoundry-2023-13-ways-to-improve-pc-games-in-2023) and they never fail to mention the existence of the problem on any new game that suffers from it. ## Ubershaders for Godot 4 The exciting part about this PR is an effective solution was developed to address this problem completely without the need to introduce extensive shader pre-compilation steps or any input from the game developer whatsoever. Instead, attempts have been made to **make pipeline compilation a part of loading assets as much as possible**. Not only does this mean most pipeline compilation is no longer resolved at drawing time, it can also even be done in **background threads** and presented as part of a regular loading screen. That means the game is no longer at the mercy of the renderer introducing these stutters when it needs to draw, but it makes the behavior much more predictable and able to be handled as part of a loading process. The main improvement this PR makes is the introduction of ubershaders once more to the engine, but these are quite **different from what was previously done in Godot 3**. Unlike the previous version of the engine, these shaders do not correspond to generating text shaders with specializations and compiling them in the background, which could lead to a lot of CPU usage that'd take lots of time in weaker systems. Instead, ubershaders are mostly still very similar to the current shaders the engine already has, with a key difference: **specialization constants are pulled from push constants instead**. This means that the engine is able to use a version of the shader already that can be used for drawing immediately while the specialized version is generated in the background. Pipeline variants are **much faster** to generate like this instead of relying on runtime shader compilation to insert the constants as part of the shader text, as they work directly on the SPIR-V and skip the need to compile the shader from text again. Specialization constants are a big part of how Godot optimizes pipelines, but they've been limited by parts of the design as to how many can actually be used. Any additional constant implied an explosion of variants that led to the pipeline cache structure getting even bigger (160 KB in just pointers in Forward+ for any single material in `master` at the moment!), and every new addition meant that if the state is very dynamic, stutters would occur due to extra pipeline compilation. This was quite evident in the Mobile renderer, which uses a specialization constant to disable lights if they're not used: as soon as a light popped up, then stutters due to pipeline compilation were inevitable. With this change, a new simple hashing system for pipeline caching is introduced instead: - The required pipeline is requested in its specialized form from the cache. - If the pipeline is not available yet, compilation is started on the background without stalling the main thread. - The ubershader pipeline (which has been compiled at loading time) is used instead. The specialization constants are pushed as part of the push constant instead along with some other rasterization state parameters (like backface culling). ## Pipeline compilation at loading time The other key part behind the PR is the introduction of pipeline compilation of the ubershaders in two extra steps. - During the creation of the surface cache in the renderer (stutter on scene setup). - During loading of ArrayMeshes (can be pushed to a background thread). The difference in how both of these changes work together is pretty evident on the TPS demo by simulating a clean run as an end user would see the first time they run the game. A big chunk of the stutters are gone, especially the one that happens the first time the character shoots, which is a typical case of a stutter that only happened at drawing time despite the effect being loaded in the scene tree already. Both of these videos have pipeline caching disabled and the driver cache deleted between each run. `master` (dc91479082bd9cd56cd74282573492fe1ba9618a) https://github.com/godotengine/godot/assets/538504/4152255e-ee0c-409c-9fa9-06cde148a62a `transfer_and_pipelines` https://github.com/godotengine/godot/assets/538504/391eaa02-00bf-4986-8972-f1d44d80ab1b It's also worth noting how the loading screen animation actually plays out more of the time instead of having one big stutter at the end due to the initial pipeline compilation at drawing time. These loading times are also significantly shortened by making multiple improvements to the behavior of both the shader and pipeline compiler, allowing it to multi-thread more effectively and use more of the system's resources. ## The negatives (and how we can mitigate them) As was expected, these benefits do not come for free. But there's multiple ways we can attempt to mitigate most of the extra cost and this is an area I'm open to feedback on and that we can further optimize in future versions as well. * An extra shader variant had to be introduced. @reduz has diligently always recommended against adding shader variants as it leads to a combinatory explosion, but this is one sacrifice that had to be made to allow ubershaders to exist. However, there's potential for reducing some of the existing shader variants to dynamic paths in the ubershader if possible. This is also significantly mitigated by improvements to the shader compiler's multithreading and it should be a non-issue for games that ship the SPIR-V shader cache. * Loading times are bound to be longer as pipeline compilation is pushed here instead. This is the intended effect and it's paying a cost upfront that would happen at drawing time otherwise (which is less preferable). That said, pipeline caching always plays a part here and it'll speed up loading times in later runs as it should. * Higher memory consumption from extra pipelines and shader variants being compiled that might go unused. This is sadly one cost that must be paid no matter what and can hopefully be mitigated by implementing better detection of features in use. The biggest reason behind these negatives is the engine's flexibility. Features can be turned on and off without explicit operations from the user at a global level: a scene can be instanced to use VoxelGI while another one might use Lightmaps instead. As a matter of fact, this is exactly what the TPS demo does, so any run of the game must pre-compile the Lightmap variants because it can't know ahead of time which method the user has chosen without looking at the scene's contents, which is yet to be instanced during mesh loading. One of the things I hope to improve while this PR is in progress is reducing the amount of variants that are pre-compiled as much as possible. Therefore it'd be great to gather feedback on which of these methods are most effective and how to implement them: - Detecting features that aren't used at the project level would help significantly. If a user never uses VoxelGI, we shouldn't be pre-compiling variants for it. The biggest culprits here that I could identify are features like separate specular and motion vectors. Adding some form of tracking somewhere at a global level so the engine can know ahead of time without having to instance scenes would be very helpful here. - Assuming features aren't enabled by default and going back to compile them if they are: this is actually something that's partially implemented with the 'advanced' shader groups already. Upgrading this to a per-feature detection could help a lot towards reducing pre-compilation and delegating it to the surface cache setup. If a developer wishes to properly delegate the pre-compilation during mesh loading, all they need to do is just instance scenes first with the appropriate features. - Allowing developers to opt in or out of variants to pre-compile at a global level instead. This is likely a very good solution for more experienced developers to fine-tune their game if it actually has a significant amount of shaders and materials that need it. This would likely be a simple set of toggles indicating which features shouldn't be compiled as the developer knows they'll never make use of them. It's worth noting that under the current implementation, none of these leading to false positives will lead to the engine misbehaving: at worst, it just causes the drawing time stutters the current version already has. ## Testing methodology - As of https://github.com/godotengine/godot/pull/90271 (which is currently merged), you can now disable the pipeline cache at the engine level (if the API supports it). It is essential to turn this setting off to do any comparisons and to simulate the experience of a clean run of a game. ![image](https://github.com/godotengine/godot/assets/538504/cedc11dc-f500-42a1-a6fb-66dda655c822) - Delete the driver pipeline cache. This is the last barrier of defense the driver has if the application doesn't implement a pipeline cache of its own. This heavily depends on the IHV and the platform (e.g. on Windows & NVIDIA it's located at `%LocalAppData%/NVIDIA/GLCache`). No test should be considered valid without deleting this cache first and foremost. ![image](https://github.com/godotengine/godot/assets/538504/dfaaa859-e7b7-4eae-ba27-2b005e62c240) - Run the game! Trying to measure the results can be a bit tricky as the results are heavily dependent on the behavior you see in a project. As the benefits are more visually evident as seen in the videos, it is hard to measure the effects of pipeline compilation at drawing time as they present themselves as stutters that happen all throughout the game instead of one particular scenario. ## New performance monitors Some new statistics have been added to the performance monitors which should help verify without a shadow of a doubt if the pipeline pre-compilation is working as intended. There's four different pipeline compilation sources that are identified and they should help towards understanding where a extended loading time or stutter comes from. ![image](https://github.com/godotengine/godot/assets/538504/28d2c83c-2aad-470f-9e21-738e1fdf0e5f) Quoted from the documentation added by this PR: * `RENDERING_INFO_PIPELINE_COMPILATIONS_MESH`: Number of pipeline compilations that were triggered by loading meshes. These compilations will show up as longer loading times the first time a user runs the game and the pipeline is required. * `RENDERING_INFO_PIPELINE_COMPILATIONS_SURFACE`: Number of pipeline compilations that were triggered by building the surface cache before rendering the scene. These compilations will show up as a stutter when loading scenes the first time a user runs the game and the pipeline is required. * `RENDERING_INFO_PIPELINE_COMPILATIONS_DRAW`: Number of pipeline compilations that were triggered while drawing the scene. These compilations will show up as stutters during gameplay the first time a user runs the game and the pipeline is required. * `RENDERING_INFO_PIPELINE_COMPILATIONS_SPECIALIZATION`: Number of pipeline compilations that were triggered to optimize the current scene. These compilations are done in the background and should not cause any stutters whatsoever. ## TODO - [x] Pass CI and address compatibility breakage (if there's any). - [x] Make sure compatibility renderer hasn't had any regressions from modifying the common classes. - [ ] Address any multi-threading issues that can possibly arise from both transfer queues and this PR. - [ ] Test D3D12 for any regressions or new issues introduced from multithreading. - [ ] Evaluate further ways to reduce the amount of pipelines being pre-compiled. - [ ] Evaluate any possible CPU-time regressions during drawing and how to mitigate them. - [ ] Evaluate adding this improvement to Canvas Renderer as well. --- Contributed by [W4 Games](https://w4games.com/). 🍀