Handling Data Streaming & Chunk Loading for Large 3D Maps in Godot

think you completely misread my posts. I’ve stated multiple times now that I have zero intentions of releasing a game, making money, or creating a commercial product. I am a hobbyist programmer playing with code and open-source plugins in my free time to see how engine architecture works.

There’s no anger here, and certainly no trolling—just a dev asking technical questions about Godot 4’s threading and rendering limits.

2 Likes

You are 100% spot on about the danger of chasing “cool systems” that lead straight to burnout. Feature creep is the ultimate project killer, especially for a solo dev.

For my specific case, because I’m treating this purely as an engineering playground rather than trying to ship a full game, building the underlying system is the fun part for me. That said, your point about pacing myself is incredibly valid. I’m keeping my scope strictly limited to just getting a player moving through streamed chunks smoothly without the editor blowing up before I even think about adding anything else.

2 Likes

That’s a fascinating take on the internal studio history, but according to CDPR’s tech leadership, the decision to drop REDengine was mostly a massive logistical pivot. They noted that they were spending way too many resources rewriting and adapting REDengine from scratch for every single game release. Switching to Unreal Engine 5 allows them to develop multiple massive projects in parallel across global studios and onboard new talent instantly without months of training on proprietary tools.

As for Unigine—it is an incredible engine, especially for massive 64-bit precision flight sims and industrial simulations. It handles large open spaces beautifully out of the box!

However, since I’m doing this purely as a hobbyist programmer to learn the inner workings of open-world architecture, switching engines every time I hit a bug defeats the purpose. The fun part for me is taking a popular, lightweight, open-source engine like Godot 4 and figuring out how to build those streaming systems myself.

Even if the plugins fight each other in the editor right now, finding the exact Vulkan conflict or bottleneck is exactly the kind of technical challenge I’m looking for. I’m going to stick with Godot for now.

4 Likes

Management is just trying to make excuses for its own incompetence. Things are actually much worse there. Having your own engine is a must-have for a large studio with its own projects.

The most valuable asset in any studio (or organization) is its employees, whom you should retain (my experience fully confirms this). If a studio frequently changes its staff to the point where the argument about the difficulty of training comes up, then something is “wrong” with the studio.

I doubt that Godot is currently ready for truly large-scale projects. Perhaps over time it will reach the necessary level of maturity. (I really hope so, since my project is supposed to reach a similar scale, though not as quickly as you have planned.)

And the issue with incompatible extensions shows that for such a project, you need to write your own extension that includes the necessary features. This, in part, brings us back to the question of having the studio’s own engine.

2 Likes

If you’re only interested in technical aspects and existing plugins are not adequate - write your own plugin. For large world streaming it’ll likely have to be a C++ extension.

2 Likes

as others mentioned - this is practically impossible for experienced solo dev, unless you have will power to work on it for 20 years straight and know you have small % of chance you’ll succed

the thing you can do is emergent design, some games don’t feel big because they were well designed or made by big teams, but because they contain elements that add to complexity without your effort, simpliest example - minecraft, it have procedural generation - which removes need of hand-design, it also have few block interactions that multiply exponentially

1 Like

However, I think there is still a slight misunderstanding about my end goal: I am absolutely not planning to spend the next 20 years hand-designing a massive, content-filled world. That would be impossible for a solo dev.

My focus is strictly on the technical pipeline.

To me, the success of this project isn’t a 100-hour RPG; it’s a stable, optimized framework where I can press a button, generate a zoned environment chunk, and have Godot stream it flawlessly. It’s all about building smart tools rather than doing brute-force manual labor.

Here is example : https://youtu.be/hSDXm6kXR7U
By combining the zoned approach of The Witcher 3 (self-contained regions) with the systemic automation you mentioned (like using custom procedural rules to scatter assets and stream data), the goal is to see how much world-building a solo dev can automate through code.

1 Like

Optimization will probably be the least of your concerns. At your core, you want to make a massive open-world game, probably an action RPG. As long as you stay within scope, I think you can put something together.

Some pitfalls I want to share, from my humble opinion, are:

  1. Don’t scope creep. Make your design document first.

  2. Be aware of your limitations and time.

  3. Don’t hyper-focus on graphics or cutscenes.

  4. Make your gameplay fun. If it is story-driven, have a solid story beforehand.

  5. Test, test, test.

The Witcher 3, at its core, is a cinematic, story-driven game trying to portray a strong narrative.

Find your core: story, combat, the world, or whatever you think you can truly portray in a way that people might find interesting enough to spend at least two hours of their lives on. Then build around that instead of trying to replicate big products.

If this is your first game, I would suggest making something small and failing small. The first few games developers make, whether solo or as a company, often fail. Learn from your mistakes. If you go too big for your first game, you might not even fail properly; you might just sink a ridiculous amount of time into something that goes nowhere.

  1. Have a clear core.

  2. Test it. Make a combat prototype if you are focusing on combat. Share your story with other people if it is story-driven. Make a small sample map if you are focusing on world creation. Judge the reactions, accept defeat if needed, and either find another core or improve it until you can prove that your core is clear and marketable.

  3. Design a game loop around your core that is fun. Even story-driven games have a game loop to support and portray the core.

Game engines, tools, and graphics are just ways of presenting the game. Godot, Unreal, and Unity all do similar things with different tooling. In game development, execution is usually not the hardest part. The hard part is making people want to play your game. Not even selling it, because you can always sell polish, but polish alone won’t make people play.

A bigger scope does not make a good game. Good design and a strong game loop make a good game. Making a good game is hard. Focus and invest in your theory and design before worrying too much about optimization. Map sizes, culling, and tools are mostly optimization concerns. People generally run powerful machines, and since you are solo, I doubt you will hit serious hardware limits with hand-crafted content.

These are just my humble opinions and personal experiences.

1 Like

The environmental composition is definitely my core inspiration. Areas like White Orchard are brilliant because they aren’t just flat maps; they are carefully structured with dense pine forests, river valleys, and scattered historical ruins. My goal isn’t to build high-fidelity copies of those assets, but to build an engine pipeline capable of handling those specific biomes—forests, canyons, and streaming structural layouts—entirely with placeholder blocks.

If the existing tools prove too unstable, writing a custom data-streaming plugin will definitely be the way forward. Looking at the big picture, a modular, plugin-based approach is exactly how I plan to handle the other high-level systems down the line:

Dialogue & Quests: A self-contained node framework for handling interactive branching paths.

Professions & Economy: An isolated subsystem for tracking items, recipes (like alchemy), and crafting logic.

Systemic AI & Spawning: A behavioral engine that dictates how entities interact independently of the player (e.g., wolves dynamically tracking and hunting deer across scene boundaries based on regional spawn conditions).

But as a few of you smartly pointed out, trying to build all of that at once is a recipe for burnout. For now, I am strictly limiting myself to the environment and data-streaming pipeline.

If I can cleanly stream stylized placeholder chunks to recreate the macro-layout of a multi-biome zone without the editor or the renderer throwing fit, that will be my milestone. Thanks again for all the technical advice and perspectives—time to go write some code!

Inspirations from Witcher 3:




Perhaps I have an idea why there is such a complete misunderstanding on all sides.

You’re given advice “based on experience,” but you want to create a one-size-fits-all solution. Yet no one has ever managed to do that. A computer game is, first and foremost, an idea. Its technical implementation is a compromise between capabilities (technical, ethical, and skill-based) and the current level of development and knowledge. That’s why every developer decides at every moment what they’re willing to sacrifice and what must be implemented.

I assume you haven’t made a single game? That’s why you’re asking questions that are crystal clear to game developers and generally considered unsolvable in principle—they can only be resolved in a specific situation “on the spot.”

I’ll be following your thread, as I find it interesting and very relevant to my field. But, speaking with disarming honesty, I don’t expect any major breakthroughs. Although there are plenty of interesting ideas that could certainly be borrowed.

Remember that The Witcher 3 was based on well-developed literary works (have you read them?), and the developers’ task was to convey their spirit in the game (which didn’t turn out very well). But without that literary foundation, this game simply wouldn’t have come together.

3 Likes

He’s focusing on the tech aspect, not pretending to do any of the writing or actual game design : he’ll build a branching dialogue system, RPG systems, dice or not dice based, etc.

Still a monumental task, but more realistically achievable.

He even said it’ll be all placeholder content to flesh out the tech.

I’ll also be following the thread, as I’m curious to see what he’ll be able to flesh out.

Bonus : he’ll probably release the stuff in the wild if it’s any good, whether as FOSS or indie commercialized, and that would give a leg up to other Godot teams to build Indie, A or AA RPGs, leading to one (maybe) up showing the big boys.

Cheers !

1 Like

Ah okay so it is purely technical experiment then developing game.

I would suggest to culling methods and agressive LOD’s with costume vertext shaders to fake animation of enviromental assets. You can ice the cake with hierarchical instance meshes to optimize the render calls.

I think Godot supports all these, you will need to write custom shaders later on. I experimented with volumatric fog on new godot and yields good results. You will need a good post processing effects especially bloom optimization for the sun light. Then move into lighting and light baking.

1 Like

I’m not sure that some colored boxes are enough to simulate a real open world. The problems start with thousands of detailed meshes with a lot more PBR textures. You would need some kind of mesh and texture streaming. AFAIK there is only a rough proposal for streaming in godot.

Haven’t read any news about it in 2026.

In this thread there is an interesting link to slides of a talk about open world in Horizon Zero Dawn (not too deep) :

There are some core mechanics to handle such a task.

IMHO the reactions here in the thread are coming from the circumstance, that a lot of beginners are targeting a project like an MMO, and none of them will come close to fulfill this mission. So the standard reaction to someone with this idea for an MMO is to keep them away from this concept.

I’m a little bit impressed by the progress of Vuntra City (procedurally generated open world city), solo dev, but not using Godot. There are some ideas for open world solutions.
https://www.youtube.com/@VuntraCity/videos

2 Likes

Yeah, it’s impressive. The girl made this in Unreal:

2 Likes

I really like your suggestion of using custom vertex shaders for the asset animations. Doing things like wind rustling or foliage displacement directly in the vertex shader—rather than relying on heavy CPU-bound logic or bone animations—is a brilliant way to offload the work to the GPU. Combined with aggressive distance culling and tightly tuned Level of Detail (LOD) steps, that should give the engine plenty of breathing room.

thank you so much for linking those Horizon Zero Dawn streaming slides—that is an absolute masterclass in runtime architecture!

You are entirely correct that moving simple placeholder cubes is a completely different beast than streaming thousands of high-resolution PBR textures and dense meshes. The lack of native, fine-grained mesh/texture streaming in Godot (beyond standard background scene loading) is definitely a major limitation for a truly boundless, seamless world.

However, this is exactly why I settled on the pre-baked, zoned architecture inspired by The Witcher 3.

By introducing a traditional loading screen when moving between major regions, the goal isn’t to stream heavy assets dynamically from the disk on the fly while running. Instead, the strategy is to load the entire zone’s unique texture arrays, structural layouts, and optimized MultiMeshInstance3D matrices directly into RAM/VRAM up front. Once inside the zone, the engine only has to handle proximity culling and LOD swaps, rather than battling disk I/O bottlenecks.

1 Like

I just found the absolute smoking gun for why these plugins are crashing and artifacting together. It turns out it’s a known, documented architectural bug in Godot’s GDExtension layer.

I dug up godot-cpp issue #1022 (Not able to call C++ _process(), _ready() reliably · Issue #1022 · godotengine/godot-cpp · GitHub ), which was ironically opened by the Terrain3D developers themselves while building the plugin.

They documented a massive inconsistency in how Godot 4 handles C++ virtual methods when a GDScript is attached to a GDExtension node. Specifically, attaching a GDScript to a custom C++ node completely silences or hijacks the underlying native C++ _ready() and _process() loops depending on whether you are in tool mode, running the game, or calling super(). It either causes silent failures or causes the editor debugger to break completely.

This completely explains why combining live Terrain3D nodes with ProtonScatter sampling breaks the editor’s rendering and threading pipelines—the engine is fighting itself trying to figure out which loop to execute.

Fortunately, the thread outlines the exact, bulletproof workaround that the developers used, which I am absolutely going to implement for my own custom streaming framework. Instead of binding virtual _ready() or _process(), you handle everything via _notification:


void MyCustomZoneNode::_notification(int p_what) {
    switch (p_what) {
        case NOTIFICATION_READY:
            custom_ready_logic();
            set_process(true); // Explicitly ensure processing kicks off
            break;
        case NOTIFICATION_PROCESS:
            custom_process_logic(get_process_delta_time());
            break;
    }
}

Moving forward with my pre-baked regional streaming framework, I’ll be bypassing standard virtual methods entirely and driving the chunk loading via low-level engine notifications.

2 Likes

I likely have some of the same inspirations as you although mine is perhaps more in the Skyrim realm as I have played Skyrim VR a lot and wanted to do some kind of VR project and play with interactions in VR. Without repeating what I have posted about mine, just playing with the capabilities of Godot has made me convinced it is perfectly possible to make a nice and rich VR world in it. The art assets will always be sub-par to whatever AAA studios would have so I have to be sober in this department, and most of them downloaded from SketchFab for a very very long time except for some specific use cases that i want to try out things.

I am also a developer who generally dont like frameworks and plugins as they often limit what I want to do instead of help me. Yes sure you can quickly get something running but after a while the limitations often surface and you then likely need to figure out how to modify it or get the people who made it to expand on it (which is very hard). So in my case I just made my own procedural generation to test out what I could comfortably run at 72 hz refresh rate and was quite surprised at how performant Godot is in the 3D department. In time I will pre-generate worlds that can be edited also to have some artistic control where I feel it is needed/wanted since its supposed to live on a multi user server anyway (likely coded in Nodejs).

Like you I am exploring this as a fun hobby project and have no plans to release anything either at this point. If I see it becomes something fun and solid I can reconsider but that is years away.

I will likely upload some kind of video to showcase some elements just to share progress in the near future.

2 Likes

I would absolutely love to see that video whenever you get around to uploading it. Sharing these kinds of progress milestones and comparing custom framework solutions is easily the best part of a technical hobby project. Good luck with the VR optimization, and I’ll keep an eye out for your updates!

1 Like

Good luck on your project. I added an update post on some experiences and where I plan to go next.

One thing I love about Godot is how fast things compile and start - the test cycles are very short - and naturally having many variables visible in the inspector for your nodes makes it very easy to tweak things in real time even as you run it.

1 Like