Blend multiple textures using shaders

[bmorledge-hampton19]

Godot Version

4.2.2

Question

I am trying to blend multiple textures by averaging the rgb values (Lerp?) and getting the product of the inverse alpha values at every pixel where they coincide. However, the default behavior of coinciding textures is for them to overlap, like so:

Instead, I would like the overlapping regions to blend together, as described above. Basically, I have two questions regarding this:

1. Is there a simple way in the editor to accomplish this task? I.e., can I use something like a CanvasGroup to blend all the child nodes?
2. Regardless of the answer to question 1, how would I implement a custom shader to perform this? I’m trying to get a better grasp on shaders and am not sure how to approach this problem. To be clear, the number of textures that need to blended can change dynamically during runtime, and I am not sure how to pass this information to the shader.

Conceptually, this seems like a simple task, but I’m having a really tough time implementing it. Any advice or additional resources would be appreciated.

[zdrmlpzdrmlp]

make use of canvasitem material

change the Blend Mode

[bmorledge-hampton19]

This is definitely a step in the right direction, but it’s still not the desired effect. Using the initial example, I would expect the region where all three squares overlap to be “more blue” than regions where two of the squares overlap. Instead, this is the result with additive blending:

It’s close, but it’s simply adding the color values, not averaging them.

Additionally, this method has the problem that all objects behind these will contribute to the additive mixing. E.g., if there is a white background behind the squares, they all become white, and I can’t find an obvious way to prevent this.

[ThreadAndSandpaper]

So take this with a grain of salt… I’m just getting started with shaders too:

I think I know what steps you would have to take in your shader to get it to do what you want, but I don’t have the skills to actually implement/test my hypothesis.

I’d start by setting up my textures as children of a canvas group node with tweaks set to 0, which is mostly just to get a single texture to write to that’s the correct size to hold your other textures.

I’d make a shader on the canvas group, and use uniform 2dsamplers to feed in the textures from the children, and uniform vec2s to get their positions into the shader.

here’s where it get a bit fuzzy: you sample all the textures in your fragment function, offsetting the sampled UV by the texture’s position. The math will be tricky because everything will be normalized, but I think it’s possible.

anyway, the last bit is easy: once you have the vec3s you just add them together and divide by the number of samples

what would be AWESOME is if you could customize render modes, I could really use that, too.

[dharhix]

Seems to me you’re only blending the green square with the other two? For getting the three squares to mix properly they all have to have the blend mode in CanvasItem. The other two are just overlapping without mixing.

Also, quite possible I misread your intentions.

[ThreadAndSandpaper]

I think the op is looking for a blend mode outside of the supported ‘add’, ‘subtract’ and ‘multiply’. If that’s the case they need more bespoke shader magic. I feel their pain, I’m working on a similar problem where i need hard light, and getting multiple textures to play nice together is not coming easy.

[jesusemora]

is this 2D or 3D?

[bmorledge-hampton19]

I’m doing this all in 2D

[bmorledge-hampton19]

Thank you for the thorough reply. I’m still having trouble implementing your suggestions, but you’ve at least made me think a bit more about identifying the problems with my approach.

Namely, how do you convert an existing node to a texture for use in the shader? In my example with the squares, they’re simply ColorRect nodes, and I assumed there would be a way to directly sample them in the shader. However, this doesn’t seem to be the case.

Eventually, my implementation will use dynamic Line2D nodes, so caching static textures isn’t really an option. It looks like SubViewports can be used to get around this, but this feels very hacky and inefficient, as I could potentially have 100s of Line2D nodes that would need to be sampled individually.

I’m beginning to think that I may be approaching this problem incorrectly. If I’m using Line2D nodes, perhaps a shader interpolating between all of them just isn’t an option and I’m tethered to the blend mode on canvas materials.

Of course, I’d love to be wrong here. If anyone has any additional insight into this problem, I’m all ears.

[jesusemora]

@bmorledge-hampton19

Regardless of the answer to question 1, how would I implement a custom shader to perform this? I’m trying to get a better grasp on shaders and am not sure how to approach this problem. To be clear, the number of textures that need to blended can change dynamically during runtime, and I am not sure how to pass this information to the shader.

if these are different objects that must interact, you are limited by render order.
your objects render one at a time. with a shader you get access to the result of the frame and can obtain the color, and then write to it. and the next sprite will get the result of this.

[ThreadAndSandpaper]

Yeah, this is a doozy of a problem and I’m not sure what you can do. I know I’m the blind leading the blind here, but if you could make multiple back buffer copies (one for each node) and reference their textures in a single shader that might work… but I don’t know if that’s even possible.

Okay, totally just spitballing here (I hope that’s okay?):
What if you (somehow?) sampled the screen and for each pixel you got a count of how many line 2ds were drawing at that point. That’s just a big array (or dynamically generated texture) you could pass into a shaders on each line (probably on a back buffer copy on each line, actually so that all the shaders and the array were working in the same normalized space). For each fragment, you multiply the alpha by 1/num lines at that UV. That would effectively give you full opacity at each pixel with an equal proportion supplied by each layer. Again, no idea if it’s do-able, though.

[bmorledge-hampton19]

@ThreadAndSandpaper

Okay, totally just spitballing here (I hope that’s okay?):

Don’t worry. This dialogue is much appreciated! I just looked into back buffer copies, and it seems like a really useful tool. However, as @jesusemora mentioned, I’m still limited by render order. If I’m understanding things correctly, the back buffer copy gets the entirety of the viewport that has been rendered so far. This still doesn’t isolate the graphics associated with each node.

As for manually extracting the fragment data for each line, I can’t think of a way to do this either. My Line2D objects have both a gradient and a width curve, and I assume there’s no easy way to reverse engineer the output.

Again, thank you for bringing all this up, even if it doesn’t seem like a solution just yet. I’m still learning a lot through this process!

[ThreadAndSandpaper]

I’ve been mulling this over (more interesting than my job today) and I think I might have a plan of attack. It’s hacky and unproven, but here I go!

First, each line 2D is the child of a back buffer copy set to the rect of the screen/viewport. I think this will give you the line as a texture on transparency that you can work with in the shader. It has the advantage of every line 2D being normalized to the same size. (i.e. a given UV is in the same location on every line)

Next, you set up what I’m going to call your density map. This is a texture that stores how many lines are present at a given pixel, and it’s where this idea gets a a bit silly. You create a viewport with the same pixel dimensions as your screen space (or whatever you defined in your back buffer copy) and for every line in your project you create a corresponding line in the viewport. Basically you are mirroring everything from your main scene here, so it’ll need to update accordingly.

You set the color of all of those lines to a very dark gray, something like Color(1/256,1/256,1/256) and give them an “add” canvas item shader. This will result in a texture that gets lighter where there are more lines and darker where there are fewer. If there are no lines the color should be (0,0,0) if there is one line (1/256, 1/256, 1/256), two lines would be (2/256, 2/256, 2/256) and so on.

Alright, you send that viewport texture to a uniform sampler 2D on each line in your real scene. You can do this via script using material.set_shader_parameter(). Not sure if you have to send it each frame, and not sure if tree order will affect this… but now you have the data you need.

Each back buffer copy will need to use the “add” render mode. Each fragment will look something like this:

void fragment() {
	COLOR.a = 1.0/(texture(density_map, UV).a*256.0);
}

Which should drop the alpha to (1/number of lines found at a given pixel) then it’s adding them all together. Functionally you are averaging… just reversing the order of operations: dividing first then finding the sum.

Ooft. That was an essay! and a fun thought experiment. Seems like a lot of work, I wish there were more render modes!!! Again, totes unproven, but might work ¯_(ツ)_/¯.

PS. Because you are using the add mode in the shader, too, you might want to build all your real lines in a viewport. Then you can lay the whole viewport over your background without having to worry about it being weird with the render mode.

[bmorledge-hampton19]

Well, it took longer than I was hoping, but I finally finished a proof of concept that achieves the desired effect! Thanks so much @ThreadAndSandpaper for your last post. It was a huge help in finally figuring out how to do this. Encoding information in viewports helped immensely, and I actually didn’t even need to perform any back-buffer reads beyond the final pass with the CanvasGroup to ensure that transparency was properly preserved. You can find the completed project here:

GitHub - bmorledge-hampton19/blended_average_renderer_test: A test project...

All in all, this was a BIG learning experience for me. I hit more roadblocks than I’d like to recount while putting this all together, so I’ll just give a couple of the highlights here:

• Information that needs to be altered by and passed between shaders can be encoded in a SubViewport and retrieved using a uniform sampler2d. This still feels a bit hacky to me, and I don’t have a great idea of what the performance implications are for such a method, but hey, it works! I’ve actually started a separate thread to discuss this further.
• CanvasGroups are a great way to separate a texture’s render mode from the rest of the canvas. This is how I was able to use additive blending for the main computations but not in the end result. (I.e., the result can be rendered on a white background without getting completely washed out.)

I’m marking the thread as solved for now, but if I think of any ways to refine/simplify the solution I’ll post about it here again.

[ThreadAndSandpaper]

Holy cow! I’m so glad it worked! Great job seeing it through. I’m sure it was full of unexpected complications, but I’m so glad you got there <3