Weirdness when comparing depth in compositor effect shader

Help Shaders
, ,
1

Godot Version

4.7.1.stable

Question

So, I’m working on a compositor effect shader, I’ve posted here before about it, and I’ve finished the hardest part of it. There is just one problem now, that I can’t even begin working out. I’ve made an MRP for the previous post, and I’ll link it here so anyone can tell me if I did anything wrong, but I’ll still try and explain my issue as best I can, regardless.

I’m making a datamosh shader. It’s a kind of mpeg corruption that sometimes happens, and it’s a great effect for glitchy scenes. However, I want it to apply only to some objects in the scene, and I did that by adding a second camera (per viewport), and only allowing that camera to see some objects in the scene, using rendering layers for that.

Now, the issue itself: I want the affected objects to still occlude behind the other geometry in the level, and to achieve that, I’m getting the depth buffer of both cameras, and comparing them with vec4(vec3(float(depth < maskdepth)), 1.0) (I also tried step(maskdepth, depth), and passing that as a mask to mix() (I’m not casting to vec4 in mix(), but I’m outputting it to screen like that).

The issue is that, instead of it outputting the expected result, a mask where it’s black by default, and white when the effect is supposed to trigger, it’s outputting a black mask with random mesh faces as white, and highlighting the faces with Z fighting when I move the camera.

This is the scene normally:

image
image1833×760 239 KB

The icosphere rotates along 4 randomized axis.

With the bugged shader, it looks like this:

image
image1833×760 22.8 KB

I can see the depth textures work exactly as intended:

Normal scene:

image
image1833×760 144 KB

The main depth:

image
image1833×760 22.3 KB

The mask depth:

image
image1833×760 19.1 KB

The masks are working exactly as intended. They are outputting only what they can see, as described. The problem is when I combine them, it looks all buggy, and only really shows up when I move the camera.

Am I doing something wrong? Is my GPU damaged and can’t do boolean anymore? What the hecc is going on here?

2

Don’t render the affected object by both cameras.

3

Tried that, did nothing. The effect relies on the target object being visible on both cameras anyways, so I should be able to do it regardless of object visibility to the main camera.

4

“Relies” in what way?

Are camera near/far ranges exactly equal?

You can try using <= instead of <.

5

If you remove the depth masks from the equation, you can still see a little bit of the actual mesh on the outline of the affected mesh, which gets carried away to the rest of the mesh.

image
image1833×760 323 KB

I also tried <= and >=. Didn’t work.

6

I still don’t get why the object needs to be rendered by the “background” camera. Assign the affected and non-affected objects to their respective cameras and do a standard depth composite after the effect is applied to the affected objects.

7

I tried doing that, and it didn’t work. At this point, I’d take a solution that occluded the object in the main camera.

And also, that wouldn’t work in the editor, because I can’t exactly change the bit mask in the editor camera (although, I assume you can do that with code, it’d just not be very intuitive to teach that).

8

A friend suggested I added a video to the topic, so

9

By depth composite, do you mean step(affected_depth, background_depth);? I did that and it resulted in the error I’m facing.

I was also informed that this is a perfect use case for stencils, but what I got from the very scarce data on this (I should really start making tutorials for the things I find, this field of graphics programming is really undocumented, but I digress), it isn’t exactly easy to get the stencil texture onto a compositor effect. Or at least that’s what the 2 reddit posts I could find said…

Also, for the use case I have in mind, I’d rather go with the unoptimal approach, because it’s easier to implement (although I’m taking anything I can have here, if it means this effect even works to begin with)…

10

By depth composite I meant - use the pixel that has the smaller depth. Normally in depth compositing there’s a strict division of objects between layers. The same object cannot be on two layers so there cannot be any z fights.

11

That is exactly what I am trying to do by using a step(). Even if an object is on both layers, the value should still return the same value every frame, which isn’t happening here?

Weird, though, because it was working just fine before I added the functionality of multiple editor viewports, as that is very important to this. I just can’t figure this one out.

Also, the error just changed shape randomly, so I have no idea of what could be going on anymore…

12

I have no idea what I changed in the shader, but it works now. Updates can be found in the MRP commit history.