(using Godot 4.7.0)
I’d like to achieve a visual effect in shadowing where, given a mesh which is being illuminated by lights which have shadow casting enabled, the shadow mapped onto the mesh consists only of fragments which are not self shadowing; in other words, only shadows which visually travel through space from one area on the mesh surface to another area will be displayed. So self shadowing is very much possible for the mesh as a whole, but not for it’s fragments being rendered.
Below is a visual explanation of what I’d like to achieve, using a mesh which is a sphere that has a small knob outset from it’s surface to show how shadows are being cast.
Image 1 - light has shadow casting disabled:
Image 2 - light has shadow casting enabled, using the usual shadow mapping process where fragments are shadowed by self and through space:
Image 3 - light has shadow casting enabled, but the shadow mapping process has been modified to exclude self shadowing of fragments (the effect here is what I want to achieve):
So in Image 3, the self shadowing of fragments facing away from the light has been excluded, but still a shadow is cast from the surface of the knob facing away from the light to the surface of the main sphere.
And this effect is only relevant within each individual mesh’s shadowing - a mesh would still be fully shadowed as usual by other meshes in the scene, assuming they cast shadows.
I have read about this a bit and from what I understand, Godot’s shadow mapping process is pretty hard coded, and the only way I would be able to practically directly modify how shadows are cast is to compile an edited version of source code that includes any changes I want, which is not something I would be comfortable with doing at present. So I would appreciate any ideas anyone has as to how I could achieve this effect without modifying source code.
What would be the actual use case for this?
Visual stylization. More specifically, I have a toon shader which lights a mesh such that it has a lighter portion and a darker portion, with the darker portion facing away from the light and emulating a shadow without the need to actually cast any shadows; combining that with this effect would make the character look fully shadowed while largely avoiding shadow artifacting (which is often very prevalent since the mesh is low poly).
Can we see those artifacts?
Here you can see what shadowing currently looks like on a human body mesh using the usual shadow mapping process (as in Image 2 above), with their head’s shadow being cast onto their left shoulder
But due to this self shadowing, artifacts appear, typically in the form of triangular patches of unwanted shadow (below, you can see shadow spikes on the back of the head and the upper left leg, which would not be present if excluding fragment self shadowing)
Adjust the shadow settings to minimize artifacts.
I have done that - the problem there is that the further the light’s shadow bias is from 0, the more the shadows don’t reach the edges and crevices that they should, and that’s regardless of whether fragments are being self shadowed.
It’s not only bias that’s relevant. Shadow map resolution, max distance and cascade splits can play the role.
Well yeah, I’ve already optimized all of those - directional shadow size is set to 16384 for maximum crispness, and the directional light is using adjusted PSSM 4 splits. The only way I can remove the artifacting through these settings is increasing the shadow bias, which again is guaranteed to cause the problem described in my previous reply. I don’t think the effect I’d like to achieve can be accomplished by simply tweaking settings in the editor; I was hoping something more programmatic could do the job, such as a trick involving builtin variables in the spatial shader light function, something like that.
What about the max distance? That’s a parameter that affects directional shadow quality to a very large degree.
In any case, what you want is not an “effect”. You’re just obsessing over getting “perfect” shadows while zoomed in on details. This is not going to happen with real time shadows. They will never be perfect.
It’s a waste of time to tweak shadows until you have the actual player model (colors, textures…), and a complete mockup of your typical game scene with all of the planned visual density in place, looked at from a typical camera angle. That’s the proper visual context to deal with shadow quality and minimization of eventual artifacts. It will always be a compromise.
What you’re currently doing is an equivalent of premature optimization.
This really looks like an XYproblem type of question. You already invented some imaginary semi-solution to a problem that may not even be a problem, and now you’re asking how to technically achieve that semi-solution. Technically, you cant cast shadows selectively in the way you described. Realtime shadow mapping simply doesn’t work like that.
