I have an animated mesh with custom normals and I’m using dot(normal, light) for basic cel shading. The problem is that because animation rotates the normals, lit/unlit areas that I intentionally created get broken up. Is there a way to disable normal transforms in skeleton shader? Or another way to achieve this?
I think either the way you’re applying normals (if it’s done through a shader) or more likely the cel shader is affecting the wrong space, by space I mean local space or world space.
I’m pretty sure that it’ll be the cel shader itself, and you just need to switch the normal function either from affecting world normals to local, or vice-versa.
Off hand I can’t remember how to actually switch the space in shader code, but I’m pretty sure it’s the answer you’re looking for.
As far as I understand, normals in vertex shader are either in local space or in world space. In either case, bone transforms are already applied and can’t be skipped in a spatial shader.
The solution I found was to bake custom normals to a normal map in object space, sample in vertex shader, transform to world space, and do lighting in world space in fragment shader. I can also use vertex color to control what parts of the mesh should use which normal (vertex normal with bone transforms or normals from the texture).