Custom Vertex Shader Impact on Shadows

Hello everyone,

I am aiming to replicate the effect provided by the Unity Asset Store plugin named Curved World from the link Curved World | VFX Shaders | Unity Asset Store. Given my less than proficient grasp of mathematics, realizing this same effect within the Godot game engine is proving to be highly problematic for me.

I am working on a project using Godot engine and have been experimenting with a custom vertex shader to achieve a specific vertex rotation effect where the model rotates around an axis perpendicular to the world Y-axis and formed by the line connecting the camera position and the model’s center. Here’s my vertex shader code snippet:

shader_type spatial;

vec3 rotated(vec3 original, vec3 vertex, vec3 axis, float angle) {
    vec3 temp = vertex - original;
    vec3 temp2 = cross(axis, temp);
    return cos(angle) * temp + sin(angle) * temp2 + dot(axis, temp) * (1.0 - cos(angle)) * axis + original;
}

void vertex() {
    vec3 axis = cross(normalize(NODE_POSITION_WORLD + VERTEX - CAMERA_POSITION_WORLD), vec3(0.0, 1.0, 0.0));
    float angle = -0.1;
    vec3 vertex = VERTEX;
    VERTEX = rotated(CAMERA_POSITION_WORLD, vertex, axis, angle);
}

The issue I’m facing now is that after applying this vertex shader to the model, its shadows are displaying abnormally. Specifically


I greatly appreciate your assistance!

Well, this may be caused by a normal error.

1 Like

Yea normals are tricky, how I would approach this would be to use the MODEL_MATRIX.

Some one did this not to long ago.

I think if you can modify the matrix uniforms you wouldn’t need to recalculate the normals.

https://en.m.wikibooks.org/wiki/GLSL_Programming/Vertex_Transformations

This is incomplete but you can create a transform matrix and apply it to bothe the model view and normalview.

shader type spatial;

uniform mat4 mod_matrix mat4(0.0);

void vertex() {


mat4 rotated_model = VIEW_MATRIX *(MODEL_MATRIX+mod_matrix);
MODELVIEW_MATRIX = rotated_model;

MODELVIEW_NORMAL_MATRIX = mat3(rotated_model);

}


Appreciate your response, though it seems that my original intention was not fully grasped. In fact, I am aiming to replicate the effect provided by the Unity Asset Store plugin named Curved World from the link Curved World | VFX Shaders | Unity Asset Store. Given my less than proficient grasp of mathematics, realizing this same effect within the Godot game engine is proving to be highly problematic for me.

Oh okay, I didn’t get to look at it because it wants 3rd party cookies enabled. So I just looked at your math.

Anyway looking at the description this seems like my approach could still apply… do you have a mathematical algorithm for reference? I guess this would be considered a non-linenear transformation. So I guess it’s a tossup if you want to really change the normals depending on the outcome.

Also would you pay me $20 bucks if I could cheese this out? :wink:

It’s hard to tell what those dark shards are, you say shadows are displaying abnormally, are there objects to the left of the screen casting shadows on that white mesh, or are you talking about the shading on the white mesh being abnormal?

There is only one light and meshinstance in the scene.

The average hourly wage in the city where I live is only two dollars,I don’t even have a job now, which is too expensive🙁

In 4.3, you can write to LIGHT_VERTEX when you need to correct shadow map casting: Add `LIGHT_VERTEX` to fragment shader by basicer · Pull Request #91136 · godotengine/godot · GitHub

This isn’t necessarily the correct solution to the problem outlined here, but it’s worth keeping in mind. LIGHT_VERTEX is mostly designed for things like raymarching shaders (that draw 3D-looking geometry without using triangles/lines/points).

Yeah, just trying to understand if this is shadows or shading, you can test this by toggling off cast shadow on the existing light, or move the light around and see if you see the the shapes move with the light.

You can also show the wireframe on the mesh to see if those dark shards that show up are following your geometry or something else.

This looks more like an issue with the vertex shader to me. The vertex shader is changing the vertex position based on the camera position. But when rendering shadows the camera position is the position of the light. So the shadows end up completely messed up.

Instead of CAMERA_POSITION_WORLD, you should try using MAIN_CAM_INV_VIEW_MATRIX[3].xyz

He is trying to modify the mesh, so he would have to recalculate the normals. But to do so correctly is a little involved.

This can be done if you take the average of two cross products with a vertex as the origin. The hard part is querying the modified mesh to find valid points along the mesh surface. You would need to select the points through mathematical mean that described the curvature to calculate the new normal of the face.

But from your example, you probably don’t have too depending on the outcome you want.

But after reading your code, I don’t understand

temp = local space vertex position - camera world origin

Not sure that makes sense to me.

Anyway I think he is after this?

You can also disable shadows in the shader.

shadows_disabled to test if it’s a shadow glitch because of the vertex manipulation.

Idk if this will apply.

I was playing around on the web editor (in compatibly mode), I saw that during concave angle the planemesh was casting a shadow on itself and neighboring meshes from some invisible ghost position.

I found I could control it by modifying the depth.

And by accident I somehow disabled shadow casting on from the mesh by just sampling the depth texture, but not doing anything with the value. And it still received shadows fine.

1 Like