Check my Work: Cutout shader (Dim the screen EXCEPT for one section)

Help
,
1

Godot Version

4.3rc3

Question

Goal: Dim the whole screen except for one region (an area i want to highlight).

Task: Given BigTexture and LittleObjectOnScreen, draw BigTexture except where it overlaps LittleObjectOnScreen.

Questions:

  1. i have a solution. It’s probably horrible. Suggestions on improvement?
  2. My solution only works for a rectangle. i’d prefer to pass in a texture/mask. Is that possible (and if so, how?)

Here’s what i have. Note that the shader is on a fullscreen ColorRect which is on a CanvasLayer that’s one layer higher than the UI’s CanvasLayer so that the mask covers the UI.

GDScript:

func highlight(object):
	var top_left     = global_to_uv(object.global_position)
	var bottom_right = global_to_uv(object.global_position + object.size)
	shade.material.set_shader_parameter("tlx", top_left.x)
	shade.material.set_shader_parameter("tly", top_left.y)
	shade.material.set_shader_parameter("brx", bottom_right.x)
	shade.material.set_shader_parameter("bry", bottom_right.y)

func global_to_uv(coord_global: Vector2) -> Vector2:
	return coord_global / get_viewport().get_visible_rect().size

Shader (i am so bad at shaders):

shader_type canvas_item;

uniform float tlx;
uniform float tly;
uniform float brx;
uniform float bry;

void fragment() {
	float alpha = COLOR.a;
	if ((UV.x >= tlx) && (UV.x <= brx) && (UV.y >= tly)  && (UV.y <= bry)) {
		alpha = 0.0;
	}
	COLOR.a = alpha;
}

That took me hours to write (and days of reading docs) and i’m guessing it’s dumb and there’s a shaderier way of doing it.

image
image799×596 96.8 KB

Q1. Is my code insane?

Q2. How would i get this to work with arbitrary shapes?

i originally passed in a mask texture but i couldn’t get the texture to keep its size, making the mask useless.

2

Hmm… I don’t think your code is “insane” - just very specific. A more generic approach, such as the mask texture you mention, is probably the way to go.

There’s a great video about how to make screen transitions based on a gradient mask. The solution in the video is made in Unity but the general concept(s) will work in Godot as well. Although, you would need to figure out how to grab/create a bitmask for your items.

That said, I’m not sure you need to use a shader to achieve your desired result. You should just be able to alter the z_index of the items you want to remain fully “lit”. Having a semi-opaque black box (with a lower z-index) behind your “lit” items would produce the same look as seen in your example picture.

I hope this gives an overview of your options. If it doesn’t work, feel free to ask any questions.

3

i love simple but…

It looks like z_index only works within a layer. This doesn’t work:

  • UI Layer (CanvasLayer)
    – Button A
  • Game objects
  • Tutorial Layer
    – Shade (CanvasLayer)

UI Layer draws above game objects because it’s in a CanvasLayer and Tutorial Layer draws above UI Layer either because it has a higher Layer number or because it’s at the bottom of the scene tree. Nothing in UI Layer draws over Tutorial Layer no matter what the z_index, even when both layers have the same Layer number. If i move the Shade object to UI Layer then the z_index matters. So layer > z_index. So for this to work i have to merge all my tutorial layer into the UI layer. Which isn’t the end of the world, i just have a ton of objects and Tutorial Layer only exists on one map (the tutorial, obviously) so i liked having it separate. But i love the idea of something as simple as a z_index (i am painfully clueless and bumbling with shaders).

Do you happen to know of a way to have z_index work across layers?

4

That video is pretty cool. It does a lot with a texture. Unfortunately, it involves creating a full screen-sized texture for each specific case which for me would be one per button to highlight which is the same amount of work as manually creating a set of overlay shade images. If i want to do generic animated transitions i can reuse across screens, however, that video has some great ideas.