Help with Top-Down 2D Tilemap Masking w/ Y-Sorting

Help Shaders
, , ,
1

Godot Version

Godot Engine v4.5.1.stable.steam [f62fdbde1]

Question

I have a top-down 2D strategy game that is using AnimatedSprites and TilemapLayers. I’m having trouble working out how to mask Tiles when a Unit is standing behind them, and was hoping anyone might be able to push me in the right direction.

ysortproblem1
ysortproblem1544×256 24 KB

Here I have a bunch of units standing in a forest. The Units are on layer 0, and so is the Tree TilemapLayer. Both have y-sorting enabled because I think it looks good, and the Tree tiles have a size of 1x2 in the atlas, so they are functionally one tile on the tilemap.

ysortproblem2
ysortproblem2544×256 24.7 KB

The problem arises when a unit stands behind them. They get obscured and it’s hard to see which unit it is without panning the cursor over them. If you’ve made a 2D game this is a very common problem, but since the tiles are in a TilemapLayer, and since they’re y-sorted, this has become a bit of an issue for me. The easiest and most practical solution is to simply have all tiles that a unit can stand behind be stumps - like the one below the red zombie on the right, but these tree’s are destructible, so a player could theoretically burrow into the forest and still have this problem. I could also simply discard the muti-tile-sized-tiles, but I like how it looks with the tall trees as opposed to 1x1 sized trees.

ysortproblem3
ysortproblem3544×256 25.8 KB

Additionally, some units can fly, and stand on top of objects that are normally walls, so a scenario like this is possible: Where the unit is sorted to the front (thanks to y sorting) and you can still see the tree behind them. This is currently functional and doesn’t need any changes to make work.

ysortproblem4
ysortproblem4544×256 24.7 KB

How I’d like everything to work is something like this. Where the tile is masked out, and the unit is visible, but only when they’re behind the tile.

So the solution needs to fit three conditions:

  1. Tiles and units still have y-sorting enabled on them, so that the forest still looks cool and so units can stand in a line vertically and still look good
  2. Units can stand behind a 1-by-2 or X-by-Y sized tile and the tile is masked
  3. But if the Unit’s position is actually on the Tile’s origin the Unit stands on top of it

I’ve tried a couple of things:

  1. I followed this tutorial here that utilized the BackBufferCopy node to mask out the Tilemap with an image. This didn’t work for two reasons:
    1. All of these assets are on the same layer (because of y-sorting), so the masking-object ends up masking away the Unit asset as well
    2. It ends up masking the tile, regardless of if the unit is standing behind them or in front of them. This could in theory be solved with code that detects if the tile you’re standing on is the origin, and then only enabling the masking image if it isn’t.
      1. I could let the Units stand on Layer 1 and the Tilemap stand on Layer 0 and then the mask works, but I’d still be masking out the Tile behind them if they’re flying. And abruptly turning off the mask in those scenarios feels clunky and jarring. It’s not a clean solution.
  2. I tried changing the modulate of the tile using the get_cell_tile_data() method when a unit is behind them but:
    1. That changes the modulate of every instance of that tile on the map and
    2. That changes the transparency of the whole tile, not just the top of the tile like I want
      1. I could painstakingly have the tops of the tiles on an entirely separate tilemap layer and then when the tree is destroyed, delete the tile on that extra layer - but again it feels clunky.
  3. I’ve investigated a couple of other shader options but came up empty-handed.
    1. In Unity I’d use a multipass shader (I believe that’s the right term?) to write any tilemap on a visibility layer to an image, then take any unit on another visibility layer and stencil around them if they overlapped, then write that to an image to be rendered to the camera. I feel like the BackBufferCopy is sort of doing that, but if it’s capable of doing what I want, I’m not sure how.
    2. Godot has support for 3D stencil buffers, but afaik I can’t utilize that in 2D. Though if I’m wrong, that could be the immediate and best solution, as all I need is to be able to see the silhouette of the unit so I know what it is.

Any advice of potential solutions that I can investigate would be greatly appreciated! I’m getting the sense that I could be SOL and might have to compromise on either the visuals of the Tiles, or the readability of the Units, but I’m hoping someone might know of a silver bullet that can get me exactly what I’m looking for. Thanks!

2

Instead of creating the 1x2 tiles keep the tiles as 1x1 and create patterns from the trees. This way you can modulate only the top part of the tree while being able to position the pattern easily. You will need to configure the top part of the tile y-sort origin correctly or y-sorting may not work. Also, you can also mark those top tiles with some custom metadata to know if you need to modulate them or not.

You’ll also need to implement TileMapLayer._use_tile_data_runtime_update() and TileMapLayer._tile_data_runtime_update() to be able to correctly update the specific tile TileData so it won’t affect other ones. Remember to call TileMapLayer.notify_runtime_tile_data_update() to trigger an update.

4

Sorry for the post deletion, I thought of something after replying and wanted to make sure I was working with all the knowledge I could before continuing the discussion.

Thanks for letting me know how to properly use the use_tile_data_runtime_update part, that’s good to know in the future!

Unfortunately, I don’t think that the pattern based solution will work for me.

image

image425×455 3.42 KB

When each part of the tree is it’s own tile, it becomes impossible to layer them in a visually appealing way. Either the tree tops overwrite the tile below it, or the trunk overwrites the one above it. No mannor of tweaking the y-sort origin or the texture origin gets around this either unfortunately. If the tree tops are it’s own tile, it’s always going to behave like this.

image

197×353 9.43 KB

And spacing the tree’s out so that it doesn’t happen is a problem too, because then the area between them looks like it’s an area a unit can move into, when it shouldn’t be.

5

Coming in with an update! I got it working via the use_tile_data_runtime_update system mrcdk pointed out to me.

image
image861×705 14.9 KB

Just in case someone else is having the same issue, here’s some psudocode. The docs aren’t very clear on how you’re actually supposed to use those methods, or how to make a change over time, so hopefully this helps someone.

In game, when a Unit updates their position on the map they check if the tile they’re on is obscured via an internal system of marking hidden tiles when the map is instantiated. If it is obscured, then RegisterTile is called, and when they leave that tile, DeregisterTile is called. Then ObscurableTilemapLayer handles modifying the alpha of the Tile for me.

extends TileMapLayer
class_name ObscurableTilemapLayer

@export var transparencyLimit : float = 0.35
@export var transparencyStep : float = 0.1


# Dict of Vector2i to internal Tile class
var stack_enter : Dictionary
var stack_exit : Dictionary


func RegisterTile(_tile : Tile):
	if _tile != null && !stack_enter.has(_tile.Position):
		stack_enter[_tile.Position] = _tile
		notify_runtime_tile_data_update()

func DeregisterTile(_tile : Tile):
	if _tile != null && !stack_exit.has(_tile.Position):
		if stack_enter.has(_tile.Position):
			stack_enter.erase(_tile.Position)
		stack_exit[_tile.Position] = _tile
		notify_runtime_tile_data_update()

# Update tile
func _tile_data_runtime_update(coords: Vector2i, tile_data: TileData):
    # PSUDOCODE: REPLACE WITH WHAT YOU NEED, THIS IS JUST A GUIDELINE
	# The tile in the stack dictionaries refers to the tile that is obscuring, not the tile the unit is currently on
	if stack_enter.has(coords):
		## Change the tile's modulate value by decreasing the alpha to hide it
	elif stack_exit.has(coords):
		## Change the tile's modulate value by increasing the alpha to show it again
        ## Make sure that you remove the tile from the stack_exit list if alpha is 1 or else you're going to have performance issues
        if foo_alpha == 1:
           stack_exit.erase(coords)


	notify_runtime_tile_data_update()
	pass

# In physics so it isn't affected by framerate. Could also save the delta in a variable to use in the tile update, but this works.
func _physics_process(_delta: float):
	if stack_exit.size() != 0 || stack_enter.size() != 0:
		notify_runtime_tile_data_update()

## Returns: Should tile at coords be updated?
func _use_tile_data_runtime_update(coords: Vector2i):
	return stack_enter.has(coords) || stack_exit.has(coords)

My initial attempt was to have a specific shader material attached to those tiles to allow me to only change the top half of the tile, but because of the bug linked below it just doesn’t work. Hopefully in a future update it will and I can use that shader, but for now this will work I suppose.