Help figuring out how to do a colour quantisation and dithering shader

[Indee]

Godot Version

Godot 4.2.1

Question

I’m fairly new to shaders and all that, and I’ve figured out how to do a simple colour quantisation post-processing shader, and I wanted to try to make the shader dither the regions between the colour bands. Here’s what the regular colour quantisation looks like:

And here’s a few screenshots of the results I’ve been able to get from the dithering:

The first one is like the colours are negated, not sure how that happens, but that’s an older screenshot. The last two are what I’m up to now and have a very interesting pattern, for some reason the green and blue channels fade in from the left and top, like if you tried to colour a rect based off of UV coordinates. I have no clue why this happens but feel like it’s super important to answering why this doesn’t work. You can see however (easier to see in the last picture) that the dithering works just fine, the pattern is there, it’s just the colours being messed up.
Here’s a picture that describes what I want to happen:

Anyways, here is the shader:

shader_type canvas_item;

uniform bool dithering = false;
uniform float num_channels = 20.0f;
uniform sampler2D screen_texture : hint_screen_texture, repeat_disable, filter_nearest;

bool xor(bool a, bool b) {
	return (a || b) && !(a && b);
}

// This is the function that tries to apply the dither effect,
// read from the fragment function to get better context
float dither(float col, bool x_even, bool y_even) {
    // This line tries to determine if the colour is odd or even
	if(mod(col, 2.0f) < 0.99f) { // This line is the problem,
    // if I remove this line then I get the dithering effect,
    // but it's across the whole screen and not where I want it.
    // If I keep this line then it makes the weird trippy images you see above with all the green and red.
    // I've tried different methods of determining if it's odd or even, but that doesn't seem to matter.
		if(xor(x_even, y_even)) { // This just applies the pattern.
            // Since the colour is odd and in between two even proper colours,
            // if I add or minus one then it should become an even number and this a proper colour value,
            // this is all I'm trying to do, it will bump up some values based on the pattern,
            // and bump the rest down to the lower colour.
			return col + 1.0f;
		}
		else {
            // Whether or not this actually applies the effect on every even or odd pixel does NOT matter,
            // I care more about the effect working then if it is on every even pixel or not.
			return col - 1.0f;
		}
	}
}

// quantises the colour of each pixel, and optionally adds a dither effect to the quantisation
void fragment() {
	vec3 col = texture(screen_texture, SCREEN_UV).xyz;
	
	if(dithering) {
        // Quick explanation: What I'm trying to do is to double the number of colours
        // I'll be using and find which colours are odd, because if a colour is odd
        // then that means it is between two colours which I will actually have in the end.
		col *= (num_channels * 2.0f);
        // Find the pixels integer coordinates, and find if those coordinates are even
		bool x_even = mod(SCREEN_UV.x * float(textureSize(screen_texture, 0).x), 2.0f) < 0.99f;
		bool y_even = mod(SCREEN_UV.y * float(textureSize(screen_texture, 0).y), 2.0f) < 0.99f;
		
        // Here I round the colour value to the nearest integer to find which colour it belongs to
		col.x = round(col.x);
        // This just passes the colour in to be dithered
		col.x = dither(col.x, x_even, y_even);
        // This presses the colour back to the 0-1 range
		col.x = col.x / (num_channels * 2.0f);
		
        // Same thing as above for the other two channels
		col.y = round(col.y);
		col.y = dither(col.y, x_even, y_even);
		col.y = col.y / (num_channels * 2.0f);
		
		col.z = round(col.z);
		col.z = dither(col.z, x_even, y_even);
		col.z = col.z / (num_channels * 2.0f);
	}
	else {
		col *= num_channels;
		col.x = round(col.x) / num_channels;
		col.y = round(col.y) / num_channels;
		col.z = round(col.z) / num_channels;
	}
	
	COLOR.xyz = col;
}

[Indee]

Well, I managed to fix my problem by restarting from scratch and going about it in a different way. Basically instead of doubling the colours and finding values that are half way between the colour values I’d use, I instead just look for values that are within a certain range of the halfway points between colour values. This also has the added benefit of making the bands more customisable in width as you can adjust the size of the range. E.g. colours sit on integers, 1 and 2, and any value within a range of 1.5 would be dithered between the colours at 1 and 2, if the range is 0.2 then the colours between 1.3 and 1.7 would be dithered.

Here’s the full code:

shader_type canvas_item;

uniform float num_channels = 20.0f;
uniform bool dithering = false;
uniform float dithering_amount = 0.5f;
uniform sampler2D screen_texture : hint_screen_texture, repeat_disable, filter_nearest;

bool xor(bool a, bool b) {
	return (a || b) && !(a && b);
}


float dither(float col, bool x_true, bool y_true) {
	float channel = round(col);
	if(abs((floor(col) + 0.5f) - col) < dithering_amount || abs((ceil(col) - 0.5f) - col) < dithering_amount) {
		if(x_true && y_true) {
			return channel - 1.0f;
		}
		else {
			return channel;
		}
	}
	return channel;
}


// quantises the colour of each pixel, and optionally adds a dither effect to the quantisation
void fragment() {
	vec3 col = texture(screen_texture, SCREEN_UV).xyz;
	
	if(dithering) {
		bool x_true = mod(SCREEN_UV.x * float(textureSize(screen_texture, 0).x), 3.0f) < 0.99f;
		bool y_true = mod(SCREEN_UV.y * float(textureSize(screen_texture, 0).y), 3.0f) < 0.99f;
		
		col *= num_channels;
		
		col.x = dither(col.x, x_true, y_true);
		col.x = col.x / num_channels;
		
		col.y = dither(col.y, x_true, y_true);
		col.y = col.y / num_channels;
		
		col.z = dither(col.z, x_true, y_true);
		col.z = col.z / num_channels;
	}
	else {
		col *= num_channels;
		col.x = round(col.x) / num_channels;
		col.y = round(col.y) / num_channels;
		col.z = round(col.z) / num_channels;
	}
	
	COLOR.xyz = col;
}

Here you can see the effect, though it is light: