CharacterBody2D gets stuck on a wall when it should change direction

Godot Version

4.2.1

Question

I have an enemy which is a CharacterBody2D and it should change direction when colliding with a wall or detecting a ledge. However, the enemy gets stuck as soon as it collides with a wall and is flipping rapidly in one place. What’s interesting, is that the script works for other two enemies that extend from the same script but not for this particular one. Here’s my Enemy code:

extends CharacterBody2D

class_name Enemy

var direction = Vector2.RIGHT

@onready var animated_sprite_2d = $AnimatedSprite2D as AnimatedSprite2D
@onready var ledge_check_left = $LedgeCheckLeft as RayCast2D
@onready var ledge_check_right = $LedgeCheckRight as RayCast2D

func _physics_process(_delta):
	if is_on_wall() || !ledge_check_left.is_colliding() || !ledge_check_right.is_colliding():
		direction *= -1

	animated_sprite_2d.flip_h = direction.x < 0

	velocity = direction * 70
	move_and_slide()

If colliding with a wall or the ray is not colliding or the other ray is not colliding, reverse the direction.
Isn’t one of those rays always going to be not colliding, thus making the direction always flip? How is this ever working correctly?