Raycast2D collides with hidden node

Godot 4.x

In the editor I have made the node with collision shape not visible, yet the raycast still finds it.
Is that what is supposed to happen?

Being not visible is different than being disabled. Try enabling the disabled property.