High speed collisions don't work for me

Godot Version

4.1.2.stable.mono

Question

I’m making a game, where you can control a space-ship.
The ship itself is RigidBody2D and I move it using apply_central_impulse so that ships can interact with each other on collision.
I tried to solve this by storing the difference in position of every ship every physics tick and then moving every entity by the delta position of the ship they are standing at using: move_and_collide(passenger_on.difference_in_position);

This almost works, but there’s an issue - the hitboxes of the entities on a moving ship are shifted/offset by the speed of the ship.

The red rect is the hitbox of the entities.
They shouldn’t be intersected.
image

Any idea why this happens and how could I prevent this?