Resync node properties on inactive client

Godot Version

4.4.1

Question

I’m making an online multiplayer browser game with a dedicated server.
I have a Ball scene with a MultiplayerSynchronizer where I sync the position and linear_velocity of the ball only on spawn, so the server decides where the ball spawns and how fast it goes, and then the client simulates the physics.
This works well enough for my use case but there is a problem: when the player switches tabs and then comes back, all the balls on the client are at their original spawn location, since the client wasn’t active and didn’t simulate the ball movement while the tab wasn’t visible. This desync eventually fixes itself since the server will call to free the balls that exited a certain area, but it also look bad because while the server frees the balls off screen, on the client they will be freed while still being on screen, since they didn’t simulate the full movement, so the player will see some balls disappear.

Here’s a showcase of the desync, on the right is the server and on the left is the client.

Now for my question: is there a way to properly resync the balls position and velocity when the tab is visible again? I already have a way to check when the tab becomes visible, so I only need the actual code to do the resync.
For now I have tried this approach:

func on_tab_visible():
	send_balls_state.rpc_id(1)


@rpc("any_peer", "reliable")
func send_balls_state():
	var peer_id = multiplayer.get_remote_sender_id()
	for ball: Ball in balls_node.get_children():
		sync_ball.rpc_id(peer_id, ball.name, ball.position, ball.linear_velocity)


@rpc("reliable")
func sync_ball(ball_name: String, ball_position: Vector2, ball_velocity: Vector2):
	var ball := balls_node.get_node_or_null(ball_name) as Ball
	if ball:
		ball.position = ball_position
		ball.linear_velocity = ball_velocity

So here the client asks the server to send the position and velocity of all the balls and then those properties get set on the client. The problem is this still doesn’t work, it looks like the balls immediately snap back to their wrong position when I try to change it.

All the balls have multiplayer authority 1 (the server). I thought that was the problem and I tried to change the authority to the client before setting the position and the velocity but that still didn’t work. I also tried to turn off the visibility for that peer before syncing, but still no luck.
This is the code with those changes:

@rpc("any_peer", "reliable")
func send_balls_state():
	var peer_id = multiplayer.get_remote_sender_id()
	for ball: Ball in balls_node.get_children():
		ball.get_node("MultiplayerSynchronizer").set_visibility_for(peer_id, false)
		sync_ball.rpc_id(peer_id, ball.name, ball.position, ball.linear_velocity)


@rpc("reliable")
func sync_ball(ball_name: String, ball_position: Vector2, ball_velocity: Vector2):
	var ball := balls_node.get_node(ball_name) as Ball
	if ball:
		ball.set_multiplayer_authority(multiplayer.get_unique_id())
		ball.position = ball_position
		ball.linear_velocity = ball_velocity

As I was writing this I came up with a solution where on the client I just hide the original desynced balls and spawn new ones with the correct parameters. For the new balls I used a new scene that is a modified version of the original ball scene without MultiplayerSynchronizer and other small things to make it work.
This solution seems to work fine and it gives me no errors in the console, but still, it seems a little hacky and I was wondering if there was a simpler, cleaner solution.
Thank you in advance for trying to help.