Issues with Camera2D position clamping after zooming (MagnifyingGesture & PanGesture)

Godot Version

4.2

Question

Hey everyobody. First post here so please let me know if I am doing something wrong.
I am currently working on something 2D top down and have a Camera2D node as child in my main scene. I want to clamp my camera position to avoid the “camera sticks to the limit” issue.
Works fine so far. However, if I zoom first and then pan afterwards (everything on mac touchpad). The camera position is jumping to the top left corner.

Limits of the camera are:
left: -100px
right: 2000px
bottom: 1100px
top: -100px
Viewport size: 1920 * 1080

My script is the following:

extends Camera2D

# Zoom settings
var min_zoom = Vector2(1, 1) # Maximum zoom-out level
var max_zoom = Vector2(5, 5) # Maximum zoom-in level
var zoom_speed = 0.5 # Speed of zooming in/out
@onready var viewport_size = get_viewport().size

func _input(event):
	if event is InputEventMagnifyGesture or event is InputEventPanGesture:
		if event is InputEventMagnifyGesture:
			var previous_zoom = zoom
			zoom.x += event.factor * zoom_speed - zoom_speed
			zoom.y += event.factor * zoom_speed - zoom_speed
			zoom.x = clamp(zoom.x, min_zoom.x, max_zoom.x)
			zoom.y = clamp(zoom.y, min_zoom.y, max_zoom.y)
		# Handle panning with touchpad
		if event is InputEventPanGesture:
			position -= event.delta * zoom *-1  # Using zoom to scale the pan speed according to the current zoom level
			# this works for only panning. but it doesn't work anymore if I zoom first. then it only zooms in the top left corner
			clamp_position()

## currently doesn't work properly when zoomed in
func clamp_position():
	viewport_size = get_viewport().size
	var x = zoom.x*viewport_size.x/2
	var y = zoom.y*viewport_size.y/2
	position.x = clamp(position.x, limit_left+x, limit_right-x)
	position.y = clamp(position.y, limit_top+y, limit_bottom-y)
	

func _process(delta):
	# moving camera with keyboard
	var InputX = int(Input.is_action_pressed("ui_right")) - int(Input.is_action_pressed("ui_left"))
	var InputY = int(Input.is_action_pressed("ui_down")) - int(Input.is_action_pressed("ui_up"))
	position.x = lerp(position.x, position.x + InputX* pan_speed*zoom.x, pan_speed*delta)
	position.y = lerp(position.y, position.y + InputY* pan_speed*zoom.y, pan_speed*delta)

I have tried around with positioning the call to “clamp position” at various points, but that doesn’t seem to do the trick. If anyone could point me to the right direction, that would be nice.
I have searched this forum and reddit, but have not found anything yet.