Translate mouse movement and inputs to a virtual screen

[mrm1779]

Godot 4.2.1

Hey all,

I’m trying to create a computer simulation and am struggling to translate mouse movements and input to the virtual screen. Is this more of a geometry question?

The player window is 1920x1080, and the virtual window is 320x180.
The virtual window is at Vector3(0,0,0) and the camera is at Vector3(0,0,2.721)

I figured out how to get control nodes to render on a sprite3D through a viewport node. For the mouse movement, I took a crack at it, but it wasn’t quite right.

I have a scene called gui_panel_3d that looks like this:

The gui_panel_3d has a script on it:

extends Node3D

#func _input(event):
	#if event is InputEventMouseMotion:
		#var position2D = get_viewport().get_mouse_position()
		#%mouse.position = position2D / 6
		
	#if Input.is_action_pressed("click"):
		#var position2D = get_viewport().get_mouse_position()
		##var dropPlane  = Plane(Vector3(0, 0, 10))
		##var position3D = dropPlane.intersects_ray(%Camera.project_ray_origin(position2D),%Camera.project_ray_normal(position2D))
		##print(position3D)
		#
		#%Label.text = "this is a test"

# Define the size of the virtual screen
const VIRTUAL_SCREEN_SIZE = Vector2(320, 180)

# Reference to the 3D camera and the virtual screen node
var camera
@onready var virtual_screen = self

func _ready() -> void:
	#Input.set_mouse_mode(Input.MOUSE_MODE_HIDDEN)
	camera = get_parent().get_node("Camera")

# Function to map screen position to virtual screen position
func screen_to_virtual_screen(screen_pos):
	# Cast a ray from the screen position into the 3D world
	var ray_origin = camera.project_ray_origin(screen_pos)
	var ray_direction = camera.project_ray_normal(screen_pos)

	# Define the distance to the virtual screen
	var distance_to_virtual_screen = 5.0 # Adjust based on your setup

	# Calculate the hit position in 3D space
	var hit_position = ray_origin + ray_direction * distance_to_virtual_screen

	# Transform the hit position to the local space of the virtual screen
	var local_hit_position = virtual_screen.to_local(hit_position)

	# Map the local position to the virtual screen coordinates
	var virtual_x = (local_hit_position.x / virtual_screen.scale.x) * VIRTUAL_SCREEN_SIZE.x
	var virtual_y = (local_hit_position.y / virtual_screen.scale.y) * VIRTUAL_SCREEN_SIZE.y

	return Vector2(virtual_x, virtual_y)

func _process(_delta):
        var mouse_pos = get_viewport().get_mouse_position()
        var virtual_screen_pos = screen_to_virtual_screen(mouse_pos)
        #%mouse.position = virtual_screen_pos / 6
        %mouse.position = virtual_screen_pos

Then the gui_panel_3d is instanced in a 3d scene with a camera:

This is what it looks like when I move my mouse: the image moving around is the placeholder for a virtual mouse.

Also, here are my errors, just in case that means anything:

EDIT: I figured out a way to check if the virtual mouse is within the boundaries of a UI element:

func _input(event):
	if Input.is_action_pressed("click"):
		print(is_point_within_area(virtual_screen_pos, %Button.position, %Button.size))

func is_point_within_area(point: Vector2, area_position: Vector2, area_size: Vector2) -> bool:
	return point.x >= area_position.x and point.x <= area_position.x + area_size.x and \
			point.y >= area_position.y and point.y <= area_position.y + area_size.y

[eviltrout]

It sounds like you want push_input on your viewport.

Viewport

You can find unhandled input in your 3D environment, and the ray cast is within the bounds of your screen push the event there at the appropriate X, Y coordinate.

[mrm1779]

Yup, thats exactly what I wanted. Thank you!

[mrm1779]

Hey, sorry to re-open this, but I just realized the solution you gave me works great if the game screen and the virtual screen are the same size. My player screen is 1920x1080, and my virtual screen is 1600x1200.

I can reach the top buttons, but if I try to click on the start button at the bottom, then technically, my real cursor is outside of the game and will de-focus on the game window. Normally, I set the mouse to confined, but the virtual cursor can never reach the start button.

I need to scale the input somehow, but I’m unsure.

[eviltrout]

Yes, you’re going to need to translate the 3D coordinates to the 2D local coordinates before you push_input, then use the second in_local_coords parameter.

This will involve some math and likely some raycasting. Figure out where the mouse pointer intersects your plane, and translate that to a Vector2 and pass that through, it should work.

[mrm1779]

If anyone is looking for the formula later, this seems to work.

#This script receives the input event from the original / main viewport
func _input(event):
	if event is InputEventMouseMotion:
		var translated_position = translate_mouse_position(event.position, Vector2(1920, 1080), Vector2(1600, 1200))
		
		# Create a new InputEventMouseMotion with the translated position
		var new_event = InputEventMouseMotion.new()
		new_event.position = translated_position
		new_event.relative = event.relative
		new_event.velocity = event.velocity
		
		# Propagate the new event to the virtual screen
		%Viewport.push_input(new_event)

func translate_mouse_position(mouse_pos: Vector2, screen_size: Vector2, virtual_screen_size: Vector2) -> Vector2:
	var x_ratio = virtual_screen_size.x / screen_size.x
	var y_ratio = virtual_screen_size.y / screen_size.y
	return Vector2(mouse_pos.x * x_ratio, mouse_pos.y * y_ratio)

#I put this script on the second viewport that I want to send the input to, which becomes a texture for a sprite3D.
extends SubViewport

func _input(event: InputEvent) -> void:
	if event is InputEventMouseMotion:
		%mouse.position = event.position