Stream real-time Camera3D feed through UDP connections

Help
,
1

Godot Version

4.3

Question

I’m working on a project where I need to implement a Camera3D feed over a UDP connection for further experiments. Additionally, I want to enable real-time streaming features, including the movement of objects in front of the camera.

Currently, I’m experiencing a drop in framerate when connecting from a Python client, particularly due to moving objects in the scene.

Here’s my GDScript for the main scene:

extends Node3D

@onready var camera_3d: Camera3D = $Camera3D
var server: UDPServer = UDPServer.new()
var peers: Array[PacketPeerUDP] = []

func _ready() -> void:
	server.listen(4242) 
	print("UDP Server is listening on port 4242")

func _process(delta: float) -> void:
	server.poll()
	
	if server.is_connection_available():
		var peer: PacketPeerUDP = server.take_connection()
		peers.append(peer)
		print("Accepted connection from: %s:%s" % [peer.get_packet_ip(), peer.get_packet_port()])
	
	if peers.size() > 0:
		send_camera_feed()

func send_camera_feed():
	var image: Image = camera_3d.get_viewport().get_texture().get_image()
	var byte_array = image.save_png_to_buffer()  
	
	var chunk_size = 8192 
	for i in range(0, byte_array.size(), chunk_size):
		var chunk = byte_array.slice(i, i + chunk_size)
		for peer in peers:
			peer.put_packet(chunk)

And here’s my Python client code:

import socket
import cv2
import numpy as np

def main():
    server_address = ('127.0.0.1', 4242)
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.settimeout(10)

    message = b'test'
    sock.sendto(message, server_address)

    received_data = bytearray()
    
    try:
        while True:
            data, _ = sock.recvfrom(8192)
            received_data.extend(data)

            if len(data) < 8192:
                break

        if len(received_data) > 0:
            np_arr = np.frombuffer(received_data, np.uint8)
            img = cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
            if img is not None:
                cv2.imshow('Received Camera Feed', img)
                cv2.waitKey(0)
            else:
                print("Failed to decode image.")
        else:
            print("No data received.")

    except Exception as e:
        print(f'An error occurred: {e}')

    finally:
        sock.close()
        cv2.destroyAllWindows()

if __name__ == "__main__":
    main()

Can anyone suggest ways to improve real-time streaming without experiencing framerate loss?

2

The framerate drop happens inside the Godot-Scene? Or does the stream lag?

3

Inside Godot scene, stream in python side is just frozen or low framerate

4

You can use the Profiler inside godot to see where the framerate drops are coming from

5

Its more optimization of image buffer problem. Now I figured out, I need to optimize image buffer but with native extensions maybe Rust and I can use computing shaders to split image into chunks.

Maybe I need to move UDP processing to native extension too

6

If peformance is that important i strongly recommend to use a low-level language like rust or c++