Most efficient way to transfer an image from js into gdscript?

Help
, ,
1

Godot Version

4.3

Question

I’m wondering what the most efficient way is to transfer image data between javascript and a godot web game, running in the same browser context.

I’m building some AR mini-games with web export so I can use webcam and mediapipe for human segmentation and pose detection (ie, people will move around and interact with a game scene). I’ve built this a few times without godot, but I think godot would make the game part a lot easier.

I have everything working, but image data is crazy slow (human keypoint data is small and easy to trasmit). I’m looking for suggestions on the most efficient way to transmit the data to Godot and load it into a Godot Image. In my ideal world I’d share WebGL textures between the two, but that seems unrealistic based on the code, so I’m just looking for few-copy sharing with no loops.

I can get the images back as a wrapped js Uint8Array or Float32Array, but then I don’t have a quick way to transmit that into an Image.

Things I’ve tried:

  • Copy form the js array into a Godot ArrayBuffer in gdscript, but that’s a slow loop.
  • Send as raw pixels encoded as base64, but that ends up slow on the gdscript base64_to_raw call. I could have less data if I encoded it as a PNG, but that seems silly for within-session communication.

Any suggestions on the most efficient way to do this?

2

Are you familiar with WebAssembly and emscripten?

webassembly: Practical WebAssembly - Conciso GmbH
emscripten: Interacting with code — Emscripten 3.1.73-git (dev) documentation

const wasmMemory = Module._malloc(imageData.length);
Module.HEAPU8.set(imageData, wasmMemory);

I’m not familiar with Godot-Web yet.
I asked Chat GPT how to connect it. Haven’t tried yet whether this example actually works.

ChatGPT suggestion

JS:

const imageData = new Uint8Array([/* pixel data */]);
const wasmMemory = Module._malloc(imageData.length);
Module.HEAPU8.set(imageData, wasmMemory);
GodotRuntime.call('process_image_data', wasmMemory, imageData.length);

GDScript:

func process_image_data(buffer_ptr, buffer_len):
    var byte_array = PoolByteArray()
    byte_array.resize(buffer_len)
    var byte_array_data = byte_array.write()
    for i in range(buffer_len):
        byte_array_data[i] = buffer_ptr[i]
    byte_array_data.release()

    var image = Image()
    image.create_from_data(width, height, false, Image.FORMAT_RGBA8, byte_array)
    var texture = ImageTexture.create_from_image(image)
3

I haven’t tried emscripten but am familiar with the concept. But the problem is convincing Godot to accept the data format, moreso than making my own code efficient. Any idea if godot web will accept the emscripten data structure?

I did find a more successful approach after I posted:

Before, I was using javascript-to-Godot callbacks to pass in the js TypedArray objects, which gdscript can’t interpret directly (you can access individual values but not pass the whole thing to Image.create_from_array). But if you call JavascriptBridge.eval(some_js_array_variable), that will return it as a gdscript PackedArray, which you can use efficiently.

So the fix was to stash the big array in some variable rather than passing it via a callback, and then using JavascriptBridge.eval(that_variable) to get it out.

I think this as few copies as I can get away with and no indexed array accesses, so probably the best I can do, though ugly.

1 Like