PackedFloat32Array has 0 AND multiple elements after RPC call

Godot Version

v4.3.dev6.official [89850d553]

Question

Hello! I’m trying to implement a voice chat (peer-to-peer). Since I’m new to Multiplayer and thus Voice Chat, I used a Godot 3 Voice Chat plugin and adjusted it to Godot 4. The microphone and everything works, even sending the packets (because both clients receive them). However, there is no audio playing. I’ve narrowed it down to what is likely causing the issue. When someone speaks, the voice data is sent to the other clients as a PackedFloat32Array. However, the audio can’t be processed, as for some reason, the array has a size of 0 when reaching the function that processes the data. HOWEVER, when I check the array size in the function that receives the data instead, it keeps growing (currently unprocessed because… well. the processing function thinks there is nothing to process)

I put a print(_receive_buffer.size()) in both functions the output is the following
image_2024-05-20_013923462

when i move it to _process() it looks like this
image_2024-05-20_013935310

When checking it with breakpoints, it seems like the array size swaps between 0 and the big number of unprocessed packets every frame.

extends Node
class_name VoiceInstance

signal received_voice_data
signal sent_voice_data

@export var custom_voice_audio_stream_player: NodePath
@export var recording: bool = false
@export var listen: bool = false
@export_range(0.0, 1.0) var input_threshold: float = 0.005

var _mic: VoiceMic
var _voice
var _effect_capture: AudioEffectCapture
var _playback: AudioStreamGeneratorPlayback
@export var _receive_buffer := PackedFloat32Array()

var _prev_frame_recording = false
var iter = 0

		
func _process(delta: float) -> void:
	if is_multiplayer_authority():
		_process_mic()
	if _playback != null:
		_process_voice()


func _create_mic():
	_mic = VoiceMic.new()
	_mic.set_multiplayer_authority(get_multiplayer_authority())
	add_child(_mic)
	var record_bus_idx := AudioServer.get_bus_index("Record") #_mic.bus
	_effect_capture = AudioServer.get_bus_effect(record_bus_idx, 0)
	
	if _playback == null:
		_create_voice()

func _create_voice():
	if !custom_voice_audio_stream_player.is_empty():
		var player = get_node(custom_voice_audio_stream_player)
		if player != null:
			if player is AudioStreamPlayer || player is AudioStreamPlayer2D || player is AudioStreamPlayer3D:
				_voice = player
			else:
				push_error("node:'%s' is not any kind of AudioStreamPlayer!" % custom_voice_audio_stream_player)
		else:
			push_error("node:'%s' does not exist!" % custom_voice_audio_stream_player)
	else:
		_voice = AudioStreamPlayer.new()
		add_child(_voice)
	
	_voice.play()
	_playback = _voice.get_stream_playback()


@rpc ("any_peer", "call_remote", "unreliable")
func _speak(sample_data: PackedFloat32Array):
	#emit_signal("received_voice_data", sample_data)
	_receive_buffer.append_array(sample_data)
	

func _process_voice():
	for i in range(min(_playback.get_frames_available(), _receive_buffer.size())):
		_playback.push_frame(Vector2(_receive_buffer[0], _receive_buffer[0]))
		#_receive_buffer.remove_at(0)


func _process_mic():
	if recording:
		iter = iter + 1
		if iter == 120:
			_effect_capture.clear_buffer()
			iter = 0
		
		if _effect_capture == null:
			_create_mic()

		if _prev_frame_recording == false:
			_effect_capture.clear_buffer()
		
		var stereo_data: PackedVector2Array = _effect_capture.get_buffer(_effect_capture.get_frames_available())
		if stereo_data.size() > 0:
			var data = PackedFloat32Array()
			data.resize(stereo_data.size())

			var max_value := 0.0
			for i in range(stereo_data.size()):
				var value := (stereo_data[i].x + stereo_data[i].y) / 2.0
				max_value = max(value, max_value)
				data[i] = value
			if max_value < input_threshold:
				return

			if listen:
				_speak(data)
			_speak.rpc(data)
			#emit_signal("sent_voice_data", data)

	_prev_frame_recording = recording

extends AudioStreamPlayer
class_name VoiceMic

func _ready() -> void:
	# UNUSED!!!
	var current_number = 0
	while AudioServer.get_bus_index("VoiceMicRecorder" + str(current_number)) != -1:
		current_number += 1

	var bus_name = "VoiceMicRecorder" + str(current_number)
	var idx = AudioServer.bus_count

	AudioServer.add_bus(idx)
	AudioServer.set_bus_name(idx, bus_name)
	AudioServer.add_bus_effect(idx, AudioEffectCapture.new())
	# UNUSED!!!
	
	# I have a Record bus that sends to a muted bus, so that I don't hear myself 
	AudioServer.set_bus_send(idx, "Mute Bus")
	bus = "Record"

	stream = AudioStreamMicrophone.new()
	play()

I’m really at my wits’ end lmao

if you are running two instances of godot you will get both peers output in the console.

it could be that one peer is receiving data, the other isn’t. use:
print(multiplayer.get_unique_id(), " ", <get buffer size>)
to determine who is printing. or use an if statement to only allow one to print.

you need to use the index [i] to push frame and not [0]

Edit see next post

Thanks for your reply! I’m running it on two different machines and already did the print statement at some point previously. It sends the data correctly to the other client and technically also receives it correctly, as I can print out the data itself and only the other client would react when I spoke into the microphone. However, as soon as it’s past the point of receiving, the buffer size drops back to zero only to go up to the correct number in the next receive call.

The playback works completely fine like that, but it doesn’t get to the part where it plays back because it always thinks that the _received_buffer array is empty.

1 Like

This isn’t true you commented out the pop front
And you are only sending the playback frame 0 each time

_playback.push_frame(Vector2(_receive_buffer[0], _receive_buffer[0]))
#_receive_buffer.remove_at(0) # commented out

You should uncomment this, as it sends frame zero each time.

In the big picture this doesn’t matter. As long as there is data to feed the player that is all that matters. And data needs to accumulate over time. This is called the buffer period. In your case looks like 256 frames. And will take time to consume .

1 Like

Also be aware that the print function and breakpoints will also be for local and remote instances of the game objects. Unless you print/check the name or authority of the player instance in the print function or breaking.

Changed the script to

class_name NewAudioManager
extends Node

@onready var input: AudioStreamPlayer = $Input
var index: int
var effect: AudioEffectCapture
var playback: AudioStreamGeneratorPlayback
@export_range(0.0, 1.0) var input_threshold: float = 0.005
var receive_buffer := PackedFloat32Array()

@export var output_path: NodePath

var iter = 0


func setup_audio(id):
	set_multiplayer_authority(id)
	
	if is_multiplayer_authority():
		input.stream = AudioStreamMicrophone.new()
		input.play()
		index = AudioServer.get_bus_index("Record")
		effect = AudioServer.get_bus_effect(index, 0)

	playback = get_node(output_path).get_stream_playback()


func _process(delta):
	if is_multiplayer_authority():
		_process_mic()
	_process_voice()
	

func _process_mic():
	iter = iter + 1
	if iter == 120:
		effect.clear_buffer()
		iter = 0
	
	var stereo_data: PackedVector2Array = effect.get_buffer(effect.get_frames_available())
	if stereo_data.size() > 0:
		var data = PackedFloat32Array()
		data.resize(stereo_data.size())
		var max_amplitude: float = 0.0
		
		for i in range(stereo_data.size()):
			var value = (stereo_data[i].x + stereo_data[i].y) / 2
			max_amplitude = max(value, max_amplitude)
			data[i] = value
		
		if max_amplitude < input_threshold:
			return
		
		send_data.rpc(data)

func _process_voice():
	if receive_buffer.size() <= 0:
		return
	
	for i in range(min(playback.get_frames_available(), receive_buffer.size())):
		playback.push_frame(Vector2(receive_buffer[0], receive_buffer[0]))
		receive_buffer.remove_at(0)


@rpc("any_peer", "call_remote", "unreliable")
func send_data(data: PackedFloat32Array):
	receive_buffer.append_array(data)

It works now, however, I’m having a very weird crackling noise (probably because of my bandwidth, so I will look into this next)

EDIT: Sent this before the new script, not sure why the other post is shown first for me

I tried changing the 0 to i and also tried uncommenting it again. However, the problem is that the _receive_buffer.size() is always zero when the function calls it, so it doesn’t even get into the for-loop. I even tried it without the loop, which returned me an error (or didn’t do anything with a return when _receive_buffer.size() is zero), because there is nothing in the _receive_buffer when _process_voice() calls it.

Yesterday, I added a label to the player scene that just prints out the packets received, but now I checked remote, and both Labels were visible at the same time. Also, it was only changing the label text on the other player’s label, but the local label remains at 0 (checked on both machines). Both labels have the same Multiplayer Authority as the Player Instance. I will dig deeper into it and update it if I find anything.

Thanks for the help btw! :))

Sure!

So I think you can ignore the 0 data.

Basically this is what I think the setup is. Let’s say player 2 is talking, and we are watching host machine logs

# host machine
Player1
- newaudiomanager ( send mic )
# prints buffer size of 0, as it will not recieve audio and is not actively sending audio to itself.
Player2
- newaudiomanager (receive mic)
# plays received player audio, prints buffer size > 0
# client machine
Player1
- newaudiomanager (receive mic)
# prints 0 as player one isnt speaking, and thus this will not recieve RPC audio.
Player2
- newaudiomanager (send mic)
# data is sent from here. Will also print 0 as it's not sending data to itself. Because RPC is "call remote."

Because there are multiple instances of newaudiomanager, you will see print logs from both instances on one machine.

Looking at your updated code there could be some auth issues happening if it is not set correctly. I don’t see how auth is set.

You can catch the right instance that has a buffer size like this

if receive_buffer.size() > 0:
  breakpoint 

This will break in the code without putting the red dot on script editor screen. Then you can check why audio isn’t making it to the player.