I was able to run a local test, and got upto 1600 stable clients, but had to rate limit the join rate to 2 clients per second. I think i could go further, but it takes some time to build the clients up.
I did see the disconnect issue, with almost all clients disconnecting with no join rate limiting, but can be reduced with join rate limiting. (I.e. using a timer to creat clients)
I was monitoring the packet peer packet loss statistic and for a single peer, it does build considerably as the client list grows. Although this number could get pretty high it did not correlate to disconnect rates. Max packet loss for first peer joined 17000.
changing ping interval didn’t seem to have as large of an effect as anticipated.
These test were performed on a single i7-8550U in Power mode. I had 3 processes, 1 host and 2 clients, using the branched multiplayer api trick to put multiple clients into one process.
After 1600 clients joined CPU usage for each client process sat at 13% CPU , 750MB mem. The server 2% cpu, 100MB mem.
Conclusion: i suspect that the single working channel in the Godot ENet implementation is congested and dropping packets for joiners. I think a custom implementation to spread clients onto available channels is the worth a try to hasten join rate.
Code
extends Node
const CLIENT_COUNT = 800
# 2 client proc (cp), 0.2 create rate (cr)
# pp.set_timeout(15000, 30000, 60000)
# pp.ping_interval(5000)
# 840 stabalized 10000+ packet loss
# 2 cp, 0.3 cr -> 888 instable, 12000 packet loss, (5000 ping interval) +timouts
# 2 cp, 0.6 cr -> 900 stable, 11000 packet loss, (5000 ping interval) +timouts
# 2 cp, 1.0 cr -> 1200 stable, 21000 packet loss, (5000 ping interval) +timouts
# 2 cp, 0.6 cr -> 900 stable, 13000 packet loss (500 ping interval) no to
# 3 cp, 0.9 cr -> ~1000 unstable, 17000 packet loss (500 ping interval) no to
# 3 cp, 1.2 cr -> ~1200 unstable, 21000 packet loss (500 ping interval) no to
# 3 cp, 1.5 cr -> unstable
# 2 cp, 1.0 cr -> 1600 stable, 17000 max packet loss (2000 ping interval) timouts, power mode
func _ready() -> void:
var peer : NetworkPeer
if OS.has_feature("server"):
peer = NetworkPeer.new()
add_child(peer)
else:
# stagger client rate
if OS.has_feature("client1"): await get_tree().create_timer(0.5).timeout
for i in CLIENT_COUNT:
peer = NetworkPeer.new()
add_child(peer)
await get_tree().create_timer(1.0).timeout
class NetworkPeer extends Node:
static var client_number : int = 0
const IP_ADDR := "localhost"
const PORT := 5555
var enet := ENetMultiplayerPeer.new()
func _ready() -> void:
var api := SceneMultiplayer.new()
if OS.has_feature("server"):
enet.create_server(PORT,4000)
self.name = "server"
api.peer_connected.connect(_on_peer_connected)
api.peer_disconnected.connect(_on_peer_disconnected)
api.server_relay = false # forgot to add this in the first run
else:
client_number += 1
enet.create_client(IP_ADDR, PORT)
self.name = "client" + str(client_number)
set_process(false)
api.multiplayer_peer = enet
get_tree().set_multiplayer(api, self.get_path())
var time:float = 0.0
var peer_id: int
func _process(delta: float) -> void:
time += delta
if time > 1.0:
time = 0
var p : ENetPacketPeer = multiplayer.multiplayer_peer.get_peer(peer_id)
var amount : float = p.get_statistic(ENetPacketPeer.PEER_PACKET_LOSS)
print(amount)
#if amount > 0.0:
#breakpoints
func _exit_tree() -> void:
multiplayer.multiplayer_peer.close()
multiplayer.multiplayer_peer = OfflineMultiplayerPeer.new()
func _on_peer_connected(id:int) -> void:
if not peer_id:
peer_id = id
var pp: ENetPacketPeer = enet.get_peer(id)
pp.set_timeout(15000, 30000, 60000)
pp.ping_interval(2000)
print(name, ": #%d peer connected %d" % [multiplayer.get_peers().size(), id])
func _on_peer_disconnected(id:int) -> void:
print(name, ": #%d disconnected peer %d" % [multiplayer.get_peers().size(), id])
update
I forgot to disable server relay, which helped immensely I was able to go 6 clients per second with almost no packet loss. CPU usage was only at 4%, and 100MB for clients. for (1600 total clients) Which means there is some array vector growing when there are a lot of packets piling up in a queue somewhere?
I do see another issue, which seems like the host crashes but nothing is logged..
