One other interesting thing. If you develop in Godot you should know that building a dedicated server app puts it in ‘low processor mode’ by default. Why?; who knows… I came across a github post on it which recommends setting ‘low processor mode sleep’ to 1. This had the effect of boosting the CPU usage for me which is desirable. I would say a fair amount, maybe 20-25% more CPU usage (VPS). Setting the sleep mode to 1 decreases forced wait times.
Vidiot_X
( video: Client player ship attached to Bot ship Melvin 39 on StarLink as we ride along. Testing out new network changes to server and client designed to better support the StarLink network. Working well.)
I have a gift for noticing patterns and I had noticed a pattern with StartLink. It seemed that packets sent over StarLink’s network did better if they were sent with less delay as relates to my dropout issue. So I asked the StarLink AI assistant the following:
… Does the StarLink system treat very low packet latency data streams differently from higher packet latency data streams? Will decreasing latency or delay between packets sent over StarLink increase stability, especially during dish to satellite hand-offs? …
To my surprise it had a good answer:
Starlink’s network is designed for low-latency applications like gaming, with typical latency of 25-60ms on land, and it dynamically switches between satellites in real-time to maintain connectivity during hand-offs, which can sometimes cause brief packet disruptions regardless of stream latency. While we don’t apply different treatment to low vs. higher latency streams, reducing packet intervals (as you’ve done) can improve stability by minimizing the impact of those transient handoffs—check our reliability page and specifications for more on performance, or submit a ticket via the app for deeper network insights.
So it turns out I did see a pattern. Testing revels even sending part of the packets at 25ms decrease the dropouts quit a lot. 30min no severe dropouts, then, one moderate. Part of building out a client server network is understating the hardware your working with and how to adapt to it.
I will be changing server to player client ship velocity updates to every 25ms. This increases the bandwidth slightly, but as a tradeoff I get much smoother more accurate player ship updates and a more stable StarLink connection. I will be testing this out more but it looks good so far, with any dropouts being much less severe, shorter in duration and less frequent. The trade off is packet loss during any dropout event. With certain packets sent more often (less latency) a degree of stability can be maintained during short high packet loss events. In my case velocity is a good choice as a foundation to help in stabilizing short high packet loss events.
More soon…
Vidiot_X
I am examining packet size and network history in this image. If you look at the WireShark app where it says ‘protocol’ You can see that the server to client data is DTLS encrypted. You can see the actual packet byte lengths or sizes at the column ‘Length’. Each packet is less than 1400 bytes in length which is the packet size limit for UDP Godot/Enet.
Basically I am making sure that there are no fragmented packets (split up in many packets). If a data send is to large it gets split up into many packets and if there is an issue (corrupt or out of order) Enet will disregard all of the packets associated with that data send. Not good and this leads to packet delivery issues, packet loss and more. So I am making sure I have no fragmented packet and everything is encrypted. Encryption is a huge deal and will prevent cheating along with other techniques.
More soon…
Vidiot_X
Big network and state smoothing changes have things running smoothly and glitch free. As seen in this development snapshot video. I have taken a new approach to the way I send data to clients from the server. The new ‘packet heavy’ approach to networking coupled new with state smoothing (smoothing position, velocity, etc) changes has made a huge difference. This new networking approach is more stable, robust and is able to handle packet loss and network dropouts very well.
This update video also includes a good look at all the recent changes including the background, lighting and more.
I encourage you to view the video. It shows how well, smooth and stable the new networking approach is. A flawless example of the performance you can expect. I also hope that you share this and other videos and post as it’s time to start letting others know about whats to come.
I will be working on all the other issues to get a client out for testing. The next thing is network load testing of the server. At first it will be a simple ‘sanity check’ client to make sure it works on your hardware and setup. Then, I will begin adding in more and more, features, fixes and changes as we march to a pre-alpha client and server.
I will be posting more soon, as well as a technical update on the new networking approach.
Checkout the video! More Soon…
Vidiot_X
Networking my fast paced real-time MMO:
The following is my take on developing a networking strategy for fast paced MMO’s like Phoenix USC. I did a lot of experimenting and research on Enet, Godot networking and issues related to data or packet delivery over the Internet. The approach I had taken was to have the server push as much data into as few packets as possible (Packet Lite). The idea being as long as I am not fragmenting the data over several packets all would be well. I also was sticking to a hard limit of 250kb/s for 100 player population (server to client). Additionally I was also sending one quarter of the state updates from the server to the clients every 25ms. This turned out to be good for efficiency and very low bandwidth usage but suffered under packet loss and other network conditions.
After researching about Enet and UDP packet delivery I began to understand the issues involved and came up with a solution. With Enet and UDP packets the best strategy is to send smaller packets lengths (size) and at a target rate likely not to exceed the target clients one way network latency. This is about 50 milliseconds for most connections (100ms round trip) and because the server is sending state updates at that rate there is likely to be no sever induced packet loss.
Sending many more smaller packets (Packet Heavy) dose wonders to help cope with network packet loss, dropouts and higher latency. They do this because most of the state data sent from the server to the client is sent ‘ordered and unreliable’ and if any packet is out-of-order, malformed or late it will be ignored by Enet on the client end. If any of the packet data was spread across several packets all included packets will be ignored. Smaller packet lengths make the cost of packet loss less of an issue by minimizing the impact of packet loss due to their smaller size. If a 300 byte packet is lost some data is lost. If a 1000 byte packet is lost over three times the data is lost. Think of this as holes in the data stream that need to corrected. It’s way easier dealing with smaller holes.
Also, sending packets every 25ms as I was before will induce packet loss if the clients one way network latency is over 25ms. When Enet encounters this kind of situation it starts bundling packets and sending them to the client. If there is problem with the one of the packets all of the bundled packets are ignored on the client end adding to the packet loss. Sending packets at a more likely to be encountered rate of 50ms one-way makes it likely packet bundling will not occur.
The third change was the hard data limit of 250kb/s. It’s now double at about 450kb/s (kilobits a second) for a single client connection. The reason for this is that I have increased the state updates to clients from every 100ms to 50ms. This is the best rate for small packet delivery and improves everything, including handling packet loss, dropouts and more. I may be able to lower it some but I feel its fine at that bandwidth rate per client. To support a base of 100 players on a server the data bandwidth would be approximately 50Mb/s. I have a 756Mb/s bandwidth available with my VPS so this is no problem.
The challenge was to send all of the state data from the server in small enough packets to be effective. Every 50ms the server will send state data to a client in small 10 player state chunks, then send another 10 iterating until all the player state data has been sent to a client. Each chunk of 10 player states equals about a 340 byte packet. Each iteration is a separate network send (packet) to the client.
Once the player state data is sent to the client the bullet data is sent next all within the same network frame (every 50ms). The bullet state data is generally under 700 bytes per network frame and I don’t parse this data I just send it in one packet send. I may use the same technique as above if needed.
I also alternate network channels each client state update. In this way if one channel stalls the other has a good chance of getting data though. It’s all about making sure enough packets of state data arrive to keep the simulation smooth on the client end.
The basic premise of all of this is by using smaller packets I can better cope with network packet loss and other network issues. By increasing the server to client bandwidth any packets that are ignored or lost have a minimized impact due to quicker delivery and smaller packet size. This packet heavy approach allows for more stability because it limits the impact that packet loss can have on the client.
Testing this on my StarLink connection does very well now. My average ping is about 50ms or so and it’s very common for me to observer a 0.1% packet loss with this new approach. That is sweet to see. The network stability is solid and the network performance is reliable, as seen in the video post above.
There so much I could discuss here, but I think I have covered the basics of my approach to networking Phoenix USC. It’s been a journey.
More soon…
Vidiot_X
I have been reworking the base design to accommodate the larger ships. I have added many elements and improved the look and lighting. Here I have provided a 9 images showing the changes. Of note is the addition of overhead lighting and supporting tower structures, changes to base floor lighting, light glow added to all electric arcs, roomier flag room and lanes, new flag room barricade and much more.
I will be finishing up my map/base changes and then moving on to the many other fixes to get the client released.
More soon…
Vidiot_X
Godot 2D Lighting. An editor view of recent lighting changes to my game. I am setting the lighting to reflect better ambiance with each area having it’s own lighting color.
An in game zoomed out view of lighting changes. I am using Godot 4.5.1 with TileMapLayers and a combination of 2D point lights and 2D sprite ‘fake’ lighting.
More soon…
Vidiot_X
Hi,
Here are four images with a more normal play level of zoom. I am pleased with Godot’s ISO 2D lighting and performance. In the shot just below the green glow is ‘fake’ 2D sprite lighting. The rest is Godot 2D point lights with shadows. Each ISO tile has a diffuse image, normal map and specular map to support the lighting effect. Each tile also contains shadow occlusion (most) and collision polygon’s.
More soon…
Vidiot_X
Hi,
I have reworked the entire arena base. I have made design changes to accommodate the larger player ships as wells as improved the look and feel. The arena base now uses wider lanes which provides better flow and game play. I have added more detail like towers on the edges of the base along with tower lighting effects. I have also add colored lighting to improve the look and feel.
I have had to redo the navigation polygon’s for the bots to fit the new base geometry. They are working great with the new base design. Both player and bot ships are now 30-40% larger which adds to the play-ability. Overall I like the new look and I have included several images showing the bots enjoying the new base. There also many images preceding this post.
I will wrap up a few more detail regarding this base redesign and then move to more technical things like bug and issue fixes.
More soon..
Vidiot_X
Hi,
An interesting looking view from the Godot editor. The lines are the Bot path network. Each bot is assigned a path from the network. For now once a Bot reaches the path endpoint, it is free to capture the flag or roam and attack the nearest enemy. As the Bot travels the path it will fire weapons to defend itself and stray from the path a bit.
The lines are… literal Line2D nodes. I use NavigationServer2D via GDSCript, for Bot navigation. I found it convenient to just use Line2D for two reasons. I just need an array of path waypoint’s (line node points) and a visible line showing the path in my Node2D. It works great for me and is simple.
More soon..
Vidiot_X
Hi,
Closeup still shots of ships 1 - 7 from a client / in-game view. These are 2D animated sprites. I use a 3D model and render 48 rotational view images in Blender. I then import those 48 frames into a 2D animated sprite. It’s old-school for sure but it is the best way to go for 2D ISO and TileMapLayers.
More soon…
Vidiot_X
Hi,
This video shows off the new lighting effects and base redesign. All ships in this online test are 30-40% larger. Many new map elements have been added such as towers and overhead lighting effects. Color has been added to lights in the arena to add ambiance and improve the look. The arena base has been redesigned with a roomier flag room and wider lanes and more to support the larger player ships.
Many other technical improvements (see previous posts) to networking and other features have been done.
Video: 1080p 60fps : 3min
More soon…
Vidiot_X
I will be releasing a very early test client June 7th. This is more or less a ‘sanity test’ to see how the game server and client preform with a small test base of clients. It should be fun if all goes well. I am releasing it on my discord server for this first few early test. You can read about it here on my IndieDB post. I have also included a link to my Phoenix USC Discord server and an event link.
I hope to see you there.
Vidiot_X
IndieDB Article:
Invite to Phoenix USC discord server:
https://discordapp.com/invite/gRSqcQG
First client release event on discord:
https://discord.gg/mhwacy2U?event=1503193573267603547
Hi,
Here I am testing turret mode. When attached (riding along) to a player/bot an aiming ring sprite will replace your player ship. In this short clip my client player ship is attached to a bot. You can see me firing weapons and aiming with the ring as I ride along with the bot. I am also using repel while attached.
I am working on getting this all in the coming client release.
More soon…
Vidiot_X
Hi,
Here is a video showing me rapidly attaching to multiple bot players spread across the base using the player menu as shown and selecting ‘attach’ or hitting the F7 key. Once a player is selected you can attach to that player anytime using the F7 key. It’s working awesomely and you can only attach to a team player. Attach modes allow force multiplication and allows you to essentially warp to a new location by attaching and detaching from a player (as seen at the end)
Early client testing begins June 7th.
More soon….
Vidiot_X
Player Menu Images
How does the turret mode work? I get the impression it’s for ghost players to control the turret of a live player… is that right?
@phoenixdk
Hi,
I will very shortly be posting a video explaining turret mode/attach mode, other features and game play in a video I just uploaded. It should help explain many questions… I hope. 
Cool, it seems like an interesting feature. Reminds me of some old arcade games where one player controlled the gun and one controlled movement… great fun. I seem to recall there was a great Star Wars one
Hi,
I have made a video introducing the game play and features of Phoenix USC. It’s a big video (8min) showing off and explaining the game by walking trough game play and features. Part of what I need to do now is videos like this illustrating and explaining game play and features.
It’s a really good video and it shows 8min of straight up game play while online and connected to the server. The server and client really shine along with all the visual and other changes. If you are interested in Phoenix USC this might help answer questions regarding game play and features.
More soon…
Vidiot_X
