TCP Networking: messages arriving in clumps - am I missing something?

:information_source: Attention Topic was automatically imported from the old Question2Answer platform.
:bust_in_silhouette: Asked By steevm
:warning: Old Version Published before Godot 3 was released.

Edit: I have since discovered this is a Windows-specific issue - see the answer below.

I posted this on the steam forum as well, where it was suggested I post my issue here. I’m hoping someone can help.

I’m working on a client-server architecture for a 2D football game, and have come across a strange network phenomenon that I don’t think is down to a bug in my code. I’m not excluding that possibility of course :slight_smile:

I have an authoritative server that is responsible for handling the match and sends clients regular updates of the current ball position every 4 frames. The application is set to run at 60 FPS in the Project Settings to ensure a common frame rate on both client and server (which seems to work).

When I run the client on the same machine as the server, the communication from server to client behaves as I would expect. I have a debug output that displays information on the messages received each frame. If no messages have been received by the client for processing, a corresponding entry is added to the debug output instead (“No messages” or something similar). So on the client running on the same machine as the server, I see something like this:

Received: 0,BALL,100,100,10,10
No messages
No messages
No messages
Received: 4,BALL,115,115,10,10
No messages
[Etc.]

The message contains the frame (timestamp) on the server, the type of message (BALL position) and the ball’s position and motion vectors. As you can see, the frames are nicely aligned and are received at regular intervals - every 4 frames, which is the rate the server is sending them at.

If I connect to the server from a client on my LAN (so ping <1ms), the behavior is suddenly very different. The messages seem to arrive/be processed in clumps. It looks like this:

Received: 0,BALL,100,100,10,10
Received: 4,BALL,115,115,10,10
Received: 8,BALL,130,130,10,10
No messages
No messages
No messages
No messages
No messages
[etc.]

3 messages are handled together in one clump, then 11 frames pass without any messages being received, then another clump of 3. The pattern is totally regular and predictable.

I am using TCP to send the messages because UDP will not work with IPv6 at all; TCP will work as long as the server is IPv4. I looked in the Godot source, and it appears that TCP_NODELAY is set to true, so I don’t think that messages should be sent in clumps like this.

I have looked through my code for obvious issues, and I can’t really see anything strange with the way the server sends messages - and the fact that they are sent individually on the correct frame when the client and server are on the same machine suggests to me that the logic in and of itself is sound. The issue only arises when sending data from the server to the client over a network connection.

What is also interesting, is that there is no delay when the client sends data to the server: if I send network messages from the client to the server when they are both on different machines, the client sends the message immediately and the server receives it with no delay other than the network latency (which on the LAN is <1ms, i.e. almost immediately).

Yet the only difference in the code for sending messages on the server and client that I can see is that the server uses a server object whereas the clients only need to handle the connection and the TCP stream (this is how TCP works after all). I briefly looked at the Godot source code for the server, but nothing jumped out that looked like it would introduce a delay or wait until a certain amount of data needs to be sent before sending. I’m no network programming specialist though and may have missed something.

In both cases - sending data from both client and server - I am using put_var() and get_var() to send and receive the messages. Seeing as this is essentially an undocumented function, it’s hard to know if I should be using something else. I note that the description for get_data says “Send a chunk of data through the connection, blocking if necessary until the data is done sending” but there is no indication whether the other options also send data immediately or not.

However, the fact that the client sends data immediately to the server over the network suggests to me that the problem is not at the StreamPeer level but somewhere else.

For reference, I used Kermer’s networking tutorial (Create new page · Kermer/Godot Wiki · GitHub) as the basis for my networking.

Note that I am sending and receiving messages in _fixed_process(), not _process(). As I understand it, this is necessary to ensure that the networking is handled each frame.

Any help appreciated - even if you are just guessing at a possible cause :slight_smile:

And if anyone has alternative IPv6 compatible suggestions that don’t require too much effort on my part to work with Godot, I’m also all ears.

And if anyone has alternative IPv6 compatible suggestions that don’t require too much effort on my part to work with Godot, I’m also all ears.

Godot does not support IPv6 yet.

Calinou | 2016-06-18 16:43

Yes, I know, that’s why I asked if there were any alternatives.

steevm | 2016-06-18 22:41

there’s enet GitHub - empyreanx/gdnet: An ENet wrapper for Godot. don’t know if is ipv6 compatible =/… but guess you could have a look at the code and port your library of choice.

1Snakel | 2016-06-18 23:37

I did look at ENet, but as far as I could tell, it also lacks IPv6 support as well. At least I found a number of discussions on the subject of IPv6 compatibility but no indication that it actually ever got implemented. I could be wrong, but the website isn’t very helpful in that regard.

But GDNet is otherwise pretty much what I would be looking for - a networking library that already has a Godot wrapper.

steevm | 2016-06-18 23:51

For reference StreamPeer.put_data() has a friend: StreamPeer.put_partial_data() that never blocks and will report how much data of your initial request was actually sent (you would then repeat this command in subsequent passes to transfer the remainder of the data you are trying to send. The StreamPeer.get_data() has a similar friend: StreamPeer.get_partial_data().

gau_veldt | 2016-07-15 05:06

:bust_in_silhouette: Reply From: steevm

I have found the cause of this issue. It is a Windows-specific issue that is caused by a 200 ms delay in WInsock, which is explained in far more detail here: Send data segment over TCP by using Winsock - Application Developer | Microsoft Learn

In short, the 12 frame delay I have corresponds exactly to 200 ms, and this delay is caused by Winsock not immediately sending an ACK when a packet is received. Instead, the system waits up to 200 ms for a message to be sent back. If a message is sent within this time period, the ACK is sent along with this message - referred to as “piggybacking”. In theory, this can save you bandwidth, as the TCP headers entail a certain overhead.

Now the server (or whoever sent the initial message) will not send the next packet until the previous one has been acknowledged, so this stalls communication by the full 200 ms delay if the client (or recipient) does not send any messages, and by extension no ACK.

The easy solution is to send a form of acknowledgement yourself; I fixed this issue by having the client send back a single character message whenever it receives a message from the server. This forces the ACK to also be sent along with this message, and the server can then send the next packet.

The reason this issue didn’t occur when sending data from the client to the server is because the server ALWAYS broadcasts the input from the clients, so the ACK is sent immediately along with this message.