This Anet thing has been sitting around on my HDD for ages. There's two parts to the Anet server - negotiating with the clients, and then the service table subscription, distribution & keep alive pings. This covers the first part. The second part (tables and pings) to follow when I finally get around to writing things down...
So, the initial setup here is that we have an ANET server on the primary network, and a client machine
running behind a switch. The client is the machine that will actually host games for other clients.
This setup looks like this:
-------------- --------------- ------------------------ | Server | | Switch | | Game Machine | | 10.82.129.5 |<----->| 10.82.129.114 |<----->| 192.168.0.120 | | Runs ANET | | | | Hosting and/or playing | -------------- --------------- ------------------------
So, how does the server get to the client?
Although the game machine has an IP address, it's behind a switch. NAT means that the address the server will see packets coming from will not match the address that the game machine thinks it has. However the game knows the IP address of the server, and it also knows the port to start talking to ANET on (fixed at 21143).
It is actuallly up to the game to determine the IP address that the server should use to contact it, and to do that it has to obtain the public address of the switch that it is hidden behind. This is why the patched versions are required to work on a modern connection - the game is determining the address that the outside world must reply to in order to establish a connection, and only newer games know how.
The game determines this using uPnP and IGD. The network trace from the client side shows HTTP/SOAP access to the router to retrieve this information: This, very roughly, follows the sequence
- SSDP to find the router
- request to router "GET /gatedesc.xml HTTP/1.1 "
- The response shows us how to issue the request to the gateway
- request to router "POST /upnp/control/WANIPConn1," with GetStatusInfo
- The response gives us the router status
- request to router "POST /upnp/control/WANIPConn1" with GetExternalIP
- The response gives us the external IP address (10.82.129.114 in this case). This is the address we give to anet.
- request to router "POST /upnp/control/WANIPConn1" with AddPortMapping
- This request has the description "I76Nitro"
- We pass our (client) IP address - in this test it was 192.168.0.120, however this private address never reaches anet
- We pass a port number to use (21143 in this case) - this is the same number we'll give anet
- The response lets us know we're good to go
- request to router "POST /upnp/control/WANIPConn1" with DeletePortMapping
- The response lets us know we've torn down
As an aside; This is also why direct links don't work between ANET and the game when they're on the same LAN segment. This attempt to obtain the game machine "external" IP doesn't manage to get the correct switch properties when there isn't an external IP to get.
For a more "modern" approach to this problem of UDP around NAT then look up WebRTC's STUN/TURN/Relay behaviour.
So, how does the game talk to the server?
The game and the server talk to each other over UDP. The server sits listening on port 21157 for incoming data packets, and the game opens a (random) port to talk on.
The basic format of an Anet data packet on the wire is a UDP packet that starts the payload data with a 'd' (i.e. 0x64). This is followed by another character that tells us what kind of packet this is, and from there we can parse out the data and understand the message.
There are actually a handful of different types of packets, but since the top level packets can actually be made up of combinations of other packets then this gets involved fairly quickly, however in practice the kind of packets and the data that gets sent is fairly easy to follow.
At the top level you will see "Syn", "Ack", "Data", "Ping" and "Ping Response" packets. Other packet types will be contained inside Data Packets. Occasionally multiple packets will be sent in a single "Gather" packet, which combines a number of top level packets into a single transmission. In Ruby-speak we can filter the common types of packets with code like this:
case s when 'dY' @type = PacketTypes::TYPE_SYN; when 'dU' @type = PacketTypes::TYPE_ACK when 'dB' @type = PacketTypes::TYPE_PING when 'dC' @type = PacketTypes::TYPE_PINGRESPONSE when 'dT' @type = PacketTypes::TYPE_DATA when 'dG' @type = PacketTypes::TYPE_GATHER when 'd(' @type = PacketTypes::TYPE_TSERV when 'e1' @type = PacketTypes::TYPE_ADDCLIENT when 'd^' @type = PacketTypes::TYPE_SUBSCRIBE when 'd&' @type = PacketTypes::TYPE_UNSUBSCRIBE when 'd%' @type = PacketTypes::TYPE_SMALL else ...Note the "AddClient" type, which starts with 'e'. Although we can look at most Anet packets as starting with 'd' the client add request starts with 'e'. However that's contained in a data packet (dT) so top level packets will always start with a 'd'. More on that later.
The top level packets that get sent (Syn, Ack and Data) have a two byte packet number associated with them, and the sender will expect a return with this identifier to indicate reception. The main exception to this are Ping and Ping Response packets, which have a single byte sequence number. More on this later
So, how does the game establish a connection to the server?
The game sends a SYN packet to the server to announce that it's ready to host a game. It expects the server to reply with an ACK to acknowledge receipt of this packet, and a SYN from the server.
So the sequence is:
- Game sends a SYN to the server
- Server sees the SYN and sends back a SYN to the game on the open port
- The Server sends an ACK for the game SYN
- The Server waits for the ACK from the game acknowledging receipt of the SYN
The Actual Server/Game Handshake in detail
Given this and the anet server code we can trace the activity as the game connects to the server. Dumping some Wireshark traces from the server end then we can see:
Internet Protocol Version 4, Src: 10.82.129.114, Dst: 10.82.129.5 User Datagram Protocol, Src Port: 21143, Dst Port: 21157 Data (26 bytes) 0000 64 59 11 78 15 05 06 0a 52 81 72 52 97 0a 52 81 dY.x....R.rR..R. 0010 05 52 a5 07 0a 52 81 72 52 97 .R...R.rR.
This is the initial SYN from the game to the ANET server The SYN packet breaks down as:
- 64 59: dY - a SYN packet
- 11 78 : 16 bit packet number (0x7811)
- 15 : Packet Length (21 bytes)
- 05 : Version = 5
- 06 : Address Size
- 0a 52 81 72 52 97 : Src Address #1
- 0a 52 81 05 52 a5 : Dest Address
- 07 : Capabilities
- 0a 52 81 72 52 97 : Src Address #2
- Source Address 1&2: 0a 52 81 72 52 97 => 10.82.129.114:21143
- Dest Address: 0a 52 81 05 52 a5 => 10.82.129.5:21157
comm_driverInfo_t, and 0x07 means "is visible, knows the playerlist and you can send it a gamelist".
Now that the game has announced itself to the anet server, the server replies with a SYN of it's own to the game, and also acknowledges receipt of the game's SYN packet.
Internet Protocol Version 4, Src: 10.82.129.5, Dst: 10.82.129.114 User Datagram Protocol, Src Port: 21157, Dst Port: 21143 Data (26 bytes) 0000 64 59 72 f6 15 05 06 0a 52 81 05 52 a5 0a 52 81 dYr.....R..R..R. 0010 72 52 97 07 0a 52 81 05 52 a5 rR...R..R.
This is a SYN that the server sends to the client that has just announced itself. This has the same format as the incoming SYN, with the following unique properties:
- 72 f6 : 16 bit packet number (0xf672)
- 0a 52 81 05 52 a5 : Src Address #1
- 0a 52 81 72 52 97 : Dest Address
- 0a 52 81 05 52 a5 : Src Address #2
These two SYN values show us attempting to establish a symmetric link. The server has chosen a unique packet ID, but in this case then the source and address are fairly simple.
Internet Protocol Version 4, Src: 10.82.129.5, Dst: 10.82.129.114 User Datagram Protocol, Src Port: 21157, Dst Port: 21143 Data (5 bytes) 0000 64 55 11 78 80 dU.x.
And this is the ACK from the server back to the client. The packet breaks down as:
- 64 55: dU - an ACK packet
- 11 78 : 16 bit packet number - this was the packet ID in the game SYN (0x7811)
- 80 : packet number offset. The value of 0x80 means "ignore this field".
Next up we're waiting for the client to ACK our SYN. This takes the client several seconds to work through, so we actually wind up resending the SYN request a couple of times. This is a identical resend, and we attempt this every few seconds (roughly 3 or four seconds apart)
So we see these re-transmissions
174161 64.286843956 10.82.129.5 10.82.129.114 UDP 68 21157 -> 21143 Len=26 180170 66.814850322 10.82.129.5 10.82.129.114 UDP 68 21157 -> 21143 Len=26 188610 70.958528631 10.82.129.5 10.82.129.114 UDP 68 21157 -> 21143 Len=26 195822 75.101658184 10.82.129.5 10.82.129.114 UDP 68 21157 -> 21143 Len=26 209281 79.245317336 10.82.129.5 10.82.129.114 UDP 68 21157 -> 21143 Len=26 217715 83.389025871 10.82.129.5 10.82.129.114 UDP 68 21157 -> 21143 Len=26 226179 87.542323140 10.82.129.5 10.82.129.114 UDP 68 21157 -> 21143 Len=26 234626 91.686158758 10.82.129.5 10.82.129.114 UDP 68 21157 -> 21143 Len=26
At this point the ACK comes back from the client.
Internet Protocol Version 4, Src: 10.82.129.114, Dst: 10.82.129.5 User Datagram Protocol, Src Port: 21143, Dst Port: 21157 Data (5 bytes) 0000 64 55 72 f6 80 dUr..
This ACK from the client has the same format we used for ours. The packet breaks down as:
- 64 55: dU - an ACK packet
- 72 f6 : 16 bit packet number - this was the packet ID we sent in our SYN
- 80 : packet number offset. Another "ignore this field".
Following this there's a cluster of ACK's, as the client catches up with the multiple SYN packets we sent out. Since we sent out identical SYN requests the corresponding ACK packets are all identical. Since we sent 8 duplicate SYNs then we also see 8 ACK returns. It looks like the ACK is only generated for outstanding SYNs when the game actually leaves the setup screen and the player on the host is on the game level.
237028 92.782751239 10.82.129.114 10.82.129.5 UDP 60 21143 -> 21157 Len=5 237029 92.782835138 10.82.129.114 10.82.129.5 UDP 60 21143 -> 21157 Len=5 237030 92.782857474 10.82.129.114 10.82.129.5 UDP 60 21143 -> 21157 Len=5 237031 92.782880754 10.82.129.114 10.82.129.5 UDP 60 21143 -> 21157 Len=5 237032 92.782903744 10.82.129.114 10.82.129.5 UDP 60 21143 -> 21157 Len=5 237033 92.782980110 10.82.129.114 10.82.129.5 UDP 60 21143 -> 21157 Len=5 237034 92.783002020 10.82.129.114 10.82.129.5 UDP 60 21143 -> 21157 Len=5 237035 92.783024780 10.82.129.114 10.82.129.5 UDP 60 21143 -> 21157 Len=5
So, how does the server communicate securely with the game?
Following this then we are in the information exchange phase. The ANET server sends out an initial challenge packet. This is sent as a data packet (type 'dT') which contains a Tserv packet (type 'd(')
This is Activision's "Trivial Challenge Authentication", used to secure password transfer. From the header it's used to perform:
response(challenge) = MD5(TDESn(MD5(password), challenge))
However since we don't use any of the user account stuff in Nitro, then we really don't care: We'll see the server send out the TSERV packet and the game will ACK the packet.
Next we'll look at the tables and how the available games are advertised.