Saturday, 23 September 2017

Anet Part 1: Negotiating With the Clients

This Anet thing has been sitting around on my HDD for ages. There's two parts to the Anet server - negotiating with the clients, and then the service table subscription, distribution & keep alive pings. This covers the first part. The second part (tables and pings) to follow when I finally get around to writing things down...

Determining Addresses

So, the initial setup here is that we have an ANET server on the primary network, and a client machine running behind a switch. The client is the machine that will actually host games for other clients.
This setup looks like this:

 --------------         ---------------         ------------------------
|    Server    |       |     Switch    |       |      Game Machine      | 
| |<----->| |<----->|      |
|   Runs ANET  |       |               |       | Hosting and/or playing |
 --------------         ---------------         ------------------------

So, how does the server get to the client?

Although the game machine has an IP address, it's behind a switch. NAT means that the address the server will see packets coming from will not match the address that the game machine thinks it has. However the game knows the IP address of the server, and it also knows the port to start talking to ANET on (fixed at 21143).

It is actuallly up to the game to determine the IP address that the server should use to contact it, and to do that it has to obtain the public address of the switch that it is hidden behind. This is why the patched versions are required to work on a modern connection - the game is determining the address that the outside world must reply to in order to establish a connection, and only newer games know how.

The game determines this using uPnP and IGD. The network trace from the client side shows HTTP/SOAP access to the router to retrieve this information: This, very roughly, follows the sequence

  • SSDP to find the router
  • request to router "GET /gatedesc.xml HTTP/1.1 "
  • The response shows us how to issue the request to the gateway
  • request to router "POST /upnp/control/WANIPConn1," with GetStatusInfo
  • The response gives us the router status
  • request to router "POST /upnp/control/WANIPConn1" with GetExternalIP
  • The response gives us the external IP address ( in this case). This is the address we give to anet.
  • request to router "POST /upnp/control/WANIPConn1" with AddPortMapping
    • This request has the description "I76Nitro"
    • We pass our (client) IP address - in this test it was, however this private address never reaches anet
    • We pass a port number to use (21143 in this case) - this is the same number we'll give anet
  • The response lets us know we're good to go
At this point the game is in a position to supply the server with an "external" IP address to talk back to, and is able to communicate on the target port - the router handles external to internal IP port mapping. When we're done then we issue
  • request to router "POST /upnp/control/WANIPConn1" with DeletePortMapping
  • The response lets us know we've torn down

As an aside; This is also why direct links don't work between ANET and the game when they're on the same LAN segment. This attempt to obtain the game machine "external" IP doesn't manage to get the correct switch properties when there isn't an external IP to get.

For a more "modern" approach to this problem of UDP around NAT then look up WebRTC's STUN/TURN/Relay behaviour.

So, how does the game talk to the server?

The game and the server talk to each other over UDP. The server sits listening on port 21157 for incoming data packets, and the game opens a (random) port to talk on.

The basic format of an Anet data packet on the wire is a UDP packet that starts the payload data with a 'd' (i.e. 0x64). This is followed by another character that tells us what kind of packet this is, and from there we can parse out the data and understand the message.

There are actually a handful of different types of packets, but since the top level packets can actually be made up of combinations of other packets then this gets involved fairly quickly, however in practice the kind of packets and the data that gets sent is fairly easy to follow.

At the top level you will see "Syn", "Ack", "Data", "Ping" and "Ping Response" packets. Other packet types will be contained inside Data Packets. Occasionally multiple packets will be sent in a single "Gather" packet, which combines a number of top level packets into a single transmission. In Ruby-speak we can filter the common types of packets with code like this:

 case s
 when 'dY'
     @type = PacketTypes::TYPE_SYN;
 when 'dU'
     @type = PacketTypes::TYPE_ACK
 when 'dB'
     @type = PacketTypes::TYPE_PING
 when 'dC'
     @type = PacketTypes::TYPE_PINGRESPONSE
 when 'dT'
     @type = PacketTypes::TYPE_DATA
 when 'dG'
     @type = PacketTypes::TYPE_GATHER
 when 'd('
     @type = PacketTypes::TYPE_TSERV
 when 'e1'
     @type = PacketTypes::TYPE_ADDCLIENT
 when 'd^'
     @type = PacketTypes::TYPE_SUBSCRIBE
 when 'd&'
     @type = PacketTypes::TYPE_UNSUBSCRIBE
 when 'd%'
     @type = PacketTypes::TYPE_SMALL
Note the "AddClient" type, which starts with 'e'. Although we can look at most Anet packets as starting with 'd' the client add request starts with 'e'. However that's contained in a data packet (dT) so top level packets will always start with a 'd'. More on that later.

The top level packets that get sent (Syn, Ack and Data) have a two byte packet number associated with them, and the sender will expect a return with this identifier to indicate reception. The main exception to this are Ping and Ping Response packets, which have a single byte sequence number. More on this later

So, how does the game establish a connection to the server?

The game sends a SYN packet to the server to announce that it's ready to host a game. It expects the server to reply with an ACK to acknowledge receipt of this packet, and a SYN from the server.

So the sequence is:

  • Game sends a SYN to the server
  • Server sees the SYN and sends back a SYN to the game on the open port
  • The Server sends an ACK for the game SYN
  • The Server waits for the ACK from the game acknowledging receipt of the SYN

The Actual Server/Game Handshake in detail

Given this and the anet server code we can trace the activity as the game connects to the server. Dumping some Wireshark traces from the server end then we can see:

Internet Protocol Version 4, Src:, Dst:
User Datagram Protocol, Src Port: 21143, Dst Port: 21157
Data (26 bytes)
0000  64 59 11 78 15 05 06 0a 52 81 72 52 97 0a 52 81   dY.x....R.rR..R.
0010  05 52 a5 07 0a 52 81 72 52 97                     .R...R.rR.

This is the initial SYN from the game to the ANET server The SYN packet breaks down as:

  • 64 59: dY - a SYN packet
  • 11 78 : 16 bit packet number (0x7811)
  • 15 : Packet Length (21 bytes)
  • 05 : Version = 5
  • 06 : Address Size
  • 0a 52 81 72 52 97 : Src Address #1
  • 0a 52 81 05 52 a5 : Dest Address
  • 07 : Capabilities
  • 0a 52 81 72 52 97 : Src Address #2
The 6 byte address is the four byte IP plus two bytes for the port number. In this case it's:
  • Source Address 1&2: 0a 52 81 72 52 97 =>
  • Dest Address: 0a 52 81 05 52 a5 =>
The anet server uses this address to reply on. The capabilities are given by comm_driverInfo_t, and 0x07 means "is visible, knows the playerlist and you can send it a gamelist".

Now that the game has announced itself to the anet server, the server replies with a SYN of it's own to the game, and also acknowledges receipt of the game's SYN packet.

Internet Protocol Version 4, Src:, Dst:
User Datagram Protocol, Src Port: 21157, Dst Port: 21143
Data (26 bytes)
0000  64 59 72 f6 15 05 06 0a 52 81 05 52 a5 0a 52 81   dYr.....R..R..R.
0010  72 52 97 07 0a 52 81 05 52 a5                     rR...R..R.

This is a SYN that the server sends to the client that has just announced itself. This has the same format as the incoming SYN, with the following unique properties:

  • 72 f6 : 16 bit packet number (0xf672)
  • 0a 52 81 05 52 a5 : Src Address #1
  • 0a 52 81 72 52 97 : Dest Address
  • 0a 52 81 05 52 a5 : Src Address #2
In this case Src Address #1&2 are and Dest Address is

These two SYN values show us attempting to establish a symmetric link. The server has chosen a unique packet ID, but in this case then the source and address are fairly simple.

Internet Protocol Version 4, Src:, Dst:
User Datagram Protocol, Src Port: 21157, Dst Port: 21143
Data (5 bytes)
0000  64 55 11 78 80                                    dU.x.

And this is the ACK from the server back to the client. The packet breaks down as:

  • 64 55: dU - an ACK packet
  • 11 78 : 16 bit packet number - this was the packet ID in the game SYN (0x7811)
  • 80 : packet number offset. The value of 0x80 means "ignore this field".

Next up we're waiting for the client to ACK our SYN. This takes the client several seconds to work through, so we actually wind up resending the SYN request a couple of times. This is a identical resend, and we attempt this every few seconds (roughly 3 or four seconds apart)

So we see these re-transmissions

 174161 64.286843956          UDP      68     21157 -> 21143 Len=26
 180170 66.814850322          UDP      68     21157 -> 21143 Len=26
 188610 70.958528631          UDP      68     21157 -> 21143 Len=26
 195822 75.101658184          UDP      68     21157 -> 21143 Len=26
 209281 79.245317336          UDP      68     21157 -> 21143 Len=26
 217715 83.389025871          UDP      68     21157 -> 21143 Len=26
 226179 87.542323140          UDP      68     21157 -> 21143 Len=26
 234626 91.686158758          UDP      68     21157 -> 21143 Len=26

At this point the ACK comes back from the client.

Internet Protocol Version 4, Src:, Dst:
User Datagram Protocol, Src Port: 21143, Dst Port: 21157
Data (5 bytes)
0000  64 55 72 f6 80                                    dUr..

This ACK from the client has the same format we used for ours. The packet breaks down as:

  • 64 55: dU - an ACK packet
  • 72 f6 : 16 bit packet number - this was the packet ID we sent in our SYN
  • 80 : packet number offset. Another "ignore this field".

Following this there's a cluster of ACK's, as the client catches up with the multiple SYN packets we sent out. Since we sent out identical SYN requests the corresponding ACK packets are all identical. Since we sent 8 duplicate SYNs then we also see 8 ACK returns. It looks like the ACK is only generated for outstanding SYNs when the game actually leaves the setup screen and the player on the host is on the game level.

 237028 92.782751239           UDP      60     21143 -> 21157 Len=5
 237029 92.782835138           UDP      60     21143 -> 21157 Len=5
 237030 92.782857474           UDP      60     21143 -> 21157 Len=5
 237031 92.782880754           UDP      60     21143 -> 21157 Len=5
 237032 92.782903744           UDP      60     21143 -> 21157 Len=5
 237033 92.782980110           UDP      60     21143 -> 21157 Len=5
 237034 92.783002020           UDP      60     21143 -> 21157 Len=5
 237035 92.783024780           UDP      60     21143 -> 21157 Len=5

So, how does the server communicate securely with the game?

Following this then we are in the information exchange phase. The ANET server sends out an initial challenge packet. This is sent as a data packet (type 'dT') which contains a Tserv packet (type 'd(')

This is Activision's "Trivial Challenge Authentication", used to secure password transfer. From the header it's used to perform:

response(challenge) = MD5(TDESn(MD5(password), challenge))

However since we don't use any of the user account stuff in Nitro, then we really don't care: We'll see the server send out the TSERV packet and the game will ACK the packet.

Next we'll look at the tables and how the available games are advertised.

Sunday, 30 July 2017

Interstate 76 File Format Summary

I've got a bunch of stuff on ANET to come up, but it's taking a bit longer than I expected to get a working dummy implementation, so as an interim posting, here is summary of the file formats for I76 and Nitro.

This should break down all the format information so far, with my current thinking on what goes where. It should be easier to follow than tracking through the blog postings.

Saturday, 15 July 2017

Manually Repacking I76/Nitro Levels

Nitro Map Reprocessor

This is something that dropped out of a chat with one of the I76 forum posters about fixing up old maps.

There are a number of I76 & Nitro mission files that have issues, like misplaced objects or are crash prone, and could do with fixing. However without the sources this is a difficult process involving a hex editor and a lot of trial and error.

So, attached to this is a set of Ruby scripts which simplify some of the process, by decompiling missions into a text form (JSON), and then recompiling them back into runnable files.

The code is in This Tarball Here, and is in three scripts; a decompiler (main_ar.rb), a recompiler (main_raar.rb) and a utility library that handles the actual format data(objects.rb). It's a little "my first Ruby program" and needs cleaning up (we could use a smarter approach/library than pack() for a start), but it's functional enough to use.

This script extracts the mission information and this data is converted into JSON, with the decompiler taking care of some obvious conversions, such as to & from float, and also grouping data where we know what it is (position, rotation, etc).

This format is essentially the same described in earlier posts, with a few fixes - notably Erik's comment that the flag field we thought followed the object name is really a float (giving us a 3x3 transform matrix), and it also handles LDEF sections, which support linked arrays of objects and additional fields in the WDEF.

Although we could unpack the zone definitions and game scripting there's little that could be manually edited here, so they're left as "opaque" sections and the data in the JSON is just a byte array. The same is true of a couple of other sections (revision data, etc).

To Run Them

Unpack a File

To unpack a file try ruby -w ./main_ar.rb gumball.rac

Where gumball.rac is a level file. This produces a set of json files, one per section. These files have the original name with the segment name and ".json" attached to the end, so "gumball.racODEF.json" is the object section, "gumball.racWDEF.json" is the world section, etc.
It also produces an index file, ".idx", which contains the list of sections. In this case gumball.rac.idx

Repack a File

To repack use the main_raar script, and either give it an index file, or a list of the unpacked JSON sections, e.g. ruby -w ./main_raar.rb gumball.rac.idx

Which produces the file "tmp.dat" which is a mission file that can be placed in the game directory and run.

So, as an example, running ruby -w ./main_ar.rb gumball.cbt Then the output file gumball.cbtODEF.json contains the following definition (around line 400):
    "name": "bddonut1",
    "identifier_tag": 0,
    "rotation": [
    "position": [
    "trailing": [
    "class_id": 2,
    "basicstop": 0
This is the Dount Stand: Viewing it from the front and overhead view then originally it's:
We can change the position by editing the position data, so for example:
    "position": [
  "position": [
And if we run the output processor, and copy the file back into the missions directory with
ruby -w ./main_raar.rb gumball.cbt.idx  
cp tmp.dat  ~/.wine32/drive_c/Program\ Files/Activision/I76Nitro/miss8/gumball.cbt
Then when we reload the map we've moved the Dount Stand forward.

Similarly we can copy the entire Donut stand section, and by creating 3 more copies and adding 30 to the X position each time we get....

which is the in car view at the top of this post

Known Bugs

Although it should produce the same output from a given input there are a couple of files where label strings with trailing spaces are cropped during string cleanup (so from " Foo " to " Foo"). It also doesn't do much in the way of syntax validation or checking.

updated 16/7: the file was opened in text mode, not binary. This is fine on Linux, but means that on Windows the read of mission files came up short. Fixed and uploaded V2 of the tarball.

Monday, 1 May 2017

The New AngelScript Raycast Vehicle Model

This is a quick note on how to use the Urho3D vehicle model. It's based around the latest git repository, which has AngelScript interfaces for the Bullet Raycast Vehicle. It's written for the state of the Urho3D repo trunk on Apr 29.

The Vehicle Model

The AngelScript object representing the vehicle is RaycastVehicle . This is a wrapper around the underlying Bullet btRaycastVehicle.

To use this we have the basic workflow of:

  • Create a Node for the Vehicle as a child of the Scene, then under this node
    • Create a StaticModel component for the vehicle model
    • Create a RigidBody component for basic physics properties
    • Create a CollisionShape component for the physics engine to handle collisions
    • Create a RaycastVehicle component. This is the new component and we should also
      • Call Init() on this component to set it up
      • Create a node per wheel, and link the wheels to the vehicle
      • Set up the suspension and tyre properties per wheel
      • Call ResetWheels() when set up
At this point we can use the raycastVehicle.SetSteeringValue() and raycastVehicle.SetEngineForce() to set the direction and driving force through the wheels.

The Suspension and Wheel Parameters

We essentially create a node per wheel, then connect the wheels to the vehicle using AddWheel(). There are a handful of basic properties that need setting, both as part of the initial call, and also as additional parameter settings.

The wheel node has a model attached (we can use the cylinder as last time), however in the case of the Raycast vehicle we don't need to provide a collision box or any other information with this model- it's decorative with the vehicle model updating position according to the associated wheel & suspension parameters we supply (for example, if we have scale the wheel model down then the car will simply appear to float above the ground).

The parameters that we need to determine are:

  • Acceleration and Braking
    • Car Mass: Typically in Kg. Applied to the Chassis.
    • Braking Force: Higher values for sharper braking, Supplied to SetBrake().
    • Engine : Higher values for more force. Supplied to SetEngineForce().
  • Suspension Details
    • Suspension Rest Length: Maximum Length of the (unloaded) suspension (in M)
    • Suspension Travel: Maximum Suspension travel (in cm).
    • Suspension Stiffness: Higher values for a stiffer suspension.
    • Suspension Damping during Relaxation: Lower values for longer ringing suspension.
    • Suspension Damping during Compression: Lower values for longer ringing suspension.
  • Wheel Properties
    • Wheel Radius:
    • Wheel Width:
    • Wheel Friction Slip: Tire friction to ground. Limits the maximum impulse pushed through the wheel.
    • Roll Influence: A side impulse to the chassis. High values will cause the car to flip on turning.
Most of these are self explanatory, and are either supplied to the wheel constructor or as separate parameters. Also there's a mix of units and ranges here (so Rest length of 0.3M but the travel limit in cm would be 300 for full travel). We also supply some basic parameters to determine how the wheel is positioned on the car (is it a front wheel, the direction of travel and the axle axis). A simple setup looks something like this:
    raycastVehicle.AddWheel(wheelNode, wheelDirection, wheelAxle, suspensionRestLength, wheelRadius, isFrontWheel);
    raycastVehicle.SetWheelSuspensionStiffness(id, suspensionStiffness);
    raycastVehicle.SetWheelDampingRelaxation(id, suspensionDamping);
    raycastVehicle.SetWheelDampingCompression(id, suspensionCompression);
    raycastVehicle.SetWheelFrictionSlip(id, wheelFriction);
    raycastVehicle.SetWheelRollInfluence(id, rollInfluence);
    raycastVehicle.SetMaxSuspensionTravel(id, suspensionTravel);


As well as obvious missing features in the model (gearing, upper speed limits, wheel parameters, terrain traction, etc) there's a couple of limitations here - the model does not seem to support sideslip under high speed turning: The tyres either have traction or the car rolls, but nothing in between (but this may be a reflection of the settings we have). The car also needs tweaks to the collision box, and also the speed reading is almost certainly off.

The sideslip we can probably emulate using side forces and playing with tyre friction and checking the speed and turning circle, but that's more complex than this example. Further testing required.

Sunday, 30 April 2017

Adding Simple Physics

Next up we can add some simple physics handling to the basic map and model loads under Urho3D, and this lets us handle simple objects and push cars round the map.

Introducing Physics

Urho uses the Bullet Physics library to provide a physics engine that supports collision detection and rigid body simulation.

Urho embeds a complete copy of Bullet under the Source/ThirdParty/Bullet/ directory, and there is a subset of the functionality exposed through AngelScript. However the AngelScript API doesn't support the complete feature set of Bullet that the C++ code can access, which will get annoying (as we see later).

Adding A Simple Colliding Entity

So initially we have to:

Enable Physics

This is a single line in the AngelScript


Turn on Collisions With the Scene

This is fairly simple we have the Node that the terrain Component was created in, and we want to tell the Physics engine that objects should collide with this terrain. To create this behaviour the engine wants us to do the following under the Terrain node:

  • Create a Rigid Body: A solid body, which doesn't deform, and has basic physics properties (such as mass and friction).
  • Add a Collision Shape: To support collisions, marked as Terrain in this case.

So, first we make a default RigidBody associated with the Terrain Node as a fixed object (the default, for a zero mass item).

RigidBody@ body = terrainNode.CreateComponent("RigidBody");
  body.collisionLayer = 2;

The collisionLayer parameter provides a bitmask that the Bullet engine uses to determine if objects should collide. The tutorials use the mask value of 2 throughout for terrain, and we'll keep that convention here.

There are actually two parameters here: collisionLayer and collisionMask. This is a detail we won't get into since it's not something that affects this demo, but when Objects 1 & 2 intersect then we evaluate:

  • the object 1 mask and object 2 collision layer
  • the object 2 mask and object 1 collision layer
If either of these operations are non-zero then we have a collision, otherwise they pass through each other.

Next we associate a collision shape - this is created alongside the static Rigid Body under the terrain node. The shape of this collider is marked as Terrain.

CollisionShape@ shape = terrainNode.CreateComponent("CollisionShape");
And that's it.

Add Shadows

Not necessary, but the relation of objects in the world is much clearer with shadows - otherwise entities have a tendency to look painted over, rather than on, the terrain. So we have to enable light shadows when we declare our light:

light.castShadows = true;
And when we create an object we'll also set the model castShadows property to true.

And The Object

We can add a simple colliding sphere (we'll call it a Marble), by creating a node with a Static Model based on "Sphere.mdl", and then adding the RigidBody and CollisionShape declaration.

For this example we use SetSphere() to set the collision shape to match the (spherical) model, and we give the RigidBody a mass and rolling friction and watch it go. Here's a complete working class, based on the code in

class Marble {
Node@ marbleNode;
StaticModel@ mdlObj;
RigidBody@ body;
CollisionShape@ shape;

bool doPrint = false;

  Marble(Scene@ scene, float sz, float speed, float mass) {
    Create(scene, sz, speed, mass);

  void Create(Scene@ scene, float sz, float speed, float mass) {
    marbleNode = scene.CreateChild();
    marbleNode.position = camera.cameraNode.position;
    marbleNode.rotation = camera.cameraNode.rotation;

    mdlObj = marbleNode.CreateComponent("StaticModel");
    mdlObj.model = cache.GetResource("Model", "Models/Sphere.mdl");
    mdlObj.material = cache.GetResource("Material", "Materials/Terrain.xml");
    mdlObj.castShadows = true;

    body = marbleNode.CreateComponent("RigidBody");
    body.mass = mass;
    body.rollingFriction = 0.9f;

    shape = marbleNode.CreateComponent("CollisionShape");

    body.linearVelocity = camera.cameraNode.rotation * Vector3(0.0f, 0.0f, 1.0f) * speed;

    SubscribeToEvent(marbleNode, "NodeCollision", "HandleNodeCollision");

  void HandleNodeCollision(StringHash eventType, VariantMap& eventData)  {
    if (doPrint) {
    RigidBody@ otherBody = eventData["OtherBody"].GetPtr();
    RigidBody@ hitBody = eventData["Body"].GetPtr();
      Print("Ding "+ +" against "+ + " !");
This provides the basic object and also an optional debugging report on collisions.

We can play around with this code and check the behaviour of the physics simulation. There are some problems when we fire very small and very fast marbles which have a tendency to fall through the terrain or when we combine very heavy and very light objects - this is expected and the Bullet and Urho docs mention the need to handle these cases carefully.

Adding A Car

There's an example of a simple vehicle provided by the Urho file ./bin/Data/Scripts/ It's worth taking a moment to look at how this works.

The car consists of a main hull body and four wheels. The main body is a RigidBody/CollisionShape and only has mass and air resistance, and all the power and steering is done by manipulating the wheels directly.

Each wheel is a cylinder model with an associated RigidBody and a spherical CollisionShape connected to the main body via a Hinge Constraint.

The hinge constraint is used to link the body to the wheel, and the wheel spins around this constraint. The hinge constraint limits are set from -180 to +180 degrees which allow the wheel to spin completely.

The constraint connecting the wheel to the body is also used for steering: by rotating the angle of the connection between the body and the wheel, the code deflects the front two wheels in response to steering commands.

Although the hinge is restricting motion around other axis there are some damped-spring style reactions to forces which can be seen at higher mass main body values, although I suspect this may be a bug involving the mass ratio of a light wheel to a heavy body. It's actually almost a usable hack for suspension style behaviour.

By default the reference code subclasses ScriptObject, which allows it to serialise the object to a file, and to handle control events directly. We can actually fire the event handler explicitly and break this connection if we want to simplify the implementation and don't care about serialising.

When accelerating the ApplyTorque() function is used to change the angular velocity of the wheel. In addition there's a directional component introduced to the front wheel (when steering is turned). Actual acceleration results from the interaction between the rotating wheels and terrain.

Lastly there's a downforce component which keeps the car connected to the terrain, by applying a downward force based on the car velocity.

Updating The Car

Although the code works with a basic setup, it's difficult to control and tune, and doesn't work when exposed to the kind of terrains we see in I76.

Firstly we need lower the downforce component. This works well to glue the car to the rolling terrain, but it tends to let the car drive down cliff faces on the I76 maps.

Next we modify the loaded model to be one of the one piece I76 car geometries, and tweak the wheel positions and collision bounds accordingly. We also move the body mass up to a more realistic (real world) value of around 1500 (assuming 1 unit = 1kg).

To simplify the control of acceleration we replace the "Torque" model of the demo with a simple ApplyImpulse() version. This doesn't rely on the torque for acceleration, but pushing the wheels directly. We put a simple check in to ensure that the wheel is in collision with the terrain before applying force through it, but that's all. This gives a simpler more controllable vehicle, although it's a bit like driving on glass.

How This Needs To Improve

Right now there's no suspension, or notion of vehicle simulation such as engine or tyre behaviour, weight transfer, etc. which would make the driving behaviour more like the original game.
We could add some of this in AngelScript, but that has an obvious overhead in terms of processing load and even a simplified model is a very complex problem. So are there any other options?

The good news is that the bullet physics library provides a complete abstraction - the btRayCastVehicle model. This supports a single solid body type which behaves like a vehicle, has a tunable suspension and wheel model, and takes care of all the details for us. There's a great example of it in action in this GitHub repository by the Urho user Lumak.

However the downside of the Bullet vehicle is that this is currently a C++ only thing - it's not exposed to the AngelScript interfaces. This is an annoyance, since using AngelScript to keep things cross platform is a definite advantage.

So we've hit a break where I need to figure out where to go next - Is there a plausible I76-like car behaviour without too much scripting code using a few approximations, or is native code for the car the way to go, or some mix and custom scripting interfaces down to the library? At this point experimentation will be required though, so that'll do for now...

Update - having just pulled the latest from the Urho3D Repo it turns out that RayCast vehicle integration & scripting hooks have been merged into the main release as of 20-something hours ago. So this decision is probably simpler than I thought.

Sunday, 23 April 2017

Urho I76 Model Viewer

Loading Up Models

The next thing to do under Urho3D is to load up the models for I76 assets.

The Hard Way

The hard way to do this is to manually generate an Urho VertexElement array and attach the model vertex and normal data, linking a VertexBuffer and IndexBuffer through a Geometry element.

This approach works, and there's an example of it in the sample application, but it's fiddly and my implementation is error prone. So let's not do that.

The Easier Way

The simplest way to do this is to use the AssetImporter tool from the Urho3D distribution. This can take an OBJ format file, which we can generate easily, and produce output files that Urho3D can load directly. It adds an extra step, but simplifies the process.

So we generate the OBJ file format files as we do in the existing C++ code, and then use the fileSystem.SystemRun() call in AngelScript to shell out to the AssetImporter binary. There's code in the default Urho application that we can use more or less directly for a prototype, that handles assembling the command line and also tacks on the .exe extension for windows. We'll replace this at some point, but for now let's just use it as is for the demo.

Otherwise there really isn't much clever here - we reimplement the ZFS unpacker in Angelscript to extract the resources, then we can view either individual models from GEO files, or complete vehicle assemblies from the VDF file. Extracting from the ZFS takes several minutes, and so the app should be run windowed, and the terminal will report the progress of the unpack. We could implement a background loader, but it's not worth the effort for a run-once process like this.

The only other annoyance is that the floating point output from AngelScript threw up errors with the AssetImporter by default - small values will be represented by scientific format (e.g. 3.3021e-05) and the importer doesn't like some of these, and there didn't seem to be a way to format the output more precisely. So we spot small values and treat these as exactly zero when we write them out.

Here's a tarball attached with:

  • AssetLoader: - Load a ZFS file, and unpack the ZFS itself, embedded PAK files and do the basic conversion for GEO to objects and VQM to texture images.
  • ObjLoader: - View individual object files. Does on the fly conversion.
  • CarLoad: - Load VDF files - this has some issues (the tendency to mess up interior geometry fragments, and some normals are iffy), but it's functional enough to pull out some basic geometry for the composite models.

The conversion process just applies a default image file for testing at present (whichever image you copy into Textures/test.png), and requires a defaultmat.xml Material file. It's lacking basic error checking, but as a proof of concept it holds together.

Also this will not work on the Nitro ZFS, since AngelScript doesn't know how to decompress the LZO compressed files without the platform specific libraries.

Otherwise, it mostly sorta works...

AngelScript: Some Thoughts

Having used AngelScript for a few days then the thoughts that keep cropping up are

The Good

  • Fast Prototyping.
  • Straightforward UI & 3D implementation.
  • Feature rich primitives.
  • Syntax: maps neatly to C++ versions.

The Bad

  • No debugger: This is a real PITA. The code has wound up with debug Print calls scattered around.
  • Silent Failures: Some setup errors leave things silently non functional, rather than producing more obvious error reports.
  • Slow: Although the heavy lifting is mostly done by the core engine having scripts generate things like texture images and bitmap resources can result in major slowdowns.

The Ugly

  • Ownership: The Urho Angelscript version uses reference counting for memory management, but without a debugger it's easy to lose track of what references are valid at times and leak resources (such as open file handles).
  • Not Quite Vanilla AngelScript: Enough custom types that the default AS docs don't always seem to apply directly or help understand Urho's use of the language.
  • Documentation: It's there, but not always useful, and the examples take some parsing to understand: it could be better.
  • No Printing Format control? I couldn't figure out how to persuade floats to output in a specific format, hence the low value rounding utility.

Sunday, 16 April 2017

Loading the Levels and Texturing

Loading The Map

This will bring together the stuff we've done so far to load up the levels from the I76 mission files.

Initially we copy the contents of the installed Mission directory under the Urho3D Data directory so we can locate the target files easily. Go to the the Urho3D Data directory, create a subdirectory Imports and copy the mission files into that.

The AngelScript code is available in this zip file. Unzip the file contents, and run the viewer with a line like "Urho3DPlayer ./ -w". This code doesn't do much in the way of error checking, so make sure that under the Urho3D Data directory you have a directory "Imports" and that the mission files (.msn and .ter) are contained in it. A description of how the code works is below.

Parsing The Level

File Load

We use the standard Urho3D file dialog system to locate a target .msn file, and since this is all boilerplate there's nothing special to notice here.

However to simplify interaction with this then we isolate it to a separate class for handling, and we have the dialog generate an event when it's complete to feed back to the main code.

So we have a class "OurFileSelector" which uses the stock FileSelector to generate a pop up window. The HandleFileSelected event is used to flag a selection and we create an event using a VariantMap and SendEvent() to signal the file chosen.

In the main code we've added the call to launch the selector with fselect.PickFile(), and to pick up the completion we add an event handler:

SubscribeToEvent("FileChosen", "HandleFileChosen");
We can just define this event and handler in either order without any setup and rely on the dispatch to take care of it - this is fast when prototyping, but obviously error prone. Also we use a global signal here, since there's only one file selector at a time and we don't try and associate a handler with a specific object.

Reading and Parsing

The actual file load resembles the code we generated before for the C++ level to Image map conversion process, although a little cleaner since we know what we're doing.

In this case the Urho3D File library provides routines for loading correct data sizes from the target file. The only specific piece of code we need is the format mangling to get the heightmap information into the right format.

In this case we extract the height data as

  • Mask out the upper 4 bits for the terrain flags
  • Scale up the 12 bit range to 16 bits by left shifting 4
  • Split out the Upper and Lower 8 bits as the rough & fine height components
  • Form the final color as :
    • Blue for Flags
    • Green as Fine detail (Low Byte of height)
    • Red as Rough detail (Top Byte of height)
  • The final 32 bit value is packed as BGR

The heightmap will use R and G for determining the height, but will ignore the Blue channel, however we will use this to modify the texture later.

This code fragment is:

      uint32 heightval;
      uint32 terrain_flags;

        heightval = data[cursor++];
        terrain_flags = heightval & 0xf000;
        terrain_flags = (terrain_flags << 4) &0xff0000;
        heightval = heightval & 0x0fff;         // Mask top four terrain bits....
        heightval = (heightval << 4) & 0xfff0;  // Scale up to full 16 bit range

        uint32 heightColor_fine = heightval & 0xff;
        uint32 heightColor_rough =  (heightval >>8)& 0xff;
        uint32 heightColor = terrain_flags | (heightColor_fine << 8) | heightColor_rough;
        heightIm.SetPixelInt(y, x, heightColor); //Note X/Y transpose to rotate....

We also restrict the level heightmap size to rendering the in use zones, rather than trying to handle the full I76 80*80 zone map to keep performance sane. So combining this logic with the existing heightmap render gets us a functional level render.

Adding a Debug HUD

Urho3D has a built in debug overlay which can be used to report the current state of the engine and this is trivial to enable, with the code

  debugHud.defaultStyle = cache.GetResource("XMLFile", "UI/DefaultStyle.xml");
And we can then turn this on and off with debugHud.ToggleAll()

Adding a SkyBox

Again, a simple operation which adds a node with a Skybox component to the root scene. The Skybox Model and Material are the stock Urho3D versions.

    skybxnode = thescene.CreateChild();

    Skybox@ skybox = skybxnode.CreateComponent("Skybox");
    skybox.model = cache.GetResource("Model", "Models/Box.mdl");
    skybox.material = cache.GetResource("Material", "Materials/Skybox.xml");

Putting it together

You get this: From the bottom of the bowl in level m01.msn

Modifying the terrain

We've put the terrain flags in the blue channel, and we can use this information to manually tweak the texture we put on the map.

For this version then what we'll try is:

  • Create a new image matching the height map size
  • Fill in this new image with colours based on the terrain flags
  • Create a default stone material
  • Get Texture used for the Diffuse layer of this material
  • Change the source Image data to our coloured bitmap
  • Update the material information

It's worth taking a moment to look at how Urho3d uses Texture Units: By default Urho3D has five predefined texture unit types: diffuse, normal, specular emissive and environment. The stone texture XML has the lines allocating these units for the diffuse and normal map.

In this case we're going to load the texture and replace the Diffuse Image with our own, and we can access the material and get a reference to the relevant texture unit. The Material class defines the textures in the material as an Array of Texture called "textures", and when we load a material then we can reference the target unit with textures[TU_DIFFUSE].

In this case we're actually going to grab it as a Texture2D rather than a Texture type: Texture2D is a subclass of Texture, but it will allow us to set a new Image reference. This is slightly confusing since the AngelScript header doesn't explicitly flag this relationship in the way that the C++ code does, but the sample code demonstrates the use of the subclass type for these cases.

(Note that additional units are available as "desktop only" features, and the units can be used by numeric references and remapped as the Terrain shader does, but that's detail we don't need to go into right now).

To get this change we will replace the terrainblock.material assignment in the level reader with this code: The code is:

Material@ renderMaterial = cache.GetResource("Material", "Materials/Stone.xml");
Texture2D@ inTex = renderMaterial.textures[TU_DIFFUSE];

  newImg.SetSize(msn.FinalHeightMap.width, msn.FinalHeightMap.height, 3);

    for (int srcx = 0; srcx < msn.FinalHeightMap.width; srcx++)
      for (int srcy = 0; srcy < msn.FinalHeightMap.height; srcy++) {
      int v = msn.FinalHeightMap.GetPixelInt(srcx,srcy);
        if ((v & 0xff0000) != 0) {
          if ((v & 0x010000) != 0)
            newImg.SetPixelInt(srcx, srcy, 0x770000); // Rough
          else if ((v & 0x020000) != 0)
            newImg.SetPixelInt(srcx, srcy, 0x007700); // Dirt
          else if ((v & 0x040000) != 0)
            newImg.SetPixelInt(srcx, srcy, 0x333333); // Tarmac
            newImg.SetPixelInt(srcx, srcy, 0xffffff); // Other?

        else {
          newImg.SetPixelInt(srcx, srcy, 0x888888); // Sand
  renderMaterial.textures[TU_DIFFUSE] = inTex;
  terrainblock.material = renderMaterial;
And this gives us the following from the training mission....
Which is less pretty, but more useful in highlighting the terrain type distribution.

Saturday, 15 April 2017

3D Scene Setup and Heightmaps

Minimal 3D application

This is a very (very) simple 3D application - we create a simple plane, a camera and a light and then set it up to render.

The Code

Scene@ thescene;
Node@ cameraNode;

void Start() {
  thescene = Scene();

  cameraNode = thescene.CreateChild();
  cameraNode.position = Vector3(0.0f, 5.0f, 0.0f);

Node@ planeNode = thescene.CreateChild();
  planeNode.scale = Vector3(100.0f, 1.0f, 100.0f);
StaticModel@ planeObject = planeNode.CreateComponent("StaticModel");
  planeObject.model = cache.GetResource("Model", "Models/Plane.mdl");
  planeObject.material = cache.GetResource("Material", "Materials/StoneTiled.xml");

Node@ lightNode = thescene.CreateChild();
  lightNode.direction = Vector3(0.6f, -1.0f, 0.8f);
Light@ light = lightNode.CreateComponent("Light");
  light.lightType = LIGHT_DIRECTIONAL;

  Viewport@ viewport = Viewport(thescene, cameraNode.GetComponent("Camera"));
  renderer.viewports[0] = viewport;

  SubscribeToEvent("KeyDown", "HandleKeyPress");

void HandleKeyPress(StringHash eventType, VariantMap& eventData) {
  if (eventData["Key"].GetInt() == KEY_ESCAPE) {

Taking This Line by Line

Initially we create a reference to the Scene object - every node in 3d space is a child of this. We also create an Octree Component at this level, which must be added to the root scene for the render to work.
The Octree is responsible for accelerating the render by subdividing the global 3D space and rapidly determining the renderable objects. In theory we could optimise this for a given scene, but we need at least one to render at all, so we add a default version to the root scene node.

Next we create the basic objects that will populate the scene: A plane, a light, and a camera. A camera so we we have something to view the scene through, a light so we can see things, a plane so we have something to actually see.

Our node graph is this:

   |         |         |
   |         |         |
[Plane]   [Light]   [Camera]   

And for the three children we have a two part process which is:

  • Create a Node, as a child of the Scene
  • Create the Component within the Node
In these cases we apply the basic 3D transforms (scaling, direction, position) to the Node, and the Component has any specific operations for the type of thing we create (Apply a model, Set the Type of Light, etc). The Material and Model for the plane are taken straight from the default Urho3D resources, available with the distribution.

The Material is provided as an XML file describing the texturing in terms of textures, shaders and rendering passes, and in this case references images which are stored in the DDS format; (a DirectDraw Surface) for the underlying texture images.

Finally we set up a Viewport, passing it the Scene and Camera reference, and then hand it to the global renderer, and it all goes. The KeyDown handler is our basic exit.

We make the Scene and Camera nodes file globals in this example. We have to do this for the Scene node, otherwise the reference drop when we leave Start() will result in deletion and a blank screen. We make the Camera node global so that we can access it from the event handler for the next example.

Spinning The Camera

Although this gives us a 3D view the fixed image does not prove much, but we can spin the camera using a simple event handler. In the Start() function then add an event call on each frame (the Update event).

SubscribeToEvent("Update", "HandleUpdate");
And in the update then we can call the camera Rotate() method:
void HandleUpdate(StringHash eventType, VariantMap& eventData) {
    cameraNode.Rotate(Quaternion(0.0, 0.2, 0.0));
And that'll perform a slow spin

Adding a Terrain Map

This is easy to do using a supplied heightmap image and texture from the Urho3D distribution: Simply remove the planeNode and planeObject and replace these with a call to MakeTerrain() and fill that function in as:

void MakeTerrain()
    Node@ terrainNode = thescene.CreateChild();
    terrainNode.position = Vector3(0.0f, 0.0f, 0.0f);

    Terrain@ terrain = terrainNode.CreateComponent("Terrain");
    terrain.patchSize = 64;
    terrain.spacing = Vector3(2.0f, 0.1f, 2.0f);
    terrain.smoothing = true;
    terrain.heightMap = cache.GetResource("Image", "Textures/HeightMap.png");
    terrain.material = cache.GetResource("Material", "Materials/Terrain.xml");
    terrain.occluder = true;

This creates a Node (terrainNode) and adds a Terrain Component object reference. The Terrain Component has a couple of simple parameters supplied:

  • patchSize: Must be a power of 2 between 4 and 128. This controls the level of Terrain detail generated by the engine, with lower values generating more (finer) data.
  • spacing: The scaling of the map to the world: the Y scales the map vertically (screen height) with X & Z as Width & Height
  • smoothing: Flag to smooth the output Terrain
  • occluder: Use Terrain when calculating Occlusion
  • material: This is a stock texture provided in the Urho3D resources, which consists of a mix of 3 textures
  • heightmap: In this case an 8 bit greyscale image

The source heightmap for Urho3D's terrain system must be square and be of a size of "2^n+1". For the default resource it's a 1025x1025 pixel 8 bit greyscale. Although this case is an 8 bit greyscale map, if the provided image has more channels it will be treated as a 16 bit heightmap with the Red and Green channels providing the 16 bit value when concatenated.

This material actually uses four images - The TerrainDetail1 to TerrainDetail3 DDS files provide the texture image and the TerrainWeights DDS file selects the texture at a location based on the R, G or B channel. In this case we're not too worried about this since we're just using it as a placeholder.

Generating a Higher Resolution Heightmap

This is actually fairly straightforward; generate an Image() instance, set the image to be 3 channel (although anything higher than 2 will work) and set the values in the pixel map on the Red and Green channel for height.

In setting the pixel values R is the lowest byte of the value, and G is the next, so setting up a manual stepping in the output image can be done as follows.

Image@ heightIm;

uint32 height1 = 0x000f;
uint32 height2 = 0x00ff;
uint32 height3 = 0x01ff;
uint32 height4 = 0x02ff;

void MakeHeightImage(int imgsz)
uint length = imgsz + 1;
  heightIm = Image();

  heightIm.SetSize(length, length, 3);
  uint32 heightColor_fine = 0;
  uint32 heightColor_rough = 0;

  for (int y = 0; y < heightIm.height; ++y) {
    for (int x = 0; x < heightIm.width; ++x) {
    uint32 heightval;

      if (y > (7* heightIm.height/10)) {
        heightval = height4;
      else if (y > (6* heightIm.height/10)) {
        heightval = height3;
      else if (y > (5* heightIm.height/10)) {
        heightval = height2;
      else if (y > (3* heightIm.height/10)) {
        heightval = height1;
      else {
        heightval = 0;

      heightColor_fine = heightval & 0xff;
      heightColor_rough =  (heightval >>8)& 0xff;
      uint32 heightColor = (heightColor_fine << 8) | heightColor_rough;
      heightIm.SetPixelInt(x, y, heightColor);
This generates the image which we can request with a call like:
And assign to the material used for the heightmap.
  terrain.heightMap = heightIm; 
And that's it...

Adding Fly Through Controls

This is very straightforward, and this is lifted directly from the sample code: It adds a basic Mouse look and WASD movement. It's invoked during the frame update, rather than the Key Down handler, and uses the input global to scan the keys and mouse and translate this to camera node rotation and position directly.

float yaw = 0;
float pitch =0;
void MoveCamera(float timeStep)
  const float MOVE_SPEED = 50.0f;
  const float MOUSE_SENSITIVITY = 0.5f;

  IntVector2 mouseMove = input.mouseMove;
  yaw += MOUSE_SENSITIVITY * mouseMove.x;
  pitch += MOUSE_SENSITIVITY * mouseMove.y;
  pitch = Clamp(pitch, -90.0f, 90.0f);

  cameraNode.rotation = Quaternion(pitch, yaw, 0.0f);

  if (input.keyDown[KEY_W])
      cameraNode.Translate(Vector3(0.0f, 0.0f, 1.0f) * MOVE_SPEED * timeStep);
  if (input.keyDown[KEY_S])
      cameraNode.Translate(Vector3(0.0f, 0.0f, -1.0f) * MOVE_SPEED * timeStep);
  if (input.keyDown[KEY_A])
      cameraNode.Translate(Vector3(-1.0f, 0.0f, 0.0f) * MOVE_SPEED * timeStep);
  if (input.keyDown[KEY_D])
      cameraNode.Translate(Vector3(1.0f, 0.0f, 0.0f) * MOVE_SPEED * timeStep);
This really is as simple as it looks: grab the mouseMove value and use it to calculate the value we put into camera rotation, then look for W,A,S and D and call cameraNode.Translate() to move the camera position. Timestep is pulled from the event information to ensure smooth movement but that's it. Next up we'll import some actual levels...

Friday, 14 April 2017

Urho3D and I76 Levels

Urho3D and AngelScript

What's Urho3D?

Urho3d is a game engine - it provides a simple method of implementing a:

free lightweight, cross-platform 2D and 3D game engine
The complete Urho3D release is available at the Urho3D homepage on GitHub, and describes the list of features provided by the engine.

What's AngelScript?

AngelScript is the main scripting language of Urho3D, and Urho3D ships with a slightly modified version in the source tree for running script applications.

Although AngelScript has many of the typical features of scripting languages (such as higher level data types, type inference, automatic memory management, etc) it supports a "C++ like" syntax and has a low overhead when binding to the native API.

The bin/Data/Scripts/ directory in the Urho3D distribution contains a set of example files which demonstrate the basic features of the engine, alongside a level editor and sample game constructed in AngelScript. The distribution also includes an Urho3DPlayer application, which can be used to run scripts directly.

Urho3D also supports Lua as an alternative scripting language, but AngelScript is the default and I'll stick with that.

And where are we going?

The idea is to use AngelScript and Urho3D together with the file format information we have on I76 to decode and provide a realtime fly through view of some of the I76 level geometry.

Over the next few posts I'll go over the basics of running up Urho3D - getting things running, the basic application structure, etc. Then the main flythrough application.

Getting Started

Urho3d Absolute Basics

A few basic concepts to keep in mind are:

  • Urho3D provides a set of subsystems for use in game code
  • Scripts access most of these subsystems via global references
    • e.g.: files, logging, network, input, ui, audio, engine, graphics, rendering, scripting, console, scene, octree, physics, resource cache, and time
  • The Urho3D Engine handles the main running loop for the script, and provides events for the script to react to
  • Scripts include each other directly
  • Many user created items are built around "components" and "nodes"
  • Components mainly provide functional game elements
  • Nodes track 3D properties, and contain components or other nodes
  • Nodes are placed in a graph hierarchy
    • This means that nodes can be attached as children to other nodes
    • The base of the Urho3D application is a root scene node
    • Things like lights, cameras, and 3D objects are created as children of this node
  • But not everything is a node
    • e.g. ui elements are not nodes, but have a similar parent/child relationship to each other
  • Urho3D provides an event driven framework where:
    • The script reacts to global events with a set of function hooks (e.g. Start() and Stop())
    • Scripts can subscribe listener functions to events (e.g. "KeyDown" or "Update)
    • Scripts can also generate events
The complete list of scripting objects and types are in the file AngelScriptAPI.h. The events available are described in various header files and can be found with the URHO3D_EVENT macro.

Urho3D has enough minor differences from the default AngelScript conventions that the primary AngelScript documentation is useful, but not comprehensive. The basic rule here seems to be to search for a usage in the example scripts to determine the canonical way of using the scripting language.

The simplest example

This is a basic graphical hello world - it's simpler than the sample example, and uses the minimal possible code. Just save this into a file

void Start() {
    Text@ txt =  Text();
    txt.SetFont(cache.GetResource("Font", "Fonts/Anonymous Pro.ttf"), 30);

    txt.text = "Hello World, It's " + String(time.systemTime);

    SubscribeToEvent("KeyDown", "HandleClicked");

void HandleClicked(StringHash eventType, VariantMap& eventData) {

How does that work then?

You run the example with the Urho3DPlayer application: try a line like

or in a window mode with
Urho3DPlayer -w
Either press a key (or close the window if in windowed mode) to exit.


The player application loads a script file from the command line, and executes the contents.

The primary entry point of the script is the "Start()" function. This is executed by the Urho3D engine. As the official docs say this is called at engine initialization and before running the main loop.

The first line here starts "Text@ txt" which defines an object handle. The "@" designation is used by AngelScript for reference counted object handles. In this case a Text object, the default Urho3D UI text type.

Given a txt object reference we can use a C++ style member reference through a dot operator to access properties and methods. Here we set the font face and size using txt.SetFont().
In this case the font loaded is taken through the global resources cache, which loads from the installed Urho3D directory, under bin/Data/, and uses the global cache reference.
So cache.GetResource("Font", "Fonts/Anonymous Pro.ttf") call is loading the TTF Font from the bin/Data/Fonts/ directory.

Next we set the "text" property, which is the string to render. The time.systemTime() call gives us the current time.

Both cache and time are two examples of the subsystems services provided by Urho3D on initialisation, and available for the scripts to access through global references.

As mentioned Urho3D uses a parent-child set of relationships, and the AddChild() call places the text entry as a child of the root of the ui. (Although you should be aware that these are UI elements, and not Nodes).

Finally SubscribeToEvent() is used to link the events from Key input ("KeyDown") to the function "HandleClicked()". Note that in scripts we use the name of events, which are converted to the identifiers that the C++ code uses, however these are case insensitive at the script layer, so subscribing to "KeyDown", "keydown" or KeYdoWn" will all do the same thing.
For this app then the HandleClicked() method simply calls an Exit(). This exits the entire Urho process and leave us back at the command line.

A Slightly More Verbose Version

This example is slightly more verbose, and resembles the default sample slightly more. It:

  • Uses input.mouseVisible to ensure the mouse remains visible
  • Has a different event handler for the first frame update, then switches for subsequent frames
    • Where the Update event is sent each frame
  • Has a handler called on every update which will drop out a flood of console debug
  • The text handle is a file global and on each keypress we:
    • Exit if the Escape Key is pressed, otherwise
    • update the text colour and position (only seen on the first keypress)
    • change the on screen text as key presses occur

One interesting feature of this version is that we can pull away focus from the main window, and should see the time step report change, since Urho3D will deliberately lower the frame rate when the Application loses focus. The engine SetMaxInactiveFps() can be changed to modify this.

Text@ instext;

void Start()
    Font@ font = cache.GetResource("Font", "Fonts/Anonymous Pro.ttf");
    instext = Text();
    instext.text = "Hello World, It's " + String(time.systemTime);
    instext.SetFont(font , 30);


    input.mouseVisible = true;

    SubscribeToEvent("KeyDown", "HandleClicked");
    SubscribeToEvent("Update", "FirstFrame");

void FirstFrame(StringHash eventType, VariantMap& eventData)
    instext.text += "\nFirst Frame at " + String(time.systemTime);

    SubscribeToEvent("Update", "HandleUpdate");

void HandleUpdate(StringHash eventType, VariantMap& eventData)
    Print("HandleUpdate Called  at " + String(time.systemTime) + "\n");
    float timeStep = eventData["TimeStep"].GetFloat();
    Print("  Time Step " + String(timeStep) + "\n");


void HandleClicked(StringHash eventType, VariantMap& eventData)
    if (eventData["Key"].GetInt() == KEY_ESCAPE)
        instext.text += "\n Keypress at " + String(time.systemTime);
        instext.horizontalAlignment = HA_CENTER;
        instext.verticalAlignment = VA_CENTER;
        instext.color = Color(0.0f, 1.0f, 0.0f, 1.0f);

And that's all for now - next time we'll cover a simple 3D scene setup, nodes and generate a simple 16-bit heightmap...

Sunday, 29 January 2017

Frame Rates and I76 Nitro

A Problem with Old Games on Modern Systems

One of the persistent problems the I76 & Nitro games have had on modern systems is the frame rate.
The game uses the graphics frame rate to control physics and timing. Unfortunately on a modern system the frame rate is much higher than the original authors expected and the assumptions of the engine break. This makes for some odd behaviours, and impossible jumps.

However the I76 community have solved this problem by manually modifying the Nitro binaries to add a frame rate limiter.

Wait, they've done what?!

The Files

There are a few different patch files available on the interstate 76 site.

  • Nitro Patch I
  • Nitro Patch II
  • Nitro Patch III
  • Nitro 30fps Patch

Patches I - III provide updated executables and libraries; they're replacements for the versions shipped with the game.

The 30fps patch file is slightly different; it contains a single replacement nitro executable provided by the developer known as BofH, who is is also responsible for the Patch III release.

The executables in patch II & III are identical, and the 30fps version of executable is based on these.

The Change

There's a couple of changes in the binary, but the particularly interesting one for the frame rate affects the main render loop and modifying some zero data at the end of the file into a block of code, effectively implementing a binary trampoline. This change by BofH, welding into the existing binary, is (IMO) incredibly clever.

Looking at the decompiled binary code using Snowman then the giveaway to locating the completed frame is the frame dump function. This is located at 0x488ed0 and is called from 0x433069. It's easy to spot by the reference to the string literal "SCRDUMP.BMP". It's part of a monster of a function starting at 0x431760, that includes the main render loop.

In this section there's a change at 0x4327c8, where we redirect to the the trampoline.

The original code has a compare followed by a conditional jump

4327c8: cmp dword [0x4f30cc], 0x5
4327cf: jnz dword 0x43311d

The new code is different - it has an unconditional call to 0x4c02a0 replacing the compare.

4327c8: jmp dword 0x4c02a0
4327cd: nop
4327ce: nop
4327cf: jnz dword 0x43311d

So, what's at 0x4c02a0? In the original nitro executable Snowman thinks this is filled with "add [eax], al", which is basically a chunk of binary filled with 0 data.

In the new nitro executable this has been replaced with something more interesting. Snowman actually messes up the opcodes slightly - it should look like this though...

4c02a0: push ebx
4c02a1: push ecx
4c02a2: push edx
4c02a3: call dword [0x4c111c]
4c02a9: mov ebx, [0x4cc4f8]
4c02af: test ebx, ebx
4c02b1: jnz 0x4c02b5
4c02b3: jmp 0x4c02de
4c02b5: mov edx, eax
4c02b7: sub edx, ebx
4c02b9: mov ecx, [0x4c02f4]
4c02bf: sub ecx, edx
4c02c1: mov ebx, [0x4cc4f4]
4c02c7: add ebx, ecx
4c02c9: test ebx, ebx
4c02cb: js 0x4c0300
4c02cd: mov [0x4cc4f4], ebx
4c02d3: push ebx
4c02d4: mov ebx, eax
4c02d6: call dword [0x675139]
4c02dc: mov eax, ebx
4c02de: mov [0x4cc4f8], eax
4c02e3: cmp dword [0x4f30cc], 0x5
4c02ea: pop edx
4c02eb: pop ecx
4c02ec: pop ebx
4c02ed: jmp dword 0x4327cf
4c02f2: add [eax], al
4c02f4: and [eax], eax

The short version:

This loop makes two external calls; one to GetTickCount() and one to Sleep() - it's basically a time check and delay calculation which inserts a delay between frame renders.

The value at 0x4c02f4 is actually a data value of "33" - this sets the optimal frame delay of 33mS, or ~30fps. Changing this value winds the delay target value up or down.

The tail end of this code restores the context and duplicates the test that this modification removed before returning to the jnz, so the subsequent graphics loop processing is otherwise unchanged.

The long version

Let's break this down:

4c02a0: push ebx
4c02a1: push ecx
4c02a2: push edx

Entry context save

4c02a3: call dword [0x4c111c]

This is a call to GetTickCount() - so we retrieve the number of milliseconds.

4c02a9: mov ebx, [0x4cc4f8]
4c02af: test ebx, ebx
4c02b1: jnz 0x4c02b5
4c02b3: jmp 0x4c02de

This recovers a value from 0x4cc4f8 and skips the next part of the processing if it's zero; tracing through the code we can see this value gets the latest timer tick count stored back in it, so it's a simple test for a previous tick value, and obviously if there isn't one then the frame delay calculation is pointless.

4c02b5: mov edx, eax
4c02b7: sub edx, ebx

And this subtracts the current from the previous tick value - i.e. it tells us how many mS between renders.

4c02b9: mov ecx, [0x4c02f4]
4c02bf: sub ecx, edx
4c02c1: mov ebx, [0x4cc4f4]
4c02c7: add ebx, ecx

Looking forwards then there are two values here - 0x4cc4f4 is set to the "previous sleep time", and 0x4c02f4 is a constant value, 33. So the calculation is (mS throughout): "new sleep" = "previous sleep" + (33 - "time difference").
So, ideally "previous sleep" will stabilise at the target inter-frame delay for a given frame render time: think of it as "new delay = delay + error". Carrying on:

4c02c9: test ebx, ebx
4c02cb: js 0x4c0300

This tests for a delay below zero, in which case it jumps to this code fragment:

4c0300: mov ebx, 0x0
4c0305: jmp 0x4c02cd

This sets a zero delay value and jumps straight back. In this case we want to render as fast as possible because we're under the target frame rate.

4c02cd: mov [0x4cc4f4], ebx

Stores the requested delay value in 0x4cc4f4 for the next loop.

4c02d3: push ebx
4c02d4: mov ebx, eax

Sets up the delay and stores off the tick count.

4c02d6: call dword [0x675139]

This is a call to sleep, and so we delay for a time that limits the outgoing frame rate.

4c02dc: mov eax, ebx
4c02de: mov [0x4cc4f8], eax

restores the tick count and saves it to 0x4cc4f8

4c02e3: cmp dword [0x4f30cc], 0x5

This is the test we replaced in the original binary

4c02ea: pop edx
4c02eb: pop ecx
4c02ec: pop ebx
4c02ed: jmp dword 0x4327cf

Restore context, and return to the main loop at the jnz, which leaves the code back in the state we started.

Hacking up the binary some more

The magic number is the data value of "33". We can just change this directly in the new nitro.exe to modify the frame rate up or down.

So opening the nitro.exe in a hex editor and going to offset 784116 we can tune the value to verify we're understanding this correctly.

  • 0x21 => 33mS, or 30 fps
  • 0x28 => 40mS, or 25 fps
  • 0x64 => 100mS, or 10 fps
  • 0xFF => 256mS or ~4 fps

Annoyingly this doesn't appear to affect the -recordLoads debug values, but you can't have everything. For the 10fps values & below the lower frame rate is very visible.

Monday, 23 January 2017

I76 Parser Code

I'd been planning to do a little more to clean this up, but keep getting distracted, so for now here's some code that can do I76 game file translation.

This is a dump of the current I76 exporter as it's sitting on my hard disk. It compiles with Qt5.6 (or better), lzo and opencv on Linux. LZO is needed to unpack the compressed file resources in the Nitro Pack, and OpenCV is used to handle some of the higher bitdepth images that are created when level dumps are processed. Qt 5.6 is needed because of some of the utility bitmap functions I used in QImage. These dependencies should be replaceable for the motivated who only need a subset of the functionality.

It's been built up as I went along figuring out the formats, so it's very hacky in places, and could be much nicer if rewritten from scratch and knowing what's available in which files. However it works for me as is (tm).

Just qmake in the source directory, then make and the resulting binary is i76_converter. Run this with the name of a file to process on the command line.

Run it with the ZFS file from the game itself and it will unpack the individual resource files from the ZFS. If run with one of the resource files from the game (either from the ZFS or game disks) then it will report information and possibly create a more usable export, depending on the type of file (and how much of the debug is commented out).

Monday, 2 January 2017

I76: The Rest of the FSM Processor Op-Codes

The Rest of the Op-Codes

We can run through the rest of the op-codes in the way we described in the previous posting. Skipping over some of the detail the results are:

The Jumps

Op-code 8 (Case 7)

This was the operation we confirmed last time. There are two other jump instructions that our initial guesswork picked up: 9 & 10.

Op-code 10 (Case 9)

This is also an unconditional jump, and the assembler sequence is pretty much identical to op-code 8, but this also stops the running machine by returning from the execution function.

When we halt the execution we appear to go to the next available machine as described in the "Block 1" Machine list we pulled out of the mission file.
So for A01 then the first machine described starts at Location 2180, and is the machine we looked at last time. As soon as we break out of the loop running this machine then we go to run the next machine in the list (which starts at 2238), and following that the next (also at 2238 with a different stack initialiser) etc. The game runs around the list, and then returns to the first running machine and resumes execution. Basically the machines are co-operatively multitasking.

Op-code 9 (Case 8)

We thought this was a conditional jump, and this is confirmed in the code. Examining this code shows us that the value held in "edi+0x10a0" is a reference to the return value location used by actions.

Stack Pop

Op-code 7 (Case 6) is a Stack pop, with the parameter providing the depth to pop. It simply subtracts the argument from the SP. It only changes the pointer value though, and will not clean up the associated memory.

Modify SP

Op-code 6 (Case 5) is a stack modify - the stack pointer is adjusted by the argument value.


This is Op-code 14 (Case 13) and seems to be a complement of the result code. The argument isn't used.

Copy Operations

Op-code 4 and 5 are copy operations, from the "n'th" stack element to a temporary stack, referenced at "edi+0x10b4", and then the "edi+0x10b4" head is moved forward.
Actions consume values from this stack reference, and the value is reset after action calls, so I believe this is actually used as the parameter stack for the action executor.
The main difference between op-codes 4 & 5 is that 5 copes with signed parameters, which loads from below the stack "base". So let's cover the stack layout.

Stack Layout

As mentioned the running machine has a reference to the stack base, and a current stack pointer. In the initial state stack item #0 is set to the first bytecode location for this machine, and stack push moves the pointer upward, pop back downward.

The "below the stack" values are populated using a combination of the Block1 and Block2 tables. The values from block 2 are placed in memory, and the block 1 values are used to select from the list of block 2 entries and stored below the stack base.

As an example a01 has the following machine definition in Block 1:

Block1/1 Start Address: 2238 Initial stack Sz: 6 Stack: [1 7 8 9 10 14 ]

And the corresponding Block 2 in this file is:

Block2 table has 16 (0x10) Entries
0:1 1:2 2:3 3:4 4:5 5:6 6:7 7:3 8:4 9:5 10:6 11:7 12:8 13:9 14:0 15:0

Now when running this machine we see an SP base @ 0x05f62e84, and dumping the memory around this point gets:
Wine-dbg>x/16x 0x05f62e64
0x05f62e64: 04455355 05f6c574 05f6c58c 05f6c590
0x05f62e74: 05f6c594 05f6c598 05f6c5a8 00000000
0x05f62e84: 05f70ba8 00000000 00000000 00000000
0x05f62e94: 00000000 00000000 00000000 00000000

And dereferencing through these numbers we should see the values for indexes "1, 7, 8, 9, 10, 14 ", or looking those up in Block 2: "2,3,4,5,6,0"
Wine-dbg>x 0x05f6c574
Wine-dbg>x 0x05f6c58c
Wine-dbg>x 0x05f6c590
Wine-dbg>x 0x05f6c594
Wine-dbg>x 0x05f6c598
Wine-dbg>x 0x05f6c5a8

Now when we go into the next machine we have the definition
Block1/2 Start Address: 2238 Initial stack Sz: 6 Stack: [2 8 9 10 7 14 ]

And we have the stack in memory:
Wine-dbg>x/16x 0x05f63f2c
0x05f63f2c: 04455355 05f6c578 05f6c590 05f6c594
0x05f63f3c: 05f6c598 05f6c58c 05f6c5a8 00000000
0x05f63f4c: 05f70ba8 00000000 00000000 00000000
0x05f63f5c: 00000000 00000000 00000000 00000000

And de-referencing again we should see the index values of "2, 8, 9, 10, 7, 14" which works out as "3,4,5,6,3,0"
Wine-dbg>x 0x05f6c578
Wine-dbg>x 0x05f6c590
Wine-dbg>x 0x05f6c594
Wine-dbg>x 0x05f6c598
Wine-dbg>x 0x05f6c58c
Wine-dbg>x 0x05f6c5a8


Obviously there's an off-by one thing, where the stack is placed one location in memory below where the values would be if things were tightly packed, but it's close enough that I won't worry (for now anyway).

When looking at these copy operations it's worth mentioning that operation 4 copies a pointer reference to the stack location (not a value), and 5 copies the value, but it expects this to be a pointer (and invariant, since there's no way to change it). The result of this is that the copy operations are pass by reference, not value. So pointers are being passed to the actions, rather than a mix of values and pointers. I don't currently know if any bytecode/actions try to exploit this.


The action executor is op-code 13 (Case 12). This code marshals parameters to the execution function - in this case 0x40d0d0. Inside this function we move forward the bytecode instruction pointer and also reset the "+0x10b4" (parameter) stack.

One important note is that the case numbers for actions don't (directly) match the values supplied in the bytecode. There appears to be a function at 0x0040bf40 called during the mission load which maps the action names to a jump table, and the values in the case reflect the mapped values in this table. So for example the op-code:arg pair of "13:52" represents "ACTION(true)", but this is actually mapped to and handled in case 25 of the execution function.
(This function was probably called match_prototype(), given that's in the error string it reports when it fails).

The internals of the action executor are a little involved, so any more detail will have to wait for a separate post.

Op-code 12: RST

This is a slightly unexpected/odd op-code; it appears to change the running bytecode pointer to the default location and reset the stack pointer & base, and can also break out of the running loop. It looks to be used to halt the execution of a running machine.

My immediate suspicions are that this is more often a Halt than a Reset. More investigation into the use cases is required for this one....


Memory Layout

We have a stack based machine which has loaded the microcode into memory as 4 byte opcode & 4 byte argument pairs and is used to run the mission scripts.

The Stacks themselves are 4 bytes per entry.

The main stack has a base location, containing the machine start address, and grows upward. Initial parameter values are loaded below the stack base and can be referenced with a stack copy taking a negative argument. Stack Pop and Push is non destructive; the pointers are adjusted but any values in memory are not cleared.

There is an argument stack for passing data into actions, and a dedicated Result value which can be tested.

Each running machine tracks:

  • The Last Action Result (AR)
  • The Base of the loaded microcode (IB)
  • The Current Instruction Pointer (IP)
  • The Base of the Stack (SB)
  • The Current Stack Pointer (SP)
  • An Argument Stack for passing to Actions (AS)

The game keeps a list of machine details loaded from the mission file, and loops through this list, with each machine being run in turn. A machine relinquishes control using a suitable op-code, but the engine also has a harcoded upper limit on the number of instruction loops a machine can perform without being de-scheduled.

OpCode Summaries

What we have so far:
Opcode Name Description
1 PUSH Stack Push. Argument written at SP, Increment SP
2 Unused
3 Unused
4 COPY_S Copy Argument From Stack. Copy from "BP+argument" to AS
5 COPY_B Copy Argument from Parameter. Copy from "BP + argument-1" to AS. Negative Arguments Expected.
6 STACK_MOD Add Elements to stack. SP = SP + argument.
7 POP POP Elements from stack. SP = SP - argument
8 JMP Unconditional Jump. Set IP to argument
9 JZ Conditional Jump. If AR is 0 then set IP to argument
10 JMP_I Unconditional Jump and De-schedule. Set IP to argument, then halt the running instance
11 Unused
12 RST Reload IP and Reset SP
13 ACTION Perform Action given by argument. Consumes AS and Modifies AR
14 NEG Complement AR
15 Unused

A View from Tutorial Land

The first machine in the tutorial mission, A01, sets up the camera views for the introduction: Let's break that down some.

Initial Stack State

Initially the stack has the base of this machine bytecode, but is otherwise empty.

The sub-stack region is
[-1: 0]
[-2: "idx 0"]

Where "idx 0" is a pointer to the value "1", from the initial value list.

Running the bytecode

The bytecode opens:
Block3/2180 PUSH: 0x0:[0]
Block3/2181 PUSH: 0x0:[0]
Block3/2182 PUSH: 0x1:[1]
Block3/2183 STACK_MOD: 0x1:[1]
Block3/2184 JMP: 2185

So after this we have the following stack:
[0: 0x05f709d8]
[1: 0]
[2: 0]
[3: 1]
[4: 0]

Next up:
Block3/2185 ACTION(true)
Block3/2186 JZ: 2191

i.e. A call to true, followed by a JZ. The true will set the Action Result register to "1" and therefore the JZ will not trip and we will not take the jump.

The logic of this instruction sequence escapes me, since we just seem to be introducing an expensive NOP and there's no way to hit the JZ without having called true. However having seen the underling engine ASM in action I'm pretty confident this is what it does.

Next we make an argument copy:
Block3/2187 COPY_S: 4

And as a result have the argument stack:
[0: Stack #4]

This is followed by an action call:
Block3/2188 ACTION(startTimer)

This will start the timer referenced by the argument stack (Timer 0?), and following this the argument stack is cleared.

And next
Block3/2189 ACTION(pushCam)
Block3/2190 JMP: 2193
which is an action to setup a Camera, and an unconditional JMP to 2193

Checking for Timers and Keypresses

Going to 2193 we see
Block3/2193 COPY_S: 4
Block3/2194 PUSH: 0xa:[10]
Block3/2195 COPY_S: 5

Which leaves us with the stack of:
[0: 0x05f709d8]
[1: 0]
[2: 0]
[3: 1]
[4: 0]
[5: 10]

And the arg stack of
[0: Stack #4]
[1: Stack #5]

Using this setup we check the time
Block3/2196 ACTION(timeGreater)

This will adjust the result register. Based on the parameter stack it looks like we're waiting for 10 seconds.

Block3/2197 POP: 0x1:[1]

This is a bit of clean-up which removes the timer parameter from the top of the stack. It does not affect the result register and the stack is now:
[0: 0x05f709d8]
[1: 0]
[2: 0]
[3: 1]
[4: 0]

For now following the case that the timer has not expired the result register will be zero, and the next instruction:
Block3/2198 JZ: 2202

Is triggered and goes to...
Block3/2202 ACTION(isKeypress)
Block3/2203 JZ: 2205

A keypress check. If we trip this (i.e. do not take the jump) then the next instruction is a "RST", so this is the "interrupt the view" handler.

Setting up the View

However if we go with a zero result, and have not pressed the key, then next up is a process of push/copies
Block3/2205 COPY_S: 1
Block3/2206 PUSH: 0x118:[280]
Block3/2207 COPY_S: 5
Block3/2208 PUSH: 0xff38:[-200]
Block3/2209 COPY_S: 6
Block3/2210 PUSH: 0xfa:[250]
Block3/2211 COPY_S: 7
Block3/2212 PUSH: 0xf830:[-2000]
Block3/2213 COPY_S: 8
Block3/2214 PUSH: 0x0:[0]
Block3/2215 COPY_S: 9
Block3/2216 PUSH: 0x38a4:[14500]
Block3/2217 COPY_S: 10

So after this lot we have a stack of:
[0: 0x05f709d8]
[1: 0]
[2: 0]
[3: 1]
[4: 0]
[5: 280]
[6: -200]
[7: 250]
[8: -2000]
[9: 0]
[10: 14500]

And the arg stack of
[0: Stack #1]
[1: Stack #5]
[2: Stack #6]
[3: Stack #7]
[4: Stack #8]
[5: Stack #9]
[6: Stack #10]

And it passes this to:
Block3/2218 ACTION(camObjDir)

Which looks a lot like a camera view setup. Basically a camera position and camera view as a pair of triplets.

Clean-up, and sleep

Next is clean-up. We drop the view parameters we pushed onto the stack in the last step:
Block3/2219 POP: 0x6:[6]

So the stack goes back to
[0: 0x05f709d8]
[1: 0]
[2: 0]
[3: 1]
[4: 0]

And we have the "halting jump"
Block3/2220 JMP_I: 2193
This goes back to the timer check location, and pauses this machine while the game engine goes on to the next one.

What happens on timer expiry

The logic for this case is similar to the logic we have followed so far; When the initial timer expires (the check at 2198) the bytecode calls into the startTimer action to ensure the timer is running and then checks for the timer value against a 15 second limit or a keypress to halt. Otherwise it uses the camPosObj action to focus the Camera view onto the Piranha and loops again. The arguments are marshalled using the copy operations we've seen so far, and also there's an instance of COPY_B being used to grab the supplied argument to determine the target object ID.

However this post is getting a little long, and the detail isn't particularly informative over the instructions we've seen so far, so I'll skip it for now.

So, In Summary

This code sets an initial camera view on the level, and runs a timer. It then holds the view until either the timer expires, or a keypress comes in.

When the initial timer expires then the bytecode changes the camera view to focus on the Piranha, and holds this view until the new timer limit (at 15 seconds) or a keypress is seen.

When a keypress is seen, or all the timer expiry cases have been run through, this machine halts.

All of this ties up with what we initially see in terms of cutscene camera management: the view of the sign being held for ~10 seconds, then a timed switch to a view of the Piranha for ~5 seconds. The views either timeout, or we hit a key at which point we wind up in car.

To verify some of this we can copy over a01.msn to ADDONS, edit the parameters pushed to camObjDir, and change the initial views around. For example: Edit the a01.msn at 79268, and change 0x38 to 0x28 for the initial view to the left of the sign and 0x48 to move right. This is changing the argument pushed at Block3/2216. Or raise or lower the value at 79019 (0xa) to change the timeout on the initial view, etc.

So where does all that leave us?

The remaining big unknown it the behaviour of the actions, and I need to dig into the actions list more to try and get a basic understanding of the API available there and the parameters. But this is a lot of the information we need to edit missions and debug them, or at least start to build the tools to do so.