Timing

Video games have to prepare a new frame every 16 milliseconds if they want to keep up with the monitor, which typically runs at 60 hertz. That’s a lot of cpu cycles, but it’s not so much of anything else.

Movies operate at 40 milliseconds per frame (24 hertz) and people tend to ask why games should run faster than that. 40 milliseconds were not exactly chosen because they are optimal, it just happened for historical reasons. People are now so used to it that it is hard to change (see the reactions to the recent Hobbit movies which were recorded with 20 millisecond frames), but tests showed that some people can distinguish up to 90 frames per second. Also, movies have an advantage in comparison to video games that make them look more fluid at the same framerate: motion blur. Cameras do not record a single event in time, they record a time period. Fast movements are therefore blurred on the single frames, which makes it look more fluid to humans. Video games though typically render single events in time, which simplifies calculations a lot. This is changing though, more recent games apply different algorithms to render motion blur approximations. What will not change though: Games are interactive and the frame time also determines how fast a game can react to user input. Higher framerates are therefore especially important for fast action games.

Lag

One topic that generates endless discussions online is lag, i.e. a non-instantaneous reaction of the game to user input, and players usually get really upset about it. Different causes of lag can be adding up:

  • The time it takes the controller to send a button press to the PC (especially if it is wireless)
  • The time it takes to compute a reaction
  • The network time for online networked multiplayer games
  • The time it takes to render the frame
  • The time it takes to show the frame to the user (see the discussion about buffering below).

The impacts of lag depend on the type of game. Especially twitch-speed games such as shooters require lag to be as low as possible.

For VR, lag can also lead to large discomfort. If our vision drags behind our head movements, our brain tries to find an explanation - usually an unpleasant one, such as being drunk, on drugs or seasick. To counteract this, the lag in a VR game between a movement and the resulting image change should be in the area of 20 ms or below.

Buffering

In the previous chapter, we explained that monitors use the vblank period to update the image. If the image is currently being written to, we get tearing. The common solution to this is buffering. In the simplest case, we have double buffering: We are switching between two buffers into which we are rendering.

Adding more buffers can make sense. For example, a swap chain is a cycle of buffers that are used for switching. In a non-interactive but computationally intensive application like video decoding, it can make sense to calculate several images in advance. Of course, in a game or VR application, this can add tremendous lag.

In games, the most common solution besides double buffering is triple buffering: We have three buffers. Depending on the way the graphics driver realizes this mode, it can be advantageous. We discern two modes, using the naming used for the Vulkan API:

  • Mailbox/LIFO: In this mode, whenever we have a frame to render, it will replace the oldest frame in the buffer.
  • FIFO: In this mode, we have a queue of frames to be shown. The frame at the front of the queue is always shown next, and new frames are added at the end if there is space.

The advantage of the Mailbox mode is that, if we have a newer frame, we can show this one, reducing lag. At the same time, we will always have an image to show. This means that the game code will never have to wait for a vsync.

Multithreading

When all game calculations happen fast enough, there is still one additional problem – doing the right thing at the right time. Like operating systems games have to schedule a lot of events. The correct events have to show up at the proper frames.

Fundamentally, games apply a procedure called cooperative multithreading to take care of the multitude of moving game objects. That just means that every game object is called, does its thing and returns. Modern operating systems in contrast apply preemptive multithreading – different processes are called one after another, but the processes do not return themselves, instead the operating system takes control back using timing interrupts. This has the advantage that a process that hangs does not hang all of the operating system (this actually happened in MacOS up to version 9). But preemptive multithreading also has disadvantages: It’s slower (switching threads has some overhead because all current, implicit state like registers has to be saved), needs more RAM and needs proper multithreading synchronization. Multiple CPU cores can only be used using preemptive multithreading, which is on the other hand also a performance plus. Games use preemptive threads for performance critical subsystems (for example the physics pipeline). Games do allmost never use preemptive threads for high level game logic, although cooperative multithreading can be somewhat annoying to implement because every component has to take care of saveing it’s state itself when it ends its current run. Preemptive threading looks easier at first, but that changes rapidly when the first synchronization problems show up. It especially makes no sense to have one thread per game object - proper data synchronization will absolutely kill performance for typical game object counts.

Frame time

During actual game object update calculations using the actual current time should be avoided. Real time always proceeds and game objects will consequently work with slightly different times during the calculation of single frame. Objects will start to jitter as everything gets out of sync just a tiny little bit. Instead of using actual time calculate a virtual time based on the real time once per frame that stays constant during the frame.

Animation types

When a proper time value is calculated it just has to be applied to all game objects. But that can be tricky, too. There are generally two kinds of animations you will encounter in games: Functional and iterative animations.

Functional animations take in a time parameter and return a transformation (or whatever the animation needs). They remember no data. Choosing the proper state of a functional animation at a given frame is trivial: Just put in the current virtual time. Functional animations are very elegant and can also be very efficient. A typical example of a functional animation is a tree that rotates slightly using a sinus function to simulate wind movement. Typically functional animations internally work between values of zero and one, making it easy to multiply several sinus and expoential functions while still staying in the same value range. In a last step the result is multiplied with a constant to create a value in the range the game expects.

However functional animations are also rather difficult to create and mostly useless for interactive content. Interactive content behaves based on user input which is a constantly growing data stream. Putting in all of the previous input data wouldn’t be very efficient, consequently interactive animations are calculated iteratively.

Game Loop

Video games generally work by running a specific kind of endless loop, the so called gameloop. One iteration of the gameloop generates one frame that is displayed on the monitor. To do that, each gameloop iteration involves the following steps:

  • Read current data from all used input devices.
  • Calculate the next game state using game state from the previous frame and current input values.
  • Render a new frame based on current game state.
  • (Wait for VSync)

The interesting part of this is the game state. This is typically a list of game objects which each have properties like a position, orientation, speed and acceleration. Drawing a frame based on this data can be as easy as drawing an image for every game object at an offset based on its current position. It can get a little harder as one might have to incorporate rotations, camera views and 3D projections but even that is usually a straight forward process that can be performed sequentially for each game object. Advancing the game state can be as easy as for each game object adding the current acceleration to the current speed and adding the current speed to the current position. It can however become very complicated, when collissions have to be considered.

Generally when a collision between two objects is detected at a certain point in time, at that point in time both objects intersect each other. But a moveable game object (for example a ball) and a static game object (for example a wall) typically are not supposed to ever intersect. Therefore when the collision is detected the game tries to resolve the collision by moving the ball out of the wall and leaning it on the wall. When multiple game objects are involved this leads to little errors, because the movement of each object is calculated one after another while in reality everything would happen at once. Collision resolution therefore is just an approximation of what is supposed to happen. This approximation works better with smaller time steps. Therefore game logic usually works in fixed time steps which are independent of the actual framerate - otherwise game logic would work different depending on the speed of the computer it runs on. Those differences can stem from collision resolutions but also from more subtle things like rounding errors.

Even more complicated are collisions of two moving objects that should not intersect each other. Older games use a very simple trick to handle that e.g. “if (mario.collidesWith(koopa)) { gameover(); }”. But for things like moving platforms the game has to remember and update contact points - generally speaking game physics can become arbitrarily complex and the physics lectures will show how even that can be handled.