Simulation Time — Falling Behind and Catching Up

← Threading | Networking Overview | Known Issues | Home

Status: Draft for review. Nothing here is implemented. The questions that need an answer before code are in DecisionsDecisions.


The Problem

The simulation runs a fixed step, 512 times a second. When the Brain cannot finish its steps in time — a stall, a slow machine, a burst of rollback — it owes real time. Today it repays up to 250 ms of that by running extra steps and drops the rest (Known Issue 35). Simulated time is then behind the clock for good: five seconds is no longer five seconds, and two machines under different load are at different moments.

Across a network the same gap shows up as freezing. An Owner that falls behind pushes the Authority into stalling the simulation for everyone (Known Issue 17), and nothing ever pulls a drifting Owner back (Known Issue 19, Networking Without Rollback #4).

The Idea

Time owed is kept, and there are several ways of paying it back. Each is a mechanism a game turns on or off and tunes; they work together, each doing what it is good at; and which of them a game wants depends on the game. The engine's part is to make every one available, to keep them from undoing each other, and to show at all times how far behind the simulation is and what is being done about it.


What Is Owed

RoleThe simulation should be atBehind meansAhead means
Solo, Host, AuthorityThe frame the wall clock says: time since the session began, in stepsSteps the clock has passed that have not been runCannot happen
OwnerThe Authority's frame plus the input lead it was givenIts input will arrive lateIts input arrives earlier than needed: latency it did not have to pay

Debt is counted in whole steps against that reference, not built up from per-loop times that are each clamped: the clamping is what loses time now. Being ahead is debt with the other sign, and the same mechanisms pay it.

A gap that is not lag — a breakpoint, a laptop asleep, a window held in a modal loop on a platform with no keep-alive — would otherwise be repaid as ten minutes of simulation. Past a configurable length a gap is treated as a suspension and the reference is moved up to now.


The Mechanisms

#MechanismWhat it doesEvery frame still the same moment on every machine?Good for
1PacingRuns the simulation a little fast when behind and a little slow when ahead, within set limits, by an amount that grows with the debt. Nothing is skipped: more or fewer whole steps per secondYesEverything, always. Small debts vanish unseen; an Owner stays in step with its Authority instead of anyone freezing
2Quiet framesA catch-up step does the simulation's work and nothing else: no frame published for drawing, no uploads, no effects or sounds started. Only the last frame of a burst is shown. It is what a rollback's resimulated frames already areYesMaking catch-up cheap, so it finishes sooner and takes less from the display
3Coarse cosmetic stepsWhile catching up, entities that are only for show (the Volatile tier: particles, ambient movers) take one long step for many frames; authoritative ones take every stepYes, for everything authoritative. A game turns it on knowing nothing that matters reads those entitiesScenes that are mostly decoration
4Variable stepCatch-up frames cover more time each: fewer, longer steps for everythingNoBuilds without rollback: a game alone, and networked play once it is timestamp-based rather than frame-numbered (Non-Deterministic Mode)
5ResyncAn Owner too far behind to catch up stops simulating the past and takes the Authority's state as of nowYes, afterwardsA long stall on one Owner. Needs the snapshot path that host migration and late join also need
6Shared slowdownAn Authority or Host that cannot keep up lowers the session's rate and says so. Owners follow, because what they follow is its frame countYes: everyone slows togetherA session whose host is overloaded for longer than pacing can absorb
7Dropping timeWhat happens today, kept as the last resort: the reference is moved forward and the time is goneYes, but simulated time no longer matches the clockWhen nothing else is enabled or enough

Pacing is the one that changes what the others have to do. With it, an Owner a few frames behind speeds up by a few percent for a moment rather than the Authority waiting for it, so the hard limits can sit far further out than 250 ms: they become a matter of how much history is kept, not of how little lag can be tolerated.

How They Work Together

Debt is read once per loop and decides what is in use. The bands are settings; these are the proposed defaults.

DebtIn useWhat a player sees
Under a frame or twoNothingNothing
Small (up to ~100 ms)Pacing, gently (a few percent)Nothing
Medium (up to the cutoff)Pacing at its limit, quiet frames, and whichever of coarse cosmetic steps and variable step the game has enabledThe world runs visibly fast for a moment
Past the cutoffThe role's last resort: an Owner resyncs; a Host or Authority starts a shared slowdown; Solo slows down, or drops time if the game prefersA jump, or everything a little slow
Ahead (Owner)Pacing, slowingNothing

The cutoff is a setting. It can be seconds, where today's is a quarter of one, bounded by how much history a rollback build keeps and by how long a game is willing to have the world run fast.

What the Game Sets

One block of settings, in the project's configuration and changeable while running (a menu may want different behaviour from a match):

struct SimTimePolicy
{
    // 1. Pacing
    bool  Pacing        = true;
    float MaxSpeedUp    = 1.25f;   // the fastest the simulation may run to catch up
    float MaxSlowDown   = 0.90f;   // the slowest it may run to let an Authority catch up to it
    float PacingGain    = 0.5f;    // how hard it pushes: share of the debt repaid per second

    // 2-4. Cheaper catch-up
    bool  QuietFrames    = true;
    bool  CoarseCosmetic = false;
    bool  VariableStep   = false;  // refused in a build with rollback
    float MaxStepScale   = 8.0f;   // the longest a variable step may be, in fixed steps

    // 5-7. Past the cutoff
    float CutoffSeconds     = 2.0f;
    float SuspensionSeconds = 5.0f;   // a gap this long is not lag: start again from now
    PastCutoff OwnerDoes    = PastCutoff::Resync;
    PastCutoff HostDoes     = PastCutoff::SharedSlowdown;
    PastCutoff SoloDoes     = PastCutoff::SlowDown;      // or DropTime
};

A setting a build cannot honour is refused where it is set, with the reason, not ignored.


What Follows From It

  • Input lands on the frame it belongs to. Sentinel goes on polling through a Brain stall, and input is already a timestamped stream. With frames tied to a reference clock each has a span of real time, so what the player did during a stall is replayed at the moments they did it, not piled onto the first catch-up frame.
  • Rollback is the same debt. Time spent resimulating is time owed; it is repaid by the same mechanisms and shows in the same number.
  • Drawing is undisturbed by quiet frames. Encoder interpolates between the last two frames it was given; during a burst it gets the last one only. Under pacing the world runs a few percent fast or slow, evenly.
  • What is not simulation follows real time. The fallback free camera moves by real elapsed time, not by steps run, so looking around stays smooth while the simulation struggles.
  • It is all visible. Debt, the rate the simulation is running at, and which mechanisms are in use are engine statistics, shown by the overlay. The resilience demo gets a control for each setting, so the difference between them can be seen by turning them on and off under the same load.
  • The tests change. Stress_BrainHitch and Stress_BrainOverrun assert today's loss of time as the policy. They come to assert each mechanism: a long stall repaid in full under pacing; a slow Brain reaching the cutoff and doing what the role says; a step that is variable only where it may be.

Build Order

StepWorkTest
1Debt counted in steps against a reference clock; the suspension rule; the cutoff as a setting; the statistics. Behaviour otherwise as today (catch up flat out, drop past the cutoff)A stall under the cutoff loses nothing at any length up to it; a gap past the suspension length is not repaid
2Pacing for Solo, Host, and Authority; the free camera on real timeA 600 ms stall is repaid in full, at no more than the set rate; the camera travels the same distance in five seconds whatever the Brain is doing
3Quiet framesA burst of catch-up publishes one frame; the same final state as without
4Input placed by timestampA key pressed and released inside a stall acts for as long as it was held, at the time it was held
5Variable step, in builds without rollbackOff by default; refused with rollback; with it on, an overloaded Brain keeps simulated time on the clock
6Coarse cosmetic stepsAuthoritative state identical with it on and off
7Pacing for an Owner against its Authority, with continuous clock sync; the Authority's stall becomes a lockstep-only settingAn Owner made slow stays in step; nobody else's simulation waits for it
8Shared slowdown; resync (with the snapshot path)A Host overloaded past the cutoff slows every Owner evenly; an Owner stalled past it rejoins the present

Steps 1–6 need no network. Steps 7 and 8 come with Host, Join, and Solo.


Decisions

Proposed. Each needs a yes, a no, or a different answer.

#QuestionProposed
1Every mechanism is a setting a game can turn on, off, and tune, at run time, in one policy blockYes
2Debt is counted against a reference clock and never clamped away; dropping time is one of the things a game may choose past the cutoffYes
3Variable step is available only without rollback, and across a network only in the timestamp-based modeYes: with rollback, frame N has to be the same moment everywhere
4Defaults: pacing on at up to 1.25× and down to 0.9×, quiet frames on, coarse cosmetic and variable step off, cutoff 2 s, suspension 5 sTo be tuned in the demo
5Past the cutoff by role: Owner resyncs, Host and Authority slow the session together, Solo slows downYes
6The Authority stalling for a late Owner becomes a lockstep-only setting once Owners pace themselvesYes
7Order: the local mechanisms (steps 1–6) before the networked onesYes: they can be built and tested in the demo now

Open Questions

  • How pacing and a display's refresh interact: a simulation at 1.25× under a 60 Hz display is a visible speed-up; whether the limit should tighten when the debt is small enough to hide entirely.
  • Physics under variable step: Jolt is stepped once per eight fixed steps today. A longer step needs more collision sub-steps to stay stable, which costs back some of what was saved.
  • What coarse cosmetic steps mean for a Volatile entity with a physics body.
  • Sound during catch-up: what was triggered in quiet frames and is still relevant when the burst ends.