Where the
maths lives.

A physics solver built for batched simulation.

I could have used the engine's constraint solver. But training means simulating the same creature an enormous number of times, and a general-purpose solver spends most of its budget on generality I don't need. So I wrote a reduced-coordinate Featherstone Articulated Body Algorithm from scratch in C++.

Featherstone is one of those algorithms that feels like a magic trick the first time it works: it gets you forward dynamics in O(n) without ever building the mass matrix — three sweeps over the tree and you're done. The topology is a floating 6-DOF root, then 3-DOF ball joints and 1-DOF yaw joints — 41 bodies, 68 degrees of freedom.

Then I profiled it, and the part that sounds hard turned out to be the cheap part. With 64 creatures in flight, a step spends about 57% of its time in contact, 30% in damping, and 12% in the forward dynamics — the constraint solve costs roughly 4.6× what Featherstone does. Forward dynamics stays somewhere around a tenth of the frame at every batch size I measured, so it isn't an artefact of the batch size; it's just where the work is. That's 200 timed steps after 20 warmup ones, on my config.

From individual constraints to a global system.

My first constraint solver went row by row, Gauss-Seidel style. It worked, in the sense that it produced numbers. It also meant every constraint was solved in ignorance of the others, and a foot on the ground had no idea a knee was at its limit.

So I threw it out and assembled the whole thing globally — ground contacts, joint limits and limb self-collisions in one system, solved with Projected SOR. The neat part is that I never have to form the mass matrix to build it: I already had an impulse-response query lying around from the ABA, and that's exactly what each entry of A = JM⁻¹Jᵀ + R needs. That's the equation. Told you I was proud of it.

I expected the rewrite to cost me performance in exchange for correctness. The global solve came out 1.57× cheaper than the per-row path — while doing 64 solver sweeps against the old path's 16, because a dense sweep costs no tree traversals. It also cut peak penetration by more than a third and held the torso nearly half again as high. To be clear about the claim: that's the constraint solve, not an end-to-end framerate number, measured on my config.

A few solver features along the way didn't survive contact with reality. They live in the bug journal, so this page can stick to what's actually running.

Learning to stay upright before learning to walk.

The learning layer is my own PPO, written in PyTorch: clipped surrogate, GAE, KL early-stop, advantage normalisation, value clipping. No RL framework underneath it, which means the numbers that tell you whether PPO is actually healthy — loss, KL, entropy — are right there for me to read. If it's wrong, it's wrong in a way I can see.

It runs as a separate process. Python trains, a headless Unreal commandlet simulates, and the two talk over TCP on loopback with a small binary framed protocol.

What the policy sees grew a lot. It used to be 228 floats — a torso summary, joint angles and velocities, contacts. It's now 991, assembled from blocks I can switch on and off independently: root state, full per-body state for all 41 bodies (615 floats on its own), joint positions and velocities, the muscle activations, contacts, a small airborne block, and the objective. It used to infer the shape of its own body from joint angles. Now it's simply told — including the state of its own muscles, which closes the loop back to the previous section.

Two details in there matter more than they look. Rotations go in as a 6D continuous representation rather than quaternions or Euler angles, because the tidy encodings have discontinuities a network has to spend capacity learning its way around. And the observation clamp — which squashes every channel into a fixed range — used to be 2.0, which the height and velocity channels hit constantly in the air: across its whole standing range, the creature read a flat 2.0. It was blind to how high it was, in the most literal sense. Raising the clamp to 10 and adding an explicit airborne block — ground clearance, time to impact, a flag — handed that information back. That one has its own bug entry too.

The layout is also hashed into the handshake now, so a policy can't quietly be loaded against an environment that arranges its observations differently. That mistake produces a creature that looks broken rather than one that errors, which is the worst kind.

The reward says reach the target height, keep your limbs on the ground, hold a sensible pose, and stop flailing — that last one is a torque penalty plus a cost on rapid changes in muscle intensity. Without those, and without smoothing activation before it reaches the solver, you get a creature that solves standing by vibrating. Which it did, for a while.

Two policy architectures share the same trainer: a 256×256 MLP, and a pure-PyTorch Mamba with recurrent PPO. The recurrent one is meaningfully faster early — roughly 20–30% fewer samples to reach the first survival milestones — and then the two converge once the episode cap starts to bind.

How fast does it actually learn?

The run behind the clip at the top of my front page is 1346 iterations, and an iteration is 8192 env-steps — a 128-tick horizon across all 64 creatures. That's 11,034,624 simulated steps, and about seven hours on my config, spread across three sessions because I restarted it twice.

The update's share grew as the run went on, which took me a moment to understand and then made complete sense: PPO stops early once an update has moved the policy too far, and late in the run each pass was moving it less — so six epochs fitted under the same KL budget where four had before. The policy wasn't static; it was still changing from iteration to iteration, just taking smaller steps inside each update. The better behaved the learning gets, the more of it you pay for.

The genuinely annoying number is idle time. The two sides strictly alternate: collect the full 128 steps, then run the update, then collect again. Nothing is double-buffered. So Unreal spends 37% of the run doing nothing, waiting for Python to finish. That isn't an estimate either; the server counts its own waiting. It could sustain 8.5 steps per second and delivers 5.4. Overlapping collection with the update would get most of that third back, and I haven't done it yet.

One honest note on the numbers above and the ones further up this page: the solver costs meaningfully more per step during training than it does in the benchmark. They're not the same measurement. The benchmark runs a deliberately clean configuration — no drag, no friction, random torque babble — while training runs the real physical setup with a policy that spends its time in exactly the contact-heavy states a constraint solver finds expensive. Same code path, different worlds.

Launching, watching and replaying all of this happens through the editor, and that's on the tools page.

Every fix used to quietly break two older ones.

This was the loop I kept falling into: change something, fix the thing in front of me, silently break something I'd already fixed last week, and only notice much later once three of those had stacked up and the whole thing was on fire. Then I get to work out which of the last twenty changes did it. Not fun.

A physics system makes that worse, because a broken change still produces motion that looks perfectly convincing. Your eyes will cheerfully sign off on nonsense. So now there are tests, they run on their own, and I find out I've made a mistake in seconds instead of a fortnight.

The ones I trust most are invariant tests. Momentum has to be conserved whether or not the result looks nice, so I assert on that rather than on my own judgement. One of them hangs the real rig a kilometre up, hits it with full-strength random torques, and checks that its momentum changes by exactly gravity and nothing else.

The solver also has to pass a stability gate before anything trains on it: 50 seconds of continuous ground contact with all 41 bodies, under a passive drop and under random torque babble, with no divergence and no NaNs. That sounds unimpressive until you've watched a solver explode at second three for a fortnight. It is also, as the creature later demonstrated, not at all the same thing as the physics being right.