A field guide to "it fell over again".
Each entry is the same shape: what it looked like, what it actually was, and what it taught me. And yes, every tag below says fixed. For now.
AGENTSOLVER · BUG JOURNAL · PART 4 OF 4
Every bug looks
exactly the same.
The creature falls over. That's the whole error message. This is the list of what was actually behind it each time — plus the one bug that presented as the complete opposite.
// the index
// entry 01
Fixed
It learned to fly.
Every other entry on this page is some flavour of "it fell over". This one is the exact opposite, which is exactly why it's my favourite.
First, joint welding. When a leg is folded hard against its joint stop and a muscle is still pushing into it, the solver treats that joint as rigid for the substep. This simplified the solver a lot, and it helped keep things stable during the early stages of building it. Good idea.
The bug. Welding only happens inside the constraint solver. The main dynamics pass never welds. So the solver works out impulses for a stiff creature and then applies them to the real, floppy one — and those don't add up to zero momentum. On the ground, the floor soaks up the difference and nobody notices. In mid-air, every force is internal, and the error becomes thrust. That's my best explanation, and the numbers below back it hard.
The motive. I wanted the creature to stand as tall as it possibly could, so I pushed the height target higher than it could ever reach. Maybe I underestimated the learning process, haha.
I ASKED FOR STANDING
That morning, I saw that the policy was reproducing the default standing pose nicely, but it was struggling with the height it reached. So I thought: maybe I can just set every other reward to near zero and push height really high.
Going back to my little run. First thing: a very good learning curve. It had successfully learned. I was very happy, so I launched the editor to see what it actually looked like... It was flying. More specifically, it would get on its feet, jump, then crawl through the air. I was stunned, and a little proud of it. Then came the debugging.
First hypothesis. Being in the air massively narrowed down the possible sources of this infinite energy. After some value debugging, I found something that looked like the root cause: the welding.
| Switched off | cm/s² |
|---|---|
| Nothing, as shipped | +457 |
| Joint-limit rows (back to the old clamp) | +537 |
| Old per-row solver instead | +359 |
| Limb self-collision | +292 |
| Joint welding | −15 |
| Welding, limit rows and limb collision together | −228 |
Welding off takes it from climbing to barely hovering. Welding was supposed to be a solution, and it had become the problem.
Status: fixed. Welding is gone from the solver. And because a bug like this never shows up as a crash, I also wrote a test for exactly this situation: the real rig, a kilometre up, full-strength random torques, and its momentum is only allowed to change by gravity.
// entry 02
Fixed
It got paid to crouch.
This one looked like success. After a long stretch of runs ending in a heap, the creature started surviving the full 30-second episode — 99% of the time, against 101 steps for random torque babble and 610 for a CEM-optimised constant controller. Great. Except it wasn't standing. It was crouching, about 300 units tall where standing is 414.
The culprit was a safety net. A blow-up guard clamped total reward to 10 per step, so a diverging simulation couldn't flood training with enormous numbers. Sensible. But the height term was scaled by 20, so every single living step hit the cap. Measured afterwards: 9.99 reward per step, all run long. Standing tall earned exactly as much as crouching. Height had become mathematically invisible.
So the policy optimised the one thing it could still see — not falling over — and did that superbly. It wasn't being lazy. It was being correct, about a reward that was wrong.
~300 UNITS TALL · STANDING IS 414 · PAID THE SAME EITHER WAY
After the fix, under normal gravity, runs hold about 365 — right on the 370 target I'd set. Which is what made me raise the target. Which is how we got entry 01.
// entry 03
Fixed
The constant with an expiry date.
The contact solver used to have a relaxation-iteration count. I tuned it carefully, landed on 8, and moved on. The very next day, the order of the physics step changed. Nobody re-checked the 8.
So it kept doing its job under assumptions that no longer existed: the creature sank 7.5 cm into the ground at rest, and the contact forces flickered. When I finally swept it again, the right answer had become zero. At 0, resting penetration is 0.54 cm — which is just the 0.5 cm contact slop doing what it's supposed to — and contact-force flicker dropped by 96%.
And a setting whose best value is zero is a setting you don't need. It's gone from the solver now.
// entry 04
Fixed
The NaN that got away.
Fall detection compared the creature's state against a threshold. NaN compares false against everything. So a creature whose simulation had completely diverged sailed straight past the check, was declared perfectly fine, and training carried on learning from it.
Nothing crashed. Nothing warned. It just quietly learned from nonsense.
// entry 05
Fixed
Centimetres.
Unreal thinks in centimetres. Every physics paper I was working from thinks in metres. Guess how that goes.
(For the record: a length off by 100 is bad. Rotational inertia goes as length squared, so that one's off by ten thousand — and it still produces motion that looks like something.)
// entry 06
Fixed
Ghosts from the last episode.
Two smaller ones to round it out. State surviving a reset, so an episode started out carrying something left over from the one before it. And joint conventions that were wrong in a way that still looked plausible — which is the most dangerous kind of wrong, because nothing about it looks broken.
// entry 07
Fixed
It stood up. It would not stop shaking.
This is the good news entry, and it still took a fix. The creature finally got to full height — past 414, the whole thing, on the ground, on purpose. And it stood there vibrating like a washing machine on a bad floor.
Nothing was wrong, exactly. The policy outputs 68 muscle activations every control tick, and I was feeding those numbers straight into the solver as-is. A muscle could go from slack to full clench and back inside two ticks, because nothing in the system said it couldn't. Real muscle can't do that. Mine had no idea it wasn't allowed to.
Worse, that's a perfectly good way to balance. Buzzing at 60 Hz is a stable strategy: over-correct, over-correct back, never actually fall. The reward was paying for height and for staying upright, and shaking delivered both. So it shook.
BEFORE / AFTER · SAME HEIGHT · ONE OF THEM IS WATCHABLE
The fix came from both ends at once, which is why it worked. On the physics side, muscle activation is now smoothed on its way into the solver instead of using the raw policy output — a muscle ramps toward what it's told rather than teleporting there. On the reward side, rapid changes in muscle intensity now cost something, so the policy is charged for the thing it was previously getting for free.
Either one alone half-works. Smoothing without the penalty just means the policy shakes harder to push the same jitter through the filter. The penalty without smoothing leaves the physics able to accept impossible inputs in the first place. Together, it stands still.
// entry 08
Fixed
It could not see how high it was.
Observations get clamped on their way to the policy — standard practice, and for good reason: one channel spiking to 400 while everything else sits near 1 makes for miserable training. Mine clamped every channel to a magnitude of 2.0, which I set early, when the creature's whole world was a crumpled heap near the floor.
Then it learned to stand, and the number stopped moving. Height error, torso velocity — anything with real magnitude behind it pinned itself at 2.0 and stayed there. Across the entire standing range, from half-crouched to fully upright, the policy read the same constant. It was not being given bad information about its height. It was being given no information about its height, in a shape that looks exactly like information.
Nothing errors when this happens. There is no NaN, no warning, no diverging loss. The tensor is the right size, every value is finite and in range, and training runs happily for hours producing a policy that is guessing. It took noticing that the creature behaved identically at two visibly different heights — which is not a physics symptom, it's a perception symptom — to look at the clamp at all.
Raising the clamp to 10.0 fixed the saturation. The more useful half of the fix was adding an explicit airborne block — ground clearance, ballistic time to impact, and a plain airborne flag — so that "how far off the ground am I, and how long until that changes" is something the policy is told directly instead of something it has to reconstruct from a channel that was already flattened.
// what all of this changed
I stopped nudging parameters.
What actually changed, after all of the above, was how I debug. I used to nudge parameters until it looked better. Now I write down what I think is wrong, work out what test would distinguish it from the alternatives, run that, and only then touch the code.
The flying bug is the cleanest example of it. "It looks like a jump" was a perfectly reasonable reading of the video. "Airborne acceleration must equal gravity" is not a reading. It's a test, and it doesn't care what the video looks like.
Slower per attempt. Enormously faster overall. And a lot of those distinguishing tests stick around afterwards, which is how the project ended up with a test suite that checks physics invariants instead of my eyesight.
// found another one?
Spotted a bug in this page?
Tell me. I clearly collect them.
Back to the story ↺