Teaching the app to hold still
Mitamaton's home screen is a board of cards: what's next, the calendar, the inbox, whatever you keep pinned. You arrange it once and it should stay arranged. For a stretch of the beta build, it didn't. Every time the app launched, the cards would paint in one arrangement and then slide into another. Half a second of motion, every single open.
Small thing, you might think. It was the most corrosive bug we've had. An assistant's whole pitch is that it holds your day steady, and here it was fidgeting with its own furniture every morning.
Finding it took four separate fixes, because it was four separate bugs wearing one costume. The server was quietly duplicating one kind of card each day. The launch path animated changes it should have applied silently. A custom card was being deleted and recreated with a new identity on every start, so the layout engine treated it as a stranger and reflowed the room around it. And the animation system, it turned out, was allowed to override the very instruction that said "apply this without animating."
The interesting part is what we did after. The human test that caught the bug was simple: delete a card, restart the app, watch closely. So we taught the machines to run exactly that test. Every night, a harness installs the app fresh, launches it twice, records the second launch on video, and analyzes the pixels frame by frame. If the board moves after first paint, the run fails. It learned to tell real motion from content fading in, because a card filling with data is fine and a card sliding across the screen is not.
Since then the test has caught two more would-be regressions before any person saw them. The app holds still now, and it has a camera pointed at it to make sure it stays that way.