The New Britannia Herald

In this episode

• A one day move onto a new engine version
• Fishing you can now lose
• Builds soak on QA before Chaos sees them
• Blurry textures on shared memory Macs
• The window that forgot its own size

Transcript

Good evening to thee, citizens of New Britannia. Lord British before thee, with an accounting of what thy makers were about upon Thursday. The very engine beneath this world was lifted onto a newer foundation in the space of one day, in defiance of a careful document advising against it. The fish have been granted the right to escape thee, and are making free use of it. Those of you upon the Apple machines shall find the world grown sharper. And thy game window shall at last remember where in the world thou left it. We begin, though, with a pet, a tower of stone blocks, and a lesson in gravity.

Picture it. You have built yourself a tower out of stone cube decorations, because of course you have, and you are standing on top of it admiring your work. Your pet is down on the ground where pets live. The game looks at the distance between you, decides the pet has wandered too far, and helpfully puts the pet where you are. Which is the air. The pet falls. Twenty seconds later the game has exactly the same thought, and does exactly the same thing. Your summoned friend spends the afternoon being lifted and dropped like a bad idea.

We had already fixed half of this. There are two separate pieces of code that decide a pet has strayed, and the earlier pass caught one of them. Yesterday the other one went down. Now, when the game cannot find anywhere sane around you to put your pet, it either sends the pet back to where it came from or leaves it standing on the ground exactly where it is, patiently, until you climb down off your monument and rejoin the walkable world.

Now. Fishing. A few weeks ago we shipped a reel meter, a proper little tug of war between you and the fish, and the evening it reached the test shard the testers played it and came back unanimous. It could not be lost at all. And when we sat down and simulated the numbers we had shipped against the real fight model, the testers were being polite. At every skill level, at every difficulty, for any plausible human reaction time, better than ninety nine percent of fights were won.

The reason is embarrassingly simple. At a Fishing skill of a hundred and thirty, the target zone you are supposed to be holding the bar inside covered fifty nine percent of the tension track. More than half. You could hold the bar roughly where you thought the fish might be and the game would quietly count it. That zone is now about a third of the size it was. At the same skill it covers a fifth of the track. The fish is livelier to match, and most of that went in at the easy end where the report was aimed: nearly twice the speed, changing direction every one and a bit seconds instead of every two and a half, jumping further, dashing more often.

The part I like is that the whole fight budget had to move with the zone. Make the target smaller without touching anything else and the line always runs out before the fish tires, and then every hard fish is unwinnable at every skill, which is a worse game than the one we had. So the fish now tires after eight and a half to eleven seconds of good tracking instead of nine to twenty, the line survives twenty four seconds outside the zone instead of fifteen, and holding the bar on the fish gives you two percent of your line back every second. It is a tug of war in both directions now. Bites come one and seven tenths as often as a passive cast instead of one and a half, which is what pays for the fights you are going to lose. Common fish still come in almost every time. The losses live on the rare and the trophy end, which is where losses belong.

And the bite itself reads properly at last. The bobber used to reach its dip depth in a single frame, one instantaneous teleport downward, which looked almost exactly like the idle twitching it does between bites. It now gets pulled three quarters of a metre under over about a quarter of a second, more than twice as deep as before, and worries up and down while it is held there. Those numbers were not eyeballed. They were swept through the real fight mathematics in a headless harness, reading back win rates for a sharp player, an average player and a sloppy one. Our thanks to Vesper, who reported both the meter and the invisible bite, and who also asked for the tuning knobs, so forty six of the forty seven fishing values are now on the in game tuning console for testers.

Now the large one, and it starts with a document. At two minutes past six in the morning somebody finished a survey of the engine patch releases we had skipped since July, and found three defects we had been working around by hand that were already fixed upstream: an internal assertion that can fill an editor log with six figures of complaints over a long session, a crash on entering play mode, and a crash in text rendering on fallback fonts. Good news. The same write up is scrupulous about what a patch would not fix, so that nobody goes in expecting miracles.

Eighteen minutes later, a second document weighing the newer major version instead, and recommending against it. The reasoning was sound. A major bump rewrites serialized asset formats much more broadly than a patch does, the project wide reserialization that implies is still blocked behind our version control migration, and that version stops getting patches the moment the next one lands, so we would pay the whole cost twice inside a quarter.

Forty five minutes after that, we were installing it on the build machines. That was my call. I read the document, I understood the document, and I overruled the document, and the rest of the day was the team dealing with the consequences of my enthusiasm. To their credit nobody has said a word about it.

Here is the consequence I did not see coming, and it is the most interesting thing that happened all day. The new engine changed the format of the model importer, which means it reimports every model in the project and rewrites that model's settings file on the way through. Nearly sixteen thousand files marked as changed. On every machine, at once, independently. Two different developers produced the same wave within hours of each other, and it looked like a sixteen thousand file merge was coming, which is the sort of thing that eats a week and your good humour along with it.

It was nothing of the kind. Somebody sat down and compared one machine's reimport against the version the other machine had already committed, file by file, all fifteen thousand nine hundred and eighty of them, and zero differed. The rewrite is deterministic. Which means it lands once and gets thrown away everywhere else, and nobody merges anything. They then went further and worked out what the diff actually contained, because trusting a sixteen thousand file change on faith is how you lose a project. Four things. A format stamp. Two new fields written at their defaults. A single letter the new serializer appends to one packed field. And long values that the editor no longer wraps onto a second line, which accounted for every last bit of the apparent bone and rotation churn. Identical values, different line breaks. No number was re rounded and no asset reference was dropped. The merge tool that is configured for those files hangs on this project without printing anything, so merging sixteen thousand conflicting files was the one route guaranteed never to finish. We now have that written down for next time.

The move itself was not free, mind you. The new engine promotes an old numeric identifier conversion from a warning to a hard error, in a few third party plugins we carry. The tempting fix is one line that tells the compiler to be quiet about it. Somebody checked what that conversion actually does when it runs, and it throws outright. So the quiet version would have compiled beautifully and then crashed the moment a player touched it. Those conversions were removed instead. And then the nightly build went red, because one machine was still launching the old editor, which had been deleted that morning, so it failed instantly and took the Mac and Windows legs down with it.

The cause was a check that lied. Asking our build system for the controller's own configuration returns an error rather than a document, and an empty answer reads exactly like nothing configured here. So the one machine that actually runs the Linux builds was skipped, while the machines that do not run them were carefully updated. All five now read the new editor, and the runbook now says an engine bump is a five machine job.

Now something that changes how a fix reaches you. Until yesterday, the nightly build went straight to Chaos, the shard our testers play on. A change committed in the morning was in front of real people that evening. That is a wonderful thing for pace and a slightly cruel thing for testers, because every rough edge you ship lands on the volunteers helping you. From now on the nightly goes to an internal QA shard first and soaks there, and Chaos moves only when a human signs the build off. The promotion reuses the exact build that soaked. Nothing gets rebuilt on the way.

Simple to say. The deploy machinery had one shard baked into its bones, and four separate places worked out which machine to talk to without ever being told which shard we meant. So a deploy aimed at QA would have sent the patch incoming warning to Chaos players about a patch that was not coming, run its post deploy smoke test against Chaos and reported a cheerful pass while QA went completely untested, and overwritten the build number stamped in the Chaos logs, which is the field we use to tell a real regression from somebody running a stale build. The shard argument is now required. You cannot leave it out and inherit the wrong answer.

Two quiet failures on the QA box were caught before they could bite. The cluster had never been told where its login mirror lives, so it reached for the other shard's internal address, a machine that does not exist over there. That fails in the worst possible way. All seven processes come up. Every registration check passes. The automated login probe passes. And then a real person tries to log in and cannot, and everything on the board is green. The start up now tests the mirror before it starts anything. Separately, the QA login door defaulted to open, on a box that holds a restore of real account data and sits behind public naming. QA now starts testers only.

Three more guards went in, and one of them is my favourite bug of the week. Retrying a failed promotion used to poison the rollback button. Flipping a channel to the build it was already on archived that build as though it had been replaced, so the one click rollback, which points at whatever was archived last, would then point at the build that is currently live. Press it in an emergency and it flips the current build to the current build, reports success, and changes nothing, while every player is still stuck at the login screen. A flip to the version already live now does nothing at all, and there is a test sitting on it. Alongside that, the patch notes would have started omitting whole builds, because notes are generated against the previous entry, and under the new model that previous entry is a QA build nobody could install, so a promotion covering four nights would have published the notes for one. A short patch note page looks exactly like a quiet week. And deploys now refuse to restart a shard onto a build whose database change nobody has run, which no deploy path in this project has ever done, and which was survivable only while the person who wrote the change was the person watching the deploy.

Something for those of you on Apple machines, and this one bothered me. Textures have been coming out soft on Macs, and we have been unable to see it, for a reason that is almost funny. A week of telemetry says a sixteen gigabyte Apple machine dropped texture detail in a hundred percent of one thousand four hundred and seventy two readings across ten different scenes, shedding up to six hundred and eighty seven megabytes of detail at a time. Every machine in the field with a separate graphics card shed nothing at all, across four thousand readings. Every developer machine that said looks fine to me had a separate graphics card in it.

The code that decides how much memory to spend on textures has two paths. If you have a dedicated graphics card, it asks the card. If your machine shares one pool of memory between the processor and the graphics chip, which every Apple machine does, it takes a share of system memory instead. That share was a sixteenth, and it was chosen back when the small end of the range was what we worried about. At sixteen gigabytes, a sixteenth comes to just over a thousand megabytes, and the game was asking for about seventeen hundred. When the budget will not cover the request, the streamer does the only thing it can, which is drop the resolution of everything, everywhere. That is what blurry means. That share is now an eighth above twelve gigabytes: two thousand and forty eight megabytes at sixteen, three thousand and seventy two at twenty four, unchanged at the small end and unchanged at the top. It costs about a gigabyte on a machine that was not roomy to begin with, and we took that trade over guaranteed blurring in every scene you visit.

And then the window. A Mac player reported that windowed mode did not come back at the size they left it, and underneath that one sentence were three separate faults, all in the single place we store window geometry. The first is the one I want to describe properly, because it is a rare and special kind of bug. We store three numbers: width, height and refresh rate. They were read all or nothing, so one missing value discarded the other two and handed back your desktop resolution instead. The refresh rate is the one that goes missing, because a windowed Mac surface can report no usable rate at all. And the reader then wrote its own fallback back over your saved setting. So the first bad read destroyed your window size permanently, and every launch afterwards opened at roughly desktop size, for ever, with no way back. A preference that deletes itself the moment you look at it.

Each of those three values now falls back on its own, a stored rate of zero is legal and simply means we do not know, and reading a preference never writes one. The second fault was that your size only reached disk on a clean quit, so a crash, or the launcher closing the game to apply a patch, threw it away. It is written when it changes now. And the third is that window position was never saved at all, so we have added it: polled while you are windowed, saved when you move the window, and restored at launch clamped to the desktop you have today, so a monitor that is no longer plugged in cannot leave your window somewhere you will never find it. Startup also logs three things now, the size that was on disk, the size we asked for, and the size the operating system actually gave us, so the next report of this kind can be read rather than guessed at.

Three things about mounts. The movement lean is switched on for the next test build, so mounts and riders tilt forward as you set off, rock back when you stop, and bank into turns, with the rider's upper body held closer to upright than the animal underneath. Testers get one fixed amount of tilt rather than a slider, because that value is compiled into the client and cannot be retuned from the server, and the testing task says so up front. It also says, up front, that only your own mount leans in this build. Everyone else rides level until the tilt is driven from movement data that already crosses the wire, and we would rather tell you that than have somebody spend an evening reporting it. On paper, two more mounts were costed. A rideable bear, which is mostly art and data because the animation system keys off a shared rider pose, the same trick that lets the rideable chicken sit on a stag's pose. And an open umbrella that carries you a foot off the ground, drifting as though it were towing you, which the coconut halves proved is possible, since they are already a mount with no model at all.

A handful of smaller irritations went down too. If a scene load bounced or recovered rather than finishing normally, the client was left running at the loading screen frame rate for up to ten seconds after the world was already in front of you, and that is fixed at the root. The refusal you get when you try to move your house onto a spot with a yard item under the footprint now tells you which item, by name, if there are three or fewer of them. Books stopped snapping sideways on the last frame of their opening animation, because the camera pivot was being re anchored to each new shape in turn and is now placed once and left alone. Book chapter headings and that big decorated first letter now show the proper manuscript red on all seven interface skins rather than only the default one. Login no longer waits on itself while fetching the options window. And over in Solace Bridge Outskirts, Corporal Heloise's heirloom task now completes when you hand an heirloom to Heloise, rather than when somebody sells one to Hartley across the swamp, which is a single character change and a long stretch of mild confusion.

The lightning round. More than fifty tracker issues went to closed, and the great bulk of those were tester verification tasks finally being swept up in one pass: sieges, decorations, camera behaviour, inventory windows, tooltips, and a butcher in Owl's Head whose sausages had stopped placing as decorations, which is the single most New Britannian sentence I have read this month.

The rest was housekeeping, and there was a great deal of it. Eight dead vendor greetings were removed, each one a second copy of a line attached to nothing, and every deletion was checked first to make sure a working greeting remained, because removing the wrong copy would leave you standing in front of a shopkeeper who will not open. A translation check that had been red on every commit was audited line by line, all sixty offenders proved to be developer tools and dead commented out code, and eleven of them were then removed outright. The design plans got a proper front page and one shared vocabulary for status, instead of seven different phrases that all meant we have not started. Eighteen skills window tests came back from the dead. And we settled an old question: the dungeon set models really do import without any level of detail information, on both platforms, so the fault is in the models themselves.

That is Thursday. A long one, and an expensive one, and I am very glad it was not me doing most of it. I am looking forward to hearing what you make of the lean on a mount at speed, and rather more to the day somebody floats past Central Brittany under an umbrella. Go and lose a fish. It is good for you.

And I am told the new reel meter is a serious test of skill, which means that for the first time in this game's history, when you tell your friends about the one that got away, it will be true. I want you to know that took real effort. Reel effort. Good night.