The New Britannia Herald

In this episode

• Two hours of locked logins and a rebuilt release
• The spawn queue that ended for the whole session
• A stronghold fight that switched itself off
• Lot deeds upgrade in place, house untouched
• Player vendors sell a stack a few at a time

Transcript

Good evening to thee, citizens of New Britannia. I am Lord British, come with word of how thy makers spent their Sunday. The shard shut its doors upon thee for two hours, and I shall tell thee plainly why. The stronghold east of the Obsidian Trial held a battle that politely declined to begin. Thy deed may now be bettered without tearing thy house up by its foundations. And thy vendor will at last sell a great heap of potions five at a time. The locked doors first, since I am assured that tale is the most instructive of the lot.

So. Yesterday afternoon, for about two hours, the test shard looked at every player trying to log in and said no. The launcher had nothing to patch, and the server would not accept the game people already had on their machines. And all through it, the health check we built to shout at us when exactly this happens was sitting there quite happily reporting that everything was fine. It signs in every few minutes to prove the door works. It was signing in without telling the server which build it was, and the server will accept that from anybody. It was checking the wrong door.

The day had started badly and then got creative. The script that uploads a new client borrowed one function from a shared helper file, and the act of borrowing it reset two of the script's own settings back to that helper's defaults. One of those settings was where the upload goes. So the upload lost its address, concluded that the destination was the folder it was already standing in, and spent fifty-seven minutes copying the build into itself. It got all the way through a completed server swap before anybody noticed. The same mistake was sitting in the promotion script too, where a rollback would have moved the tester shard instead of the one you asked for.

The fix for that is dull and the guards around it are not. There are four new gates now. One of them reads the structure of our own scripts and fails any helper that is able to reach back and overwrite the settings of whatever called it. One hunts for characters you cannot see in committed scripts. Two days earlier, a single stray carriage return had split a comment in half and turned the back of it into live code. And the script that runs the whole release was carrying two null bytes, which meant version control had been treating it as a binary file. No change to it had ever appeared in a review. Not once. It had been invisible this entire time.

Then the afternoon, and the real reason the doors locked. One build number had been handed out twice, to two different builds, so the server was asking for a client that had never finished publishing. Underneath that, we had four different counters. The release job kept its own count, the Windows client job kept a different one, the Mac client job kept a third, and the game itself carried a fourth number entirely. Reading a broken release felt less like debugging and more like carbon dating. There is one number now, reserved from a counter that only goes up, claimed before any build starts, and burned when a release fails rather than handed back to the next one.

The shape of a release changed too. All three platform clients go up and get checked, then the server moves, then all three channels flip together. A failure anywhere before that last step puts the server back where it was, so players stay on the build they were already running. A watchdog now compares, every two minutes, the build each shard will accept against the build its channels actually serve. Those two disagreeing is the thing that locks everyone out, and until yesterday nothing in the world was looking at it. The nightly automatic release is paused until all of that has proved itself. And there is a ledger now, one row for each distinct way a release has failed since July. Thirty-eight of them. A commit that adds a new guard without adding its row gets refused.

Now here is the one I like most, because the symptom sounds impossible. Partway through an evening, towns would stop having people in them. Not empty when you arrived. The residents and creatures a scene starts with turned up exactly as they should. What went missing was everything after that. Nothing respawned. Nobody wandered into view. Creature waves never came. And once it happened to you it lasted until you restarted the game, and it followed you into every scene you travelled to for the rest of the night.

The client runs one queue that feeds every single spawn in the game, and it is started once when the game boots. One error inside that queue ended it for the session. The thing causing the error was small and ordinary: a spawner removed from the world while it still had a creature waiting in the queue. That is all. One creature's paperwork went missing and the entire hiring department went home for the day. The queue drops that entry now, survives any error at all, and reports how long each spawn took, which is how we found three smaller leaks of exactly the same shape sitting right next to it.

While we were in there, we fixed the other half. Spawns were going out at one every third of a second no matter how many were waiting, and Etceter asks for eighty of its eighty-nine residents and creatures the instant you arrive. That works out to roughly twenty-six seconds of a busy town being assembled around a player who is already standing in it. You could walk into the square and watch the stagehands finish the set. When the queue is deep it goes in short bursts now, with a hard ceiling on how long a burst may take, so it can never cost you a frame. And the Equipping Avatars pause is properly dead this time. Our first attempt on Friday left ten to thirty-eight seconds of it, because the cache that classifies items was re-reading each classification's whole item list once per item while it was building itself. It gets built once now, while the game starts up, all six hundred and twenty-seven classifications in about seven hundredths of a second.

Which brings me to the stronghold, and to my favourite thing anyone found all day. We raised that place east of the Obsidian Trial in a single day last week, and I was very proud of us, and then somebody did a second review of it. The scene had been cloned from a forest gate, and it was still carrying that gate's scenario manager. So walking into the stronghold picked a random town scenario, the client resolved the whole thing to scene number zero, and the result was that no Cabalist spawned, no power meter appeared, and no kill would have counted. We built a boss fight and shipped it with the boss holding his diary open at the wrong week.

That manager is gone, along with the two thousand one hundred and fourteen inactive objects it was switching on and off. The three exit volumes had been filed away inside one of those scenario sets, which is a wonderful place to keep your exits if you never want anybody leaving, and they are back at the top of the scene. The ground itself was stored in a form that had sixteen bytes of line endings stripped out of it, so anybody pulling a fresh copy of the project got broken terrain. And the sixteen throne seats had been placed on a flat plane across ground that runs from twenty-three metres to thirty. Six of those thrones were underground. The deepest was very nearly four metres down. Eight Cabalists and their liches, and a third of the court holding session from inside the hill.

The review kept going and the fixes kept coming. The Light of the Eight, which is a twenty-four hour celebration, was being written into the database with a three hour expiry, because the number was clamped a second time on the way in. A Cabalist holding a besieged town could heal most or all of his health bar back in the middle of a fight when players left the town, because the machine that took over running him rebuilt him from his base statistics for a moment. You walk away for a breather, so does he. That one is fixed on every town siege, not only in the stronghold.

And Vitium himself got retuned before a single player has laid a hand on him, which is the nicest kind of balance pass there is. He pays a hundred and twenty thousand experience. An ancient dragon pays a hundred thousand, so that tells you where he sits. He has a proper wall against stuns, so one player cannot simply hold him still for the whole fight. The liches sitting in the empty thrones fight at a fifth of full siege power, so a town cycle you won handily does not then hand you the easiest final battle of the lot. His loot lost four rows that had been copied straight from another Cabalist. His achievement pays a trophy like every other boss. And the fight explains itself now. A line on arrival tells you he cannot be hurt while his Cabal stands, and another tells you when the last of them falls.

Two things happened for house owners yesterday, and the first is one people have wanted for years. Lot deeds upgrade in place now. You walk to the lot sign, you pick a better deed, and that is the whole procedure. Your house stays. Your decorations stay exactly where you put them. The roles you have handed out to friends stay. The old way meant unclaiming the lot, which meant packing your house into storage, praying nobody grabbed the ground while you stood there holding it, and then rebuilding the entire thing by hand. The days of tax you had already paid move across to your account credit instead of evaporating.

The second thing is that the upgrade ladder refuses to walk backwards now. The price table allowed a handful of swaps that cost you more money and gave you less lot, and one of those would have started charging rent on a tax free lot, dated from the day you originally claimed it. The next tax sweep reads that as months of arrears. So you could have paid good money to downgrade your own home and then received a bill for back taxes stretching to the day you moved in. We have removed that opportunity.

Player vendors learned to sell a stack a few at a time. Tick sell individually when you list, set a price for one, and shoppers can buy any number out of it. If you have ever tried to sell a thousand potions in batches of five, you know the old arithmetic. Two hundred separate listings, which is every slot a vendor has, for one item. It takes one listing now, the commission is charged once for the whole stack and shared across the sales, and you get a letter for each sale telling you how many went.

A code review the same afternoon caught something sharper inside it. The commission on a partial sale was depending on the order buyers happened to turn up in. A hundred and ten potions bought one at a time out of a stack of a thousand cost the seller a hundred and ten gold in fees. The same hundred and ten bought in one go cost thirteen. It was a bulk discount running in reverse. The fee depends only on how many have sold now, so taking the rest of your stack home costs you the same either way. And if the remainder cannot go back on the shelf after a sale, it goes to your bank rather than anywhere less pleasant.

Three things in combat had been telling players untruths, and I want to name them because two of them were mine to notice and I did not. Standing in shallow water refused every spell that was not Water magic. That came in when the Water bonus learned to recognise ankle deep water, and ever since, every river shoreline in the world has been a dead zone for seven schools of magic. Wading refuses nothing now. Only actual swimming still stops you casting. Shock Weapon, Ignite Weapon and Chill Weapon were all firing their on hit effects as though the caster had never trained them, so a wand was landing a few points of damage instead of the tooltip range, and stunning on about five percent of hits where the tooltip promised thirteen. And Immolation's flames were stopping after about ten and a half minutes while the buff itself ran on, which on a strong fire caster meant most of an hour of being on fire with nothing to show for it.

Pets had a good day. If you stood above your pet, on a staircase, a balcony, an upper floor, a dungeon ledge or a steep hillside, the pet that had walked back and settled at your feet a floor below still counted as being away from you. So it stopped attacking, got yanked to you, settled, and got yanked again about every twenty seconds. It measures height on its own terms now, and a pet within about six metres of your elevation settles down and fights. In a related bit of geometry, packed player towns had been losing every pathing obstacle across a patch of ground whenever two decorations overlapped in a way the map cutter could not handle. Duanor and Jade Valley hit it on every load, which is why creatures there had developed such casual opinions about walls.

Three small mercies for your interface. Hiding it with F9 used to lock you out of the options menu, so the only route to Exit to Menu was turning the whole interface back on first. Escape opens options now, and closing it again leaves your interface hidden. The siege power meter had been following the compass downward for players who drag their compass low, straight off the bottom of the screen, and it moves above the compass now when there is no room below. And the catchphrase bar stops at the edge of the window while you drag it, so it can no longer be flung out into the void and stranded somewhere you cannot reach. Whispering a player who is away gets you an answer too, and slash afk takes an away message of up to a hundred and twenty characters.

Lightning round. Forty-eight tracker items closed yesterday, the bulk of them tester verification tasks going green, which is the sound of a backlog draining. On top of that, a content sweep of the kind that never makes a headline and improves everybody's evening. A display wand that would only agree to hang on a wall if you approached it from outside the wall. A hand lamp that painted black patches across the ground while you harvested. Grass poking up through the top of placed stone platforms. A pagoda roof that accepted a chandelier only within a patch about two metres across. A guard tower deck that believed it was a tabletop. A telescope showing you bits of its own body in the magnified view. And crystal shards on a defensive spell that had never once drawn on a real client, because the art was imported in a form the shipped game throws away.

Also a crafting trainer at Solace Bridge Outskirts who had been speaking in a woman's voice out of a man's body, harvest nodes paying producer experience twice over, a login that would sit on Connecting forever if your password was longer than the encrypted envelope could carry, and a launcher that remembers which language it was showing you, so a fresh install stops starting the game in whatever language your operating system prefers. Testers can also sign in to the tracker and the tester hub with the same name and password they play the game with, which takes one small daily annoyance away from the people doing us the biggest favour.

One last item, because the reason it was hard is interesting. A Titan of Truth is spelled Boreas, and six spoken lines in Aerie, Midras and Valhold still called him Boreus. We tried to correct that once before and had to pull it, because a translated line is stored under a fingerprint of the English words. Change the English and all eight other languages fall back to English for that line. This time the German, Spanish, French, Italian, Japanese, Portuguese, Russian and Chinese records moved across with the correction. And typing the old misspelling still finds the answer, because somebody out there has it written down.

That is Sunday. I am looking forward to the first party that walks into that stronghold and meets a Vitium who is actually there, with a meter that actually fills, and to hearing how badly he treats them. My thanks to everyone who sat at a login screen yesterday afternoon getting nowhere. That was us, and it is guarded now.

And as for those six thrones we found nearly four metres below the ground, I can only say that we dug into the problem, and the solution was groundbreaking. Good night, and safe travels through New Britannia.