The New Britannia Herald

In this episode

• The world-entry freeze is found and fixed
• Two arrow keys could take down the client
• Town sieges stop starting themselves
• Guild and town website built in one day
• Why the new client holds more memory

Transcript

Good evening to thee, citizens of New Britannia. It is Lord British, come with word of what thy makers were about upon Friday. The freeze that has been stealing thee out of the world is run to ground at last. Two arrow keys, pressed in the chat box, could put the whole client out. Thy guilds and thy towns have a home upon the web now, raised in a single day. And the sieges have been starting themselves while the switch upon the wall read off. Sit thee down and take thine ease. We begin in the arena, where a man was waiting to fight thee from inside a cupboard.

So. You walk into the Courage Club, you pay for your round, the gate comes down, and the announcer names your opponent. And then nothing happens. No opponent. You wait. You look around the sand. You check the doors. Your opponent did arrive, on time, exactly where the game put him, which was inside one of the sealed alcoves around the edge of the floor, behind a wall that does not lift for that round. He could not reach you, you could not reach him, and the round could never end. Ten spawn points were doing it, across four rounds, the named boss rounds included. They are all on the sand now.

Now the one I have been wanting to talk about for weeks. Some of you have been walking into the world, or into a town, and having the game simply stop. Not crash. Stop. The picture holds, and the client sits there until you kill it yourself. It has been happening since a build back in the summer, it did not happen to everybody, and for a long while we could not see it from the inside at all.

What broke it open was two hang dumps from the client watchdog, and a small piece of work that landed beside them. We can read one of those dumps end to end now without a debugger, because the engine ships the symbols we need, so the inside of a frozen client resolves into real names instead of a column of addresses. That is the tool. Here is what it showed.

The game's main loop was sitting inside the engine, with none of our own code anywhere on the stack, waiting for the previous picture to be handed over to the screen. That wait has no timeout on it. It is allowed to take as long as it likes. And on every affected machine, four of them now, across three different graphics driver versions, all of them carrying the same recent Windows update, the wait never came back.

So the client had not crashed. A crash, at least, has an opinion. This was our game asking Windows very politely whether it had finished showing the last frame, receiving no answer at all, and deciding that it would wait. Patient. Well mannered. Of absolutely no use to anybody. The fix is to stop using the newer way of handing a finished frame to the screen and to go back to the older one. The launcher passes that switch for every deployed build starting now, the next client build carries it as its default, and the boot log records which path a session ran on. If this ever comes back, we will know in a minute rather than in three weeks.

Chat next, which had a bad day and then a rather good one.

You press the up arrow in the chat box, you expect the last thing you said. What some of you got instead was the client leaving. The reason is embarrassing and completely ordinary. Two different pieces of code were both handling those keys. Ours, which swaps the line in your box for one out of your history, and the engine's own text editor, which was still holding the line that had just been taken out from under it. It went looking for text that was no longer there, and down went the client. Chat takes those keys for itself now and the engine never sees them.

While we were in there, we fixed the two things that had always been a little wrong about history anyway. Holding the key walks back through your lines instead of making you press it once per line. And the cursor lands at the end of what it recalled, so the next thing you type gets added to it rather than dropped in front of it. The same hazard sat in every other one-line box in the game, the rename fields and the search boxes among them, so up and down are ignored there now. The book editor and the other proper writing boxes keep their line navigation, as they should.

There was a second one from the same corner. With two chat windows open, pressing the slash key put the slash in the box, and then the very next letter you typed swallowed it. So you typed slash time, and the town heard you announce the word time. Anybody with a single chat window never saw it once. The cause had nothing to do with chat. Raising a window to the front rewrites the drawing order, which pulls its panel out and puts it back, and the text cursor gets dropped along the way. With two chat windows, that raise is unavoidable. Both places that hand focus back now put the cursor back with it. And at last, you can select chat text with the mouse. Drag to highlight, control C to copy, colour codes stripped out, double click for a word, triple click for the line.

Town sieges got a full review this week, and what it mostly turned up is that the system had not been listening to its own switches.

Start with the worst of them. Every time the coordinating server restarted, it started sieges, including on days when the settings page read off. The scheduler's first tick runs one second after it is built. The server's first read of the settings lands five seconds in. And in the gap between those two, the setting it is holding is the one it shipped with, which says sieges are on. Four sieges began one night at twenty past midnight. Two more the following night. Each of them was followed, about a minute later, by the scheduler noticing that sieges were switched off. It now reads the settings once before the scheduler exists, and the scheduler starts nothing at all until that read has landed.

Then the Cabalists. Since a recent build, killing one has freed the town and paid absolutely nobody. No experience, no defeat announcement, no credit, no name in the log. Seven clears out of seven, every one of them anonymous. The report saying the boss is at zero health was ending the siege a few milliseconds before the report saying who killed him arrived, and by then the town no longer had a boss, so the kill was refused as coming from a creature that did not exist. The ending waits ten seconds for that kill report now. In practice the kill report ends the siege itself, so the town lifts sooner and the person who did the work gets paid and named.

The Cabalist also kept going home. A creature works out where it belongs from the spot it was seated, and it does that again every time it is switched back on, which happens whenever a player comes into range. His seat is the town gate he marched in through. So every time you walked away and came back, you were quietly telling him that the front gate was home, and off he would go. He answers with the shrine now once he has reached it, and the fight re-tethers him every second. His body and his followers' bodies also stay where they fall when the siege lifts, which means he can finally be looted.

Here is my favourite. Siege creatures watched from a distance stood perfectly still with their arms straight out to the sides. They are the only creatures in the game that exist beyond the range where we switch a creature's animation and its mind off to save the work, and one that got switched off before its animation had ever run once froze in the pose a model rests in before anybody has told it to do anything. So the besieging army was out there on the approach to your town, motionless, arms wide, waiting for a hug. They keep animating and thinking from two and a half times further out now.

And one more, which I like because it explains something nobody had noticed was wrong. A Cabalist is meant to march to a shrine and defile it. We ran a census across the shard and found that not one siege it had ever started carried a shrine position, because the tool that records those predated the field that holds them. So in a player owned town, the dark lord of the siege was walking to the town crier. The census took twenty-nine seconds and found positions for six hundred and fifty-six devotionals standing on two hundred and fifteen lots. From the next siege on, he has somewhere proper to go.

Now the big one, and I still do not quite believe the shape of it. Yesterday morning, the guild and town website was a plan document with an owner's brief attached to it. By the end of the day it was a running site.

It began with a census, because you ought to know the size of a thing before you build for it. Eight hundred and ninety-six guilds, three hundred and forty-five of them with two or more human members, and a hundred and thirty-five of those active in the last six months. Four hundred and thirty-nine real player owned towns, a hundred and ninety-six with a governor or a steward active in that same window. And about one account in six came in through Steam and has no usable password, which settles an argument about sign-in before anybody has to have it.

Then the day happened. There is a machine, provisioned and locked down and backed up off site before a single line of anything a player would write existed. You sign in with your own game account, checked once by the game's own sign-in service and never stored on the site. Guild rosters show who is in the world right now, named by character. A leader can switch pages on for their guild, each page public or members only or officers only, with every save kept so it can be put back. Members can read their guild's chat history in a browser, refreshing every five seconds. There is a message board. All four hundred and thirty-nine towns have pages waiting for a governor to turn them on, with a town hall board of their own. Images upload to a separate host, re-encoded with their metadata stripped off. A calendar takes invitations and replies and shows every time in the reader's own timezone. One day.

A word about that message board. Anything posted publicly goes past the game's own chat moderation model before it appears, and I am pleased to report that in testing it refused a post offering gold for real money. So that works.

And then a review pass the same evening went back over the day's code and found twenty problems, which were fixed that night. Two of them mattered in the real world. Every reply on the message board had been failing outright. And any image larger than sixty-four kilobytes was refused before the upload code ever ran, which is to say, every image anybody would actually upload. That is what building a thing in a day buys you, and it is exactly why you go back over it before anyone arrives. Within a few hours of the town pages going up, a governor had already turned pages on for their town. Nobody asked them to. I like that enormously.

Last of the big ones, and this is a story about a player being right and wrong at the same time, which is more useful than either on its own.

We moved the client onto a new version of the engine. A player reported that the new one runs worse than the old one. So we checked that against what clients relay back to us. Same player, same graphics card, same scene, same quality setting, the new build against the three before it. Across ten matched comparisons, the median frame rate was eight percent higher on the new build. Processor and graphics frame times were both lower. Stalls of five seconds or more fell from about one and a half per thousand minutes of play to none at all. Three of the ten did get worse, one of them clearly, a player town on an older card that lost a tenth of its median and a third of its worst moments.

So by the frame counter, the new client is faster. And the player was still right, because there is one measurement where the new build is worse everywhere, on every machine we have seen, and that is memory. The new client is holding a gigabyte and a half to nearly two gigabytes more than the old one, already by the end of the first zone load.

Then we found why, and it is a lovely bit of detective work. The cleanup sweep that runs at every zone change has stopped returning anything. On the old engine, that sweep gave memory back on about one zone change in seven, ninety-five times out of six hundred and thirty-five. On the new one, it gave memory back zero times out of a hundred and fifteen. And it takes exactly as long to run as it always did. It goes away. It works hard. It comes back with nothing.

What that means for you is quite specific. A client that travels ratchets up roughly a gigabyte per zone and never comes down again. A client that stands in one place does not grow at all, and we have a session that sat on the same number for a hundred and ten minutes. On a machine with sixteen gigabytes, that is the difference between fitting and paging, and paging feels like stutter while the frame counter reads exactly the same as before. Which is how a player can be completely right about the feel while the numbers look fine.

Two suspects were cleared on the way. The new engine reserves four gigabytes of address space at start-up for its own bookkeeping, which looks alarming in a log and works out to about fourteen megabytes of real objects. And both builds ship identical game files, down to the file count and the size of every packed piece. So it is the engine itself, and the next client build carries a breakdown that reports how much memory each kind of asset is holding, so we can name the culprit instead of describing it.

Some smaller things now, each of which will improve somebody's evening.

Downloads. Game files come from a mirror with servers all over the world now instead of our one box, which should be a real difference for anybody outside North America, on patches, repairs and first installs alike. We checked that a small patch stays small through a mirror, and it does. A thirty-two megabyte file with about a hundred changed bytes in it moved eighteen thousand bytes. And because no day is perfect, we also put a request limit on the patch server, sized for what we imagined a patching client does, and then watched a real one, which asks for hundreds of things a second, because a build is around nine hundred files and each one needs asking for. Anyone patching in a forty-five minute window yesterday afternoon got an error and a stalled bar. The limit is gone.

Stealth. If you cast a hide and somebody walked into the scene afterwards, they saw you standing there in plain view, out of your combat stance, while you believed you were hidden, and it stayed that way until you cast it again. That is fixed for all four of the concealments. And the guards at the Clink will no longer stroll over to greet you while you are hiding. Six of them start a conversation on sight, and they were doing it to invisible people, out loud, in the chat log.

Your house and your pets. Pets used to climb on top of your chests and your crafting stations and just stand up there. Pets set to patrol would work their way down a shelving shoreline into the sea a step at a time, because no single step was ever a big enough drop to refuse. Rain has stopped falling through two more roofs. The windows on thirty-seven houses cast shadows again with the light coming through the panes between the bars. And cancelling a decoration move no longer drops you through the floor of the world.

Numbers that were simply wrong. Summoning, in the taming tree, had been converting your pooled experience at one eighth the rate of an ordinary skill, which came to twenty-seven casts per level at a hundred skill, against about one and a third for taming a creature in the same school. It runs at the normal rate now, about three. The potion for air mages was granting roughly a hundredth of what its siblings grant, reported by Wishdragon. Recipe names in the crafting book had gone flat grey for anybody with real skill, since grey meant a recipe you could not fail, and at a hundred skill that was eighty-four percent of all of them. Grey means outgrown now, reported by Gravidy. And the item comparison column will tell you what a shield actually does for you and name which hand each panel is describing, from three separate reports by Rinaldi.

The lightning round, and it opens with the most humbling item on the list. We audited a build log and found a shader that had shipped broken. Eight of the compiler's workers crashed on one particle shader, the editor restarted them and carried on, the half-built shader went into the packed game files anyway, and the build reported success. And the test we had written to catch exactly this had never run. Not once. The nightly picks its tests by a naming rule, and that file sits outside it, along with eighty-eight others, roughly a tenth of the suite. Twenty-one of those run every night now, and a build whose shader compiler dies fails on the spot.

After that, a sweep. Five town maps were measuring a metre differently across the page than down it, which is why your arrow drifted off the roads, and one of them had inherited another town's numbers wholesale and had never once been measured against its own picture. Fishing took a second difficulty pass, dialled in over two thousand simulated fights rather than by eye, and the line you spend is gone for the rest of the fight now. Two more control points escalate past the eighth wave forever. Books print on aged paper because a player asked for it. And the tester handbook is a public page at last, so you no longer need tester status to read how to get onto Chaos and file a report.

Eighty-eight tracker items went down yesterday, forty of them bugs and most of the rest verification tasks closing behind testers who confirmed the fixes. A good many were tooltips misbehaving in ways I would rather not describe at length. I will spare you.

That is Friday. I am looking forward to standing on a hill outside a besieged town and watching an army that moves before I get close to it, which is a low bar, and I intend to enjoy it thoroughly anyway.

One last thing, found while chasing a lighting bug. The water in this game has never drawn a reflection. Not once, not in years. The code that would render one was commented out so long ago that our own notes had invented a reason for it. We have corrected the notes. The water is unchanged. So if you have ever stood on a shore at sunset and felt that the sea was giving you nothing back, you were right, I am sorry, and I can only ask that you reflect on it, since the water clearly never will. Good night, and virtue go with thee.