It is Christmas, I'm home with my family.
Hence the textures wallapper and big floppy cushions.
And I know I've already put up a Christmas video, and in it,
I signed off until 2016, because surely there's not going to be any company
having a massive, avoidable, security problem over Christmas
that people are then going to pester me into, um, explaining...
[pause]
So when I was a kid at school, there was a thing called "chinny reckon".
Your school might have said something different,
like Jimmy Hill or itchy beard, but the idea was:
someone makes a dubious claim,
like their uncle working at NASA and
giving them a ride on the space shuttle,
and you go, yeah, sure. Chinny reckon.
Well, about the same time, back when the World Wide Web was a few servers running mostly
in academic labs,
they didn't need what are now called web caches:
a request comes in for a web page, the server, back then,
would look up the page on disk,
and load it, and then send that back.
But as the web expanded, demands on servers got a lot bigger --
not just in terms of how many individual requests for files they get,
but how much work they have to do for each of those requests.
Loading the home page of Steam involves more than fifty requests for files, images and
stylesheets,
and returns somewhere between 5 and 6 megabytes of data.
But there's a lot of work going on behind the scenes too:
their server has to check the database,
look up which games are currently being promoted out of the 10,000 that they have,
load all the pricing information,
work out which bits go in which part of the site, and if you're logged in,
it has to add a load of personalised recommendation stuff just for you.
That's all really database-intensive.
Takes a lot of processing power.
Which is where caching comes in.
Now, there are many ways that web servers do caching:
there are content delivery networks,
and even the user's own browser.
And in Steam's case, they use all of the possible techniques,
including the one that screwed up here,
which is the caching on their own web servers.
A huge number of requests to Steam's servers will be for pages that have already been generated
recently.
If they're getting ten thousand requests a second for their home page,
it'd be ridiculous for the server to go out to their database and
look up what's on offer ten thousand times a second,
just in case something's been changed.
So it looks it up once, and then it caches the result into its memory for...
well, I don't know how long, maybe a second, maybe a minute,
maybe half an hour in some cases.
Actually, a huge number of servers are doing that same thing:
there's no way that one web server could deal with a site as big as Steam,
so all those hundreds of servers check the central database every so often,
they keep their own version of each public page in their own memory,
sending that out to everyone who asks for it.
And when a request comes in, it's routed to any of those available servers.
That way, if there's a sudden spike in traffic,
Steam can just spin up some more servers,
add them to the pool that's answering questions, and it all keeps working.
And each server might have thirty different versions of that site cached: for different
languages,
for different regions, for different devices.
If that server has the right version in its memory, and it's not out of date,
there's no reason to bother the database for it --
they just send out the one that they already know about.
It means only a tiny fraction of that huge amount of traffic actually has to bother the
database:
the rest comes straight out of memory on a system that's built just for serving web pages,
which is orders of magnitude faster.
Which is great for anonymous passers-by,
who are just reading about a game or looking at the home page.
But what about the folks that are logged in?
They're expecting to see their name and avatar at the top,
to see the whole site personalised for them.
There's no point trying to sell someone a game that they already own.
Unless you already own it on CD from ten years ago and
you can't be bothered to go looking for it,
and it's about two bucks just to download it again, so...
Anyway. The traditional logic is that you don't cache pages for users who are logged
in.
Logged in users are a minority on most sites --
like on a news site, you have thousands,
maybe millions of people reading but only a few
who can actually log in and change what's on there.
So everyone who's logged out, the millions, they get the standard,
cached version; but if you're logged in as an editor,
then every time you request a page,
the server asks the database to check everything again,
just in case something's changed.
The cache doesn't touch anything that's personalised.
But: that isn't always a good strategy.
Steam might actually get more requests from logged in users than anonymous ones,
particularly when they've got a sale on and
people are regularly reloading the page to see if anything's changed, ready to buy.
In a situation like that, it might be a good idea to tell the web servers that, yes,
you should cache some pages, like the home page,
for logged in users, as long as you do two things:
First, you clear the cache if any data changes,
so no-one buys a game and then sees it still available for purchase.
And second, you never, ever, let anyone see someone else's cached page.
Well, someone in Steam, on Christmas Day,
made a quick change to the caching system,
probably to make the site run a bit faster.
And they accidentally told the cache servers to not care
about which user was logged in, or any of their settings.
So when, a moment later, a request came in from someone to see their own, personal account,
one of the caches looked up the account in the database,
then stored the resulting web page.
In whatever language it was in,
with whoever's private data was in there.
And when someone else sent a request for their account page a fraction of a second later:
that server sent the cached page,
which that first person's details, right back to them.
And each of those many, many web servers cached someone else's information --
whoever was unlucky enough to request their details from that server first,
at the exact wrong moment in time.
And after maybe a minute, maybe more,
the cache let that data expire,
and promptly stored the details of whoever the next unlucky person was,
sending that out to everyone else.
And at that point, a lot of people were checking their own account details,
wondering if they were about to have their credit card drained.
Now, if there's any consolation for Steam, it's that they're not alone:
they made the same error as the Norwegian government in 2012,
where for about fifteen minutes,
anyone who logged in to see their tax return saw someone else's personal details.
Actually, it wasn't just someone else's personal details,
it was the details of a man called Kenneth Belcovski from Oslo.
I believe they made a personal apology to Kenneth.
And the good news is, requests to update data --
so to change passwords, buy things,
and so on -- they never hit the cache.
Well, they shouldn't, not unless there's a much bigger screwup that Steam hasn't told
anyone about yet.
When you click "Buy", or "Change Password",
your browser sends something called a POST request that bypasses any caches --
so your login cookie gets checked on the way,
and it'll most likely just throw an error if it thinks you're trying to see someone
else's account.
Which means those people -- and I've seen a couple --
who were claiming that during the screwup someone bought a load of games on their account
and
immediatley want a refund? Well,
I'm not saying you're making it up,
but if this were a playground: chinny reckon.
Merry Christmas everyone.
One more thing I remembered while I was editing this:
if you know a YouTube video gamer, Twitch streamer,
who matches all these requirements:
tell them to get in touch with me,
there's a collaboration that I would love to do with them
at the start of next year.
[Translating this video? Add your name here if you'd like credit!]