That gentle hissing noise, like an engine losing compression? That was the sound of GitHub following Cloudflare in having a truly sh*tty Tuesday.
“We are currently investigating failures on all Git operations, including both SSH and HTTP” the Microsoft-owned shop admitted at 21:11 UTC.
(It shipped a fix within an hour of GitHub outage reports.)
Is it DNS? Is it? We await details and the inevitable apologies for a buggy configuration change that caused a cascade of breaking things, like some inverse Rube Goldberg machine, designed to give developers a break.
Or…
GitHub outage
Whilst we’re here, casually stealing some GitHub outage SEO traffic for shits and giggles, and using it to upsell some more serious stuff, um, this? Let’s review some other GitHub outages (see what we did there?)
Well, there’s been *a few*: 12 so far this month, 22 in October, 16 in September (many fairly short-lived and insignificant, to be fair, but…)
Their causes crudely fall into two camps: Code deployments breaking things. Inadequate/overloaded infrastructure leading to things breaking.
Incidents on October 23 and November 11 of this year were both caused, for example, by a “resource exhaustion on our backend ssh servers.”
That’s according to a quick perusal of GitHub’s status history, which shows that other recent GitHub outage causes include: “a partial deployment of a feature flag to a global rate limiter [that] triggered behavior that unintentionally rate limited all requests, resulting in 100% of them returning 403 errors”. (Oof.)
Or: “Copilot disabling routing to an upstream provider that was experiencing issues and reallocating capacity to other providers” causing network congestion. How about that old favourite, a “faulty configuration change that disrupted [everything]”; or “an exhaustion of resources in the network relay infrastructure”.
Have a read of the GitHub status history page and pick your poison.
Sometimes it’s not DNS. Sometimes it’s all the CapEx going on GPUs.
Views on GitHub’s stability? Feel free to opine.
The link has been copied!
