GitHub has admitted that a new and unreleased feature it had tried to hide behind a "feature flag" (an internal ON/OFF switch) resulted in buggy code leaking into live systems – breaking code in thousands of customer repositories. 

Its own alert systems didn’t spot the howler. The Microsoft-owned company, which has also recently suffered a string of severe outages, admitted in a post-mortem late April 23 that it was alerted by affected customers.

COO Kyle Daigle said it was “an edge case” not covered in tests.

The bug affected customers when two specific GitHub features were used together: merge queues and the squash merge method – which takes all the many, sometimes tiny commits from a feature branch and condenses them into one single, clean commit before adding it to the main branch. 

During the incident, when the merge queue tried to handle a "group" (multiple pull requests, or PRs, lined up to merge), it used an incorrect reference point to start from, resulting in "three-way merge" errors where the second PR in the queue accidentally "undid" the work of the first PR.

The incident effectively saw customers’ recent code changes start disappearing. Teams working on 2,092 projects had to manually go back, figure out what code was deleted, and "force-push" or re-merge their work to fix the history after the GitHub incident, which lasted 3.5 hours. 

The bug, admitted GitHub, “produced incorrect merge commits.”

“In affected cases, changes from previously merged PRs [pull requests] and prior commits were inadvertently reverted by subsequent merges.”

Posting on social media, GitHub’s Chief Operating Officer Kyle Daigle said that “2,804 pull requests out of over 4M merged on April 23 (roughly 0.07%) were affected. We fixed the issue, we've contacted every impacted customer, and we're expanding our automated test coverage…”

The regression was introduced by a new code path that adjusted merge base computation for merge queue ref updates. This code path was intended to be gated behind a feature flag for an unreleased feature, but the gating was incomplete. – GitHub, April 23. 

As the company put it: “we became aware…following an increase in customer support inquiries. Because the issue affected merge commit correctness rather than availability, it was not detected by existing automated monitoring and was identified through customer reports.”

GitHub says it sent “targeted remediation instructions” to those affected, with step-by-step recovery guidance. It has promised to expand test coverage for “merge correctness validation” including “regression checks that validate resulting Git contents across supported configurations, so issues affecting merge correctness are caught before reaching production.”

Criticised sharply on social media, GitHub’s COO said: “We are really focusing in on meeting the incredible scale and rewriting underlying systems to both make them more reliable and meet the new demand.”

He added: “We have more work to do tho.”

Earlier this month, Jakob Oleksy, GitHub's SVP of software engineering said the Microsoft-owned company is making “substantial, long-term investments in how GitHub is built and operated to improve resilience.” 

He added in a blog on the move: “Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements.”

GitHub has repeatedly emphasised the pressure its systems are under.

In a separate blog shared on April 20, days before the incident, VP of Product Joe Binder said: “We’ve seen usage intensify for all users as they realize the value of agents and subagents in tackling complex coding problems. These long-running, parallelized workflows can yield great value, but they have also challenged our infrastructure and pricing structure.”

GitHub has paused new Pro subscriptions and is introducing Copilot limits on existing accounts, as the platform faces a compute capacity crunch.

These are, of course, unrelated to reliability issues for customers related to how GitHub handles quality assurance and poorly tested/gated features. 

Join peers following The Stack on LinkedIn

The link has been copied!