On July 16 colossal errors in AWS customer bills started appearing around the world. Worryingly, internal alarms “detected cost anomalies but failed to halt the estimated bill generation process or alert our engineering teams.”
That’s according to the hyperscaler on Saturday (July 18), with AWS admitting that it was only catapulted into action a day after its internal alarms went off due to increasing numbers of “customer escalations.”
The issue had caused some companies to receive bills for over a quadrillion. Others had seen multi-trillion bills. These, of course, were obviously wrong. More troublingly for other companies (the impact was global) were the bills where errors resulted in sharp increases but still believable ones.
"Between July 16 at 7:38 PM PDT and July 18 at 6:00 AM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console, including the Cost and Usage Report. Customers may have received erroneous budget and cost anomaly detection alerts, and observed inflated estimated cost and usage data. On July 16 at 7:46 PM PDT, our alarms detected cost anomalies but failed to halt the estimated bill generation process or alert our engineering teams. We were alerted to this issue on July 17 at 12:19 AM PDT by customer escalations" - AWS Status page.
One erroneous AWS bill, shared on Reddit.
The cloud provider has also faced a chorus of criticism online for its “tone deaf” response to the issue on social media – where the official AWS account posted a joky response replete with a sweating emoji.
Typo alert: Some customers saw quadrillion-dollar AWS billing estimates today. Slight miscalculation on our end (very slight 😅).
We're fixing it now. No action needed on your end. Sorry for the confusion.
Real question: what will you do with those trillions instead?
Amazon said in a status update: “We identified the root cause on July 17 at 12:00 PM PDT as a configuration change in our bill computation system. This system relies on unit conversion data to calculate line item charges.
“The configuration change caused updates to the unit conversion data to fail, resulting in inflated line item costs, which propagated to the Billing and Cost Management console and triggered budget and cost anomaly alerts.”
The company added: “We have corrected our alarms to immediately halt processing and notify our engineering teams when anomalies occur. We apologize for the alarm this incident caused our customers and are conducting a thorough retrospective to prevent events like this from reoccurring, as well as improve our response when billing incidents occur.”
