GitHub has published the technical details behind an eight-hour service outage that disrupted its platform on August 17, 2025. This incident matters because it exposed how infrastructure scaling policies and client-side retry logic can combine to create a cascading failure. Developers relying on the platform for daily workflows faced significant downtime during this window.
Root cause analysis reveals infrastructure scaling and client-side retry logic issues
The disruption occurred primarily in the Central US region before traffic shifted to Northern Virginia. GitHub identified that the core issue stemmed from load balancers becoming saturated during the peak of the event. This saturation prevented the platform from processing requests efficiently for users in the affected region.
During the outage, error rates reached 20% for web and API requests. Raw content downloads experienced even higher failure rates at 50%. The root cause traced back to Istio sidecar pods failing to scale because the autoscaling policy monitored host service limits instead of sidecar limits.
A secondary issue emerged when traffic redirected to Northern Virginia triggered a latent bug in the VS Code extension. This bug caused Copilot authentication traffic to spike from 7,000 to 100,000 requests per second. GitHub resolved the immediate crisis by reducing gateway retries and blocking specific token requests with HTTP 403 errors.
The company plans to update its autoscaling policies and audit Istio capacity to prevent recurrence. GitHub also intends to fix the VS Code retry bug and improve load-balancer monitoring. These changes aim to stabilize the platform against similar cascading failures in the future.
Discussion
0 comments
Log in to join the thread with a thoughtful take, question, or correction.