Scaling the summit: Hyperscalers and the science of peak event planning

The digital world rarely sleeps, but it sprints around peak events. In the last quarter of each year, events like Black Friday, Amazon Prime Day and Cyber Monday are a high-stakes marathon for hyperscalers like AWS, Azure, Oracle, and Google Cloud. Orchestrating a flawless peak event performance is a tightrope walk in a hurricane. Success is met with silent nods, while failure can trigger a PR firestorm.

Peak events are seismic shifts in activity that expose hidden vulnerabilities and push systems to their limits. Even when a company designs a peak event entirely from the ground up, unforeseen issues can still arise, as demonstrated by the challenges surrounding the planned Netflix Mike Tyson vs. Jake Paul fight. Imagine a city built for a million residents suddenly accommodating five million—the strain is immense. This is why specialized planning is crucial. This article explores general supply chain planning and technical program management principles of peak events. 

Demand-side planning: Forecasting with precision and agility

Effective demand forecasting is at the core of any successful peak event. For hyperscalers, this means understanding and predicting the massive and dynamic demands on their cloud infrastructure during peak events. The complexity goes far beyond simple server provisioning or storage capacity—it’s about anticipating and preparing for varying types of customer needs across regions and services.

Leveraging historical data, agility and adjustments: Many companies use historical data to inform their forecasting, and hyperscalers are no exception. Data from past events provides a solid baseline, revealing predictable spikes in usage patterns—traffic surges, peak processing times, and usage hotspots. However, relying solely on historical data can be risky, especially as user behavior shifts due to new products, promotional campaigns, or even unpredictable external factors. While historical data serves as a foundation, modern forecasting involves much more. Involving various customer-facing teams is essential to capture a more accurate, real-time understanding of what’s happening in the market. These teams—ranging from product management to sales and marketing—bring valuable insights into customer behavior. Close collaboration with these teams means hyperscalers can tweak their forecasts on the fly and ensure they’re not caught off guard.

Co-planning of events and resources: There’s no such thing as a single peak event—it often exists alongside other smaller or larger events. Effective demand planning must therefore encompass a broader view, recognizing the interconnectedness of multiple peak events and their potential impact on resource needs. Considering the intricate web of dependencies in resources is also equally important. For instance, while forecasting a customer’s need for virtual machines, teams must also consider supporting elements like persistent disks and network bandwidth. True co-planning requires a holistic approach, encompassing the full spectrum of cloud resources—compute, storage, networking, and even underlying hardware. By fostering cross-functional communication and mapping dependencies, hyperscalers can create a comprehensive picture of demand, minimizing the risk of last-minute scramble.

Supply-side planning: A balancing act

On the supply side, peak event planning requires careful consideration of how to fill in the supply gaps in response to the forecasted demand. 

Phasing orders: A major challenge during peak event periods is scaling resources in a way that ensures smooth delivery without overwhelming internal systems. Hyperscalers typically rely on phased orders to incrementally increase capacity. This phased approach allows the teams in the data center to scale operations more efficiently, minimizing the risk of bottlenecks that can arise from scaling all at once. For example, receiving all the required supply for a peak event in a data center simultaneously can overload staffing resources and strain both the physical (staging space, elevators) and digital (quality and software checks) infrastructure of the facility.

Avoiding infrastructure bottlenecks: Since standard ordering quantities can increase manifold during peak events, it’s crucial to recognize that accommodating this surge can impact infrastructure across multiple layers. One key example is power. There are ample examples of the ever-increasing power requirements in data centers. In rapidly growing hyperscalers, which are typically experiencing constant expansion, certain pockets or clusters may see very high utilization. These areas are at the greatest risk of being affected by power constraints, as power planning and implementation require several quarters. Additionally, various secondary and tertiary requirements may also become constrained, and addressing these infrastructure needs is even more challenging. Therefore, it’s essential to account for peak events in long-term planning to prevent bottlenecks.

Unique characteristics of peak event planning and execution

While traditional planning may be flawless, immense peak event demand introduces unique aspects that must be considered beyond the standard process. Without these considerations into the overall planning, long-term success will be challenging.

Exaggerated problems at scale: What may appear as a minor issue in a normal environment can escalate into a major problem during peak events. From system bottlenecks to small architectural flaws, the heightened demand during peak events exposes vulnerabilities that might otherwise remain hidden. This underscores the importance of proactively addressing potential issues and continually improving systems and processes. Stress testing becomes crucial in identifying and addressing these recurring weak links. 

Leveraging special levers: Hyperscalers often have access to special levers during peak events—whether it’s extra cloud resources, engineering support, or specialized capacity. Knowing when and how to leverage these levers is crucial. It’s not just about increasing raw capacity—it’s about creating the right environment to handle unexpected surges without impacting customer experience. The ability to react within manufacturing lead time can make or break a peak event. 

Adaptability in the midst of change: While peak events are critical, other ongoing programs and system enhancements continue alongside the event. Hyperscalers must be ready to adapt their peak event plans to accommodate new tool releases, product changes, or infrastructure updates happening simultaneously. Preparing for these shifts ensures teams stay agile and responsive. Effective peak event planning includes creating a forecasted snapshot of key systemic elements and identifying potential gaps in the planning and execution process.

Technical Program Management: Structuring for success

A critical but often overlooked aspect of peak event planning is Technical Program Management (TPM). The role of TPM in organizing and executing a successful peak event is immense. This isn’t just about managing servers or scaling infrastructure—it’s about creating a program structure that can handle unforeseen challenges while ensuring that resources are deployed effectively and efficiently.

Creating a robust program framework: The program must be structured in a way that addresses every aspect of the event. A ‘MECE’ (Mutually Exclusive, Collectively Exhaustive) approach should be taken to arrive at the necessary workstreams of the program. Then, establish clear owners for each workstream, defining roles early, and ensuring tight communication across teams. TPM also involves creating metrics for tracking the overall status of the program which allows teams to anticipate and respond to potential issues before they become full-scale problems.

Proactive risk management: While program managers prepare for the expected, they must also plan for the unexpected. Even with the best preparations, something is bound to go awry. The ability to pivot and handle unforeseen challenges—be it a surge in demand or an infrastructure failure—can be the difference between a smooth operation and a full-scale crisis. Proactive risk management is a cornerstone of effective TPM, and having systems and communication channels (especially with end customers) in place to track and resolve issues as they arise ensures minimal downtime during the peak event.

Continuous post-event evaluation: The planning process doesn’t stop once the peak event concludes. After every major event, it’s crucial to conduct thorough post-event retrospectives. These evaluations allow teams to identify which strategies worked well, which areas need improvement, and what lessons can be applied in the future. Continuous feedback loops help improve both the technical infrastructure and the program management strategies, ensuring that each peak event is more efficient than the last.

Conclusion

In conclusion, peak event planning for hyperscalers is a dynamic and multifaceted process that requires collaborative forecasting, multi-layered supply management, and seamless coordination across teams. As these events continue to grow in scale and complexity, hyperscalers must not only anticipate surges in demand but also adapt to evolving challenges in real-time. Ultimately, the ability to stay nimble during unforeseen challenges will be key to maintaining the reliability and performance that customers rely on during the year's busiest shopping period.

About the Author: 

Achyutha Mohan is a Technical Program Manager in Supply Chain at Google. With over 12 years of experience in supply chain planning, he has worked across diverse industries, including hyperscalers, semiconductor manufacturing, and B2B retail. Achyutha holds a Master’s degree in industrial engineering from Clemson University.

SC
MR