One fallen power line exposed a growing AI data center problem. Here’s how to fix it.


A power line went down outside of Washington, DC, this week. Normally, the grid would only need a few seconds to recover from such an event. But this one took more than 10 minutes because more than 3 gigawatts of data centers stopped drawing power nearly simultaneously.

The event caused voltage across the PJM grid to spike from Northern Virginia to Chicago, according to data collected by Ting Labs, a startup that runs an IoT sensor network out of people’s electrical sockets.

The event didn’t cause a blackout, but it did cause lights across the region to flicker. The incident demonstrated the effect that data centers can have on the grid — an outcome that experts believe will become more frequent.

Northern Virginia, which is in PJM’s territory, is home to the highest concentration of data centers in the world.

“It’s the canary in the coal mine,” Ricardo de Azevedo, CTO at ON.Energy, told TechCrunch. These sorts of events involving large loads like data centers are “happening more and more,” he added.

The event echoes one that happened two years ago, also on PJM’s grid, and it could foreshadow larger events if data centers aren’t built to more elegantly handle disruptions to power supplies. The PJM Interconnection manages grids from New Jersey to Illinois and serves 67 million customers, making it the largest grid operator in the United States.

When the power line went down this week, it triggered data centers to switch to backup power, and about 3.1 gigawatts of load vanished in about 30 seconds, according to PJM data. The grid appeared to recover somewhat, but a short time later additional loads dropped off. At its peak, PJM’s grid had an extra 3.49 gigawatts of electricity on it. It took another 11 minutes before it stabilized. The disconnected data centers represented around 3% of total demand on PJM at the time, according to Reuters.

A few percent may not sound like much, but the electrical grid needs to operate in a state of near-perfect balance, with supply and demand closely matched. If they don’t, voltages can sag or spike. The grid and devices connected to it can tolerate small fluctuations, but if those fluctuations grow too large, they’ll trigger failsafes within the grid or within individual facilities, causing them to disconnect. 

When data centers in Northern Virginia sensed the fluctuation caused by the failed power line, they switched to backup power, which removed their load from the grid. As more data centers made the switch, they removed greater amounts of load from the grid. What started as a relatively small drop in supply became an even larger drop in demand, sending supply surging and causing light bulbs to flicker. 

Most data centers make decisions in a split second, and those that disconnected this week appear to be no different. When the voltage dip reached them, they all decided to disconnect within a few seconds of each other, Ali Zain Banatwala, senior market models specialist at the Independent Electricity System Operator, told TechCrunch. 

“We need to figure a way for these loads that are located next to each other to sequentially either disconnect or reconnect,” he said. A more orderly process would allow grid operators to develop more robust procedures in advance.

Alternatively, data centers could be built to absorb disruptions and not turn their backs to them. One startup, ON.Energy, has been working on a product to help data centers — and the grid — ride through events like the one that occurred this week. 

The company has developed an uninterruptible power supply for an entire data center campus, covering not just servers but also chillers and other equipment. The company essentially hides the data center behind a bank of batteries connected to sophisticated power conversion equipment. All the grid “sees” is one consistent, well-behaved load rather than the peaks and valleys from each individual part of the data center. ON.Energy’s system allows data centers to ramp computing workloads up and down, including AI training, without bothering the grid. 

Perhaps more important, it also means that data centers can absorb power fluctuations from the grid. Rather than disconnecting from the grid, ON.Energy’s system can use any extra power to charge its batteries, and if the flow dips, the system can dispatch power to servers. Plus, it can follow the grid’s lead within milliseconds, preventing sags or surges like the ones that caused this week’s problem for PJM.

ON.Energy is currently installing a total of 3 gigawatts worth of its systems at four different data center campuses, de Azevedo said.

Grid managers have also woken up to the problem.

ERCOT, for example, is going to require large loads like data centers to “ride through” disruptions, de Azevedo said.

The clock is ticking, though. The mass disconnection this week was twice as large as a similar event in 2024, when 60 data centers simultaneously disconnected, pulling 1.5 gigawatts of load from the grid. Back then, data centers accounted for about 6% of PJM’s load, according to Synapse Energy Economics. By 2040, they are expected to make up 24%. If the problem isn’t addressed soon, things could get a lot worse.

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.



Source link

  • Related Posts

    This Week’s Awesome Tech Stories From Around the Web (Through July 25)

    Artificial Intelligence OpenAI Says Its AI Models Went Rogue and Attacked a Digital LibraryKate Conger | The New York Times ($) “The trial was designed to keep the models in…

    Avatar Aang: The Last Airbender is the ATLA Movie We Should Have Gotten in 2010

    In 2010, there was a movie called The Last Airbender, which purported to be an adaptation of the beloved Nickelodeon show. However, the M. Night Shyamalan movie felt so soulless…

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    You Missed

    Georgia’s Okefenokee swamp becomes UNESCO World Heritage site

    Georgia’s Okefenokee swamp becomes UNESCO World Heritage site

    They grew up on the same Annex street. Decades later, she walked into his Broken Social Scene show — and they fell in love

    They grew up on the same Annex street. Decades later, she walked into his Broken Social Scene show — and they fell in love

    The Hundred highlights: Sunrisers Leeds vs Southern Brave – Issy Wong’s starring role

    The Hundred highlights: Sunrisers Leeds vs Southern Brave – Issy Wong’s starring role

    This Week’s Awesome Tech Stories From Around the Web (Through July 25)

    This Week’s Awesome Tech Stories From Around the Web (Through July 25)

    How the lighthearted social media trend of ‘teen takeovers’ took a serious turn

    How the lighthearted social media trend of ‘teen takeovers’ took a serious turn

    Canadian intern at NATO military HQ arrested in Belgium on spying charge – National

    Canadian intern at NATO military HQ arrested in Belgium on spying charge – National