Venezuela’s melt down: Blackout day six and the grid struggles to reboot

Venezuela has 31 million people and has had almost no electricity for six days. There are estimates on Twitter (#sinluz) suggesting that about half is back up as I write, but the stories of chaos, death and disaster are surely just starting to come out as communication lines open.  The water coming out of taps is black, possibly contaminated with oil (are those shots fake?),  the Pepsi plant was stripped bare, see the video. People are desperate:  Shop owners are apparently shooting looters. At least one tweeter claims people are eating zoo animals. One baker took his own life after his shop was overrun and everything was stolen.  It may not be over yet either — the grid  recovered almost as far a few days ago, then collapsed again. Indeed, today explosions have been reported at an electrical substation at La Tiama, as well as other substations. What a debacle, a disaster. Babies in hospitals are being ventilated by hand. Many life support systems are off.

Netblocks tracks connectivity in Venezuala, which seems to be a reasonable proxy for power, and clearly electricity is being rebuilt partially, then collapsing again.

Update: 119 hours after the onset of nationwide power outages #Venezuela‘s connectivity is up to 63%, marking significant progress in the restoration of utilities #SinLuz #12Mar ⬇️https://t.co/8pljYDEYae pic.twitter.com/zcBeVkqnKG

— NetBlocks.org (@netblocks) March 12, 2019

On the BBC one journalist describes the situation as like living the apocalypse. “They never thought it would come to this”. And he is not even referring to the armed pro Maduro regime gangs that are allegedly arriving on motorbikes and firing on protestors. See @MarcoRubio In the video the crowd is screaming and running for their lives.

Venezuela shows how fragile (and marvellous) a working grid is

History books will be written about this crisis. Matias Delacroix points out in Wired, that it’s very difficult to restart a decentralized grid which has been badly managed and poorly maintained. No one knows exactly what went wrong to bring it down, but fingers are pointing at the huge Guri Hydro plant, which provides a whopping 80% of the electricity.

Normally a blackstart begins with a small diesel unit to kick over a bigger turbine. Then the whole grid is gradually rebuilt bit by bit by “bootstrapping”. At all times the supply has to match the demand, so before an engineer throws the switch someone has to have already done the numbers to make sure things will stay balanced. As each new generator-load segment is added it must also match the frequency and phase set up by the original first generator.  It would be dangerous to set up a bunch of little separate grids and then try to meld them together. Getting the frequency or phase wrong can cause explosions. Hence the grid must be built from one point out, and carefully. Wind or solar generators can’t be used in a black start, but a large Hydro plant should be able to do that, assuming people can open the water gates, but restarting a badly managed grid means there are many fragile failure points that don’t reboot properly. After days of darkness it must also be difficult to estimate how much load will be waiting to take new power.

 WHY IT’S SO HARD TO RESTART VENEZUELA’S POWER GRID

Matias Delacroix, Wired

While distributed systems don’t have a single point of generation failure, they can be more difficult to black start if they do go down, since more generation sites need to be bootstrapped and there are more loads to balance.

Regardless of the setup, the crucial component of all black starts is understanding what caused the outage, having the ability to fix it, and working with a system that can handle the power surges and fluctuations involved in bringing power back online. Without all of these elements in place, says Tim Yardley, a senior researcher at the University of Illinois focused on industrial control crisis simulations, black starts can be prohibitively difficult to execute.

“Reenergizing a grid in some ways is more of a shock to the system than it operating in its norm,” Yardley says. “If infrastructure is aging, and there’s a lack of maintenance and repairs, as you try to turn it back on and try to balance the loads you may have stuff that’s not going to come back up, infrastructure that’s been physically damaged or that was in such a bad state of repair that reenergizing it causes other problems.”

I asked a network engineer once what happens to the energy if, say a large turbine, was connected to a grid at the wrong frequency. Energy can’t just disappear, and if the phase or the frequency flatlined, and and opposing waves cancelled each other out, where would all the joules go?  He replied that it could be catastrophic — those kind of events shear through the steel turbine shaft (and these turbines can weigh up to 600 tons). When there are megawatts of energy at stake, mistakes are violent.

Crews attempting to deal with black-starting a frail and brittle grid also face major safety considerations, like explosions. “You have a maintenance issue and a manpower issue, because it’s extremely dangerous to reenergize a system if you have gear that hasn’t been maintained well,” Yardley notes.

Venezuela has faced years of power instability since about 2009, including two major blackouts in 2013 and a power and water crisis in 2016. At times the blackouts were caused in part by weather conditions like El Niño, but overall they have established a pattern of poor planning, mismanagement, and lack of investment on the part of the government.

Yay. Socialism.

UPDATE: China is offering to help get the grid running.

UPDATE: This may take months

Brilliant comment from Lance #5 explaining why this is a nightmare for grid managers and may take 3 – 6 months to sort out. Or longer if they blow too many substation transformers which are custom made.

Lance     March 14, 2019 at 4:45 am · Reply