General North Texas is getting lake effect snow.

Welcome to our Community
Wanting to join the rest of our members? Feel free to Sign Up today.
Sign up
Splinty

Splinty

Shake 'em off
Admin
Dec 31, 2014
44,116
89,900

View: https://www.reddit.com/r/Austin/comments/lk7cgn/ercot_and_the_rolling_blackouts/gnii3el


Great thread about how close we got to losing the whole grid and what ercot did wrong in response.




We're now in level 3... hello rolling blackouts.

This was several hours earlier than I predicted... tomorrow's 5 GW peak shortfall estimate (from six hours ago) is now 8 GW... for ERCOT to miss their forecast by that much is quite an anomaly. This suggests one large generating plant - or several smaller ones - have gone offline (due to the cold, lack of fuel, etc).

The next 48 hours are going to be miserable for ERCOT customers.

In the past 6 hours, ERCOT's instantaneous time error (ideally 0.0) has dropped a staggering ~~22.5~~ 29.5 seconds! In other words, the load on the system is so severe right now that the generating equipment is actually slowing down... that means the pedal is to the metal and the plants just can't keep up. Put another way, the generating system as a whole is ~~1350~~ 1770 electrical cycles behind where it should be.

Update: Looking at ERCOT's data, it appears a large generating facility had an unexpected outage at around ~~1:55~~ 1:30 am: the actual generating capacity dropped nearly 2 GW in a five minute period, but this wasn't wind-related. I haven't found any details yet, but it seems like ERCOT has initiated the [Hot Shots landing](https://www.youtube.com/watch?v=6BHeaEat_aI).

Update2: Even with rolling brownouts (to shed load), ERCOT's instantaneous time error has dropped another second in just ten minutes! Yikes, this does not bode well for the next couple days. ERCOT is no longer within specified frequency tolerance.

Update3: ERCOT's instantaneous time error is now dropping off a cliff. In the past minute it's dropped another second. I've NEVER seen that happen before...

Update4: Yikes, ERCOT is running at 59 Hz! The instantaneous time error (ITE) is dropping one full second every minute, with QTY9 BAAL exceedances!

Update5: ERCOT just pulled out of a nosedive... they're now somehow managing to chew away at the ITE. IMO that's not the correct thing to do right now, but it's more likely that so many providers are now load shedding (brownouts) that they've overcompensated and the machinery is now speeding back up. Guys and gals, ERCOT briefly dipped down to 59 Hz... I'm not sure that has ever happened before. If ever you wanted to crash your power grid, that's how you would do it. IMO, ERCOT waited too long to issue these warnings... they followed their protocol to a 'T', but I'm gonna just go out there and propose that the protocol needs to change. **We just witnessed a rapid loss of control, saved just in time by providers implementing rolling brownouts.**

Update6: ERCOT's generating capacity has dropped a staggering ~~13~~ 15 GW in the past ~~ninety~~ hundred minutes! Things are not going well at Texas' generating facilities tonight. That's equivalent to three VERY LARGE power plants having unexpected outages.

Update7: ERCOT has 'stabilized' the system, and by 'stabilized' I mean rolling brownouts are keeping the generating capacity at a constant state. By my estimation, on average two million customers are without power - due to rolling brownouts - at any given time. The most important thing right now is that ERCOT keep whatever generating capacity that is available steady... there was a brief moment where it looked like ERCOT might lose control, but **right now things seem stable**, albeit compromised. I'm off to bed for now. We'll see what actually happened once ERCOT releases a press release in the morning. Yikes, what a ride!

Update8 (FEB15 12 noon): ERCOT's poor planning (long term, not just last night) has now caused Texas to lose 30% of its generating capacity. When this is over, we need to insist that ERCOT change both the emergency response load-shedding model, and their worst-case temperature event model. ERCOT's poorly-devised playbook last night has caused what should have been an 8 GW shortfall to become a 22 GW shortfall... in real-world terms, that means six million additional homes are without power, versus what *could* have happened if the grid didn't nearly fall off a cliff last night.
Note: Both numbers I mention above were possible given the *current* weather conditions. The additional 14 GW shortfall is a direct result of ERCOT's emergency measures further destabilizing the (already unstable) grid... telling everyone to start brownouts at exactly the same time (as happened last night) destabilized the grid so greatly that several baseload plants offlined themselves (so they wouldn't blow up). [Here are my long term suggestions](https://old.reddit.com/r/Austin/comments/lk8fxb/ercot_has_declared_an_eea_3_we_will_experience/gnjyil1/) to ERCOT.

Update9 (FEB18 1PM): [Here's ERCOT finally telling us](https://www.kxan.com/news/texas/ercot-controlled-outages-over-power-can-be-restored-across-texas/) how close we came to total blackout.

.............................

Here's my [post mortem](https://old.reddit.com/r/news/comments/lmywy0/ercot_texas_was_seconds_and_minutes_away_from/gny30t6/) explaining why ITE is an important metric when the wheels are falling off your 3rd world power grid.
Click to expand...
 
Hauler

Hauler

Been fallin so long it's like gravitys gone
Feb 3, 2016
47,002
58,991
Splinty said:
Did you guys know our autoclaves can't run without water?
Yikes.
We can do surgery as far as our existing kits. Can't cycle them until water... tomorrow? Saturday?
Click to expand...
I didn't know what an autoclave was.
But a quick google search has made me an expert.

They are steam sterilizers.
You need water for steam.

So, yeah.
 
M

member 1013

Guest
Hauler said:
I didn't know what an autoclave was.
But a quick google search has made me an expert.

They are steam sterilizers.
You need water for steam.

So, yeah.
Click to expand...
I knew because I have managed several hospital builds and upgrades

but I didn’t want to brag
 
ThatOneDude

ThatOneDude

Commander in @Chief, Dick Army
First 100
Jan 14, 2015
35,390
34,114
Hauler said:
Think they've learned from this experience?

I should have thought of this earlier, but Generac stock is up $100 since 2/1.

From $250 to $350 in 17 days.
Click to expand...
They haven't learned from the last time, so I doubt it.
 
ThatOneDude

ThatOneDude

Commander in @Chief, Dick Army
First 100
Jan 14, 2015
35,390
34,114
I know nothing about generators, anyone that does what do you think of this?

www.firmanpowerequipment.com

FIRMAN Power Equipment

The power you need, when you need it + expert guidance to help you conquer anything life throws your way. Shop now.
www.firmanpowerequipment.com www.firmanpowerequipment.com
 
Enock-O-Lypse Now!

Enock-O-Lypse Now!

Underneath Denver International Airport
Jun 19, 2016
12,137
20,127
with everyone's focus on Ted Cruz taking a flight to Cancun..let me divert the spotlight a few hundred miles away to Memphis Tennessee where UPS,FedEx, and Amazon all have central hubs.

Since Monday most flights out of Memphis have not gone out due to the Ice/Snow Storm.

That means all those Hubs have thousands of packages that have not moved since Monday - many flights into Memphis have also been haulted so if you had sent out something important this week more then likely it is still sitting at your local station.

Hate to be a patient in a Hosptal waiting desperately for some medication this week.


This distribution line has been temporarily shut down ...keep this in mind in the future how delicate the system really is.
 
ThatOneDude

ThatOneDude

Commander in @Chief, Dick Army
First 100
Jan 14, 2015
35,390
34,114
Enock-O-Lypse Now! said:
with everyone's focus on Ted Cruz taking a flight to Cancun..let me divert the spotlight a few hundred miles away to Memphis Tennessee where UPS,FedEx, and Amazon all have central hubs.

Since Monday most flights out of Memphis have not gone out due to the Ice/Snow Storm.

That means all those Hubs have thousands of packages that have not moved since Monday - many flights into Memphis have also been haulted so if you had sent out something important this week more then likely it is still sitting at your local station.

Hate to be a patient in a Hosptal waiting desperately for some medication this week.


This distribution line has been temporarily shut down ...keep this in mind in the future how delicate the system really is.
Click to expand...
I ordered a splitting ax off amazon, wont be here until the end of next week, this is unacceptable!
 
Lukewarm Carl

Lukewarm Carl

TMMAC Addict
Aug 7, 2015
31,000
51,649
Splinty said:
Did you guys know our autoclaves can't run without water?
Yikes.
We can do surgery as far as our existing kits. Can't cycle them until water... tomorrow? Saturday?
Click to expand...
Depending on what other systems you have you may still be good to go. Certain Sterrad or Steris machines don't require water since they use H2O2. But it would be super inconvenient and there's still the chance that you don't have one of those. Plus... You can't properly wash anything before it gets wrapped to be autoclaved so yeah... You're fucked without water.
 
You must log in or register to reply here.
Top Bottom