Reads very similar to some blackouts we had in Australia. Weakly connected grids with vast geographical distances leading to oscillations that took down the grid.
https://en.wikipedia.org/wiki/2016_South_Australian_blackout
Completely solved with lithium based grid storage at key locations btw. This grid storage has also been massively profitable for it's owners https://en.wikipedia.org/wiki/Hornsdale_Power_Reserve#Revenu...
Australia currently has 4 of the 5 largest battery storage systems under construction as a result of this profit opportunity; https://en.wikipedia.org/wiki/Battery_energy_storage_system#...
You can also read numerous stories of how Australia's lithium ion grid storage systems have prevented blackouts in many cases. https://www.teslarati.com/tesla-big-battery-south-australia-... The fact is that the batteries responsiveness is the fastest of any system at correcting gaps like this. 50/60hz is nothing for a lithium ion battery nor are brief periods of multi-gigawatt draw/dumping as needed.
There's even articles that if Europe investing in battery storage systems like Australia they'd have avoided this. https://reneweconomy.com.au/no-batteries-no-flexibility-spai...
> nor are brief periods of multi-gigawatt draw/dumping as needed.
Actually this is typically an issue for grid batteries.
Spinning generators can easily briefly go to 10x the rated current for a second or so to smooth out big anomalies.
Stationary batteries inverters can't do 10x current spikes ever - the max they can get to is more like 1.2x for a few seconds.
That means you end up needing a lot of batteries to provide the same spinning reserve as one regular power station.
Collectively Australia's battery storage systems will be able to beat any single power plant for peak output in Australia once fully built out based on pure numbers. But for these sorts of grid oscillations the more important thing is the localization of generation. Which obviously favours the batteries over large centralized power stations in any case.
Being enthusiastic about battery technologies is one thing, spreading misinformation to support someone convictions is an other.
What causes the Iberian blackout is excessive reactive power and a lack of compensation at a given time (due to multiple factors).
Compensation of reactive power has strictly nothing to do with localization of generation. It barely even can be assimilated to oscillations
That's complete garbage and it shows mainly you do not seem to know much about Electricity power in general.
> Spinning generators can easily briefly go to 10x the rated current for a second or so to smooth out big anomalies.
I'm very suspicious of this, because it would also imply 10x overcurrent on the associated transmission gear. There's limits on how much you can overcurrent a transformer before the core magnetically saturates, for example. Also I would expect protection systems to trip out at such a huge divergence from rated current. Do we have a citation?
A spinning generator is not outputting 10x it's rated current over any significant amount of time. You can only add so much steam or fuel to a turbine, and the rotor has a lot of inertia, but not enough to account for 10X its rated capacity for a second. The electrical switchyards would trip nearly instantaneously if it's connected plant output 10X its rated input.
A properly running grid with spinning generators never needs the 10x rated current. More importantly, oscillations are dampened by the rotating masses.
Let alone the transmission lines totally catching fire after a 10X output increase.
Where are you getting this information from?
Although I’m hardly an expert on power lines(my factory produces HV switchgear), a 1s short circuit current rating of 10x(actually more) is normal, standard to IEC norms.
You can if the load was 10x under max. before getting tripped.
Misaligned oscillation can occurs under ANY load.
Batteries would also be able to 10x if their load was 10x under max, so I'm not sure how this is relevant to GP's point.
The inverter would melt presumably.
Just looking for an excuse. Lol
That... doesn't sound correct. Inverters are the cheap part, you can literally wire as many as you want in parallel. Batteries have immense power availability, with most chemistries you can trivially deliver the entire capacity in half an hour or so (more like 5 minutes with lithium cells).
Basically I'm dubious. I'm sure there are grids somewhere that have misprovisioned their inverter capacity, but I don't buy that battery facilities are inherently unable to buffer spikes. Is there a cite I can read?
Agreed. The relatively small battery substation linked above can output 2GW of equivalent inertia generation (a measure to align batteries to inertial power systems) when needed. That's an entire power station they can match for short periods of time. Link: https://www.energymagazine.com.au/sa-approves-world-first-ba...
Australia's largest power plant has 2.9GW of inertial generation assuming all generators are running at 100%. As in the small battery substation alone comes close to the countries largest power station. I'm not sure where the idea that lithium ion can't dump power quickly comes from. They are absolutely phenomenal at it. Australia's building dozens of these substations too since they are so cheap and reduce overall power costs. It's a win from all points of view.
The article does not describe an inertia constant. Without the time component, any comparison to traditional systems is meaningless. Inertia is a measure of energy, not power.
Large spinning masses can provide several seconds of inertia. For 2GW of traditional turbine, you would have between 10-20 gigawatt-seconds of energy that is instantly available at any moment to resist RoCoF.
> assuming all generators are running at 100%.
which they won't ever be given the habits of coal plants to suffer outages whenever it's convenient to pump the price up.
> Inverters are the cheap part
The whole point with actual inertia is that you get a large multiple of your maximum capacity without any redundant parts or added system complexity.
Keeping around 10x+ more semiconductors than you need to cover a tiny fraction of operational scenarios is difficult economics.
A semiconductor device cannot be overloaded like a spinning generator or transmission infrastructure can. You cannot trade temperature and maintenance schedule for capacity in the same way. Semiconductors have far more brittle operating parameters.
it is technically correct, but so are you.
More inverters in parallel will achieve the same end goal - fast frequency response.
Are you actually certain there are insufficient inverters though? Again, that doesn't pass the smell test and I'd want to see a cite for "batteries don't work for high frequency spike buffering because of inverter shortfalls" or something.
i'm certain there are more than enough inverters.
what's correct is that each individual inverter can only increase its power output momentarily to 20% or so above its maximum. Add more inverters and that problem is solved.
You can google "system inertia" as a starting point.
When it comes to the grid, there's a lot of outdated information left over from the 20th century, so any web search for "system interia" needs to also include some searches on "grid forming inverter"' to make sure that the info is complete.
(And "reactive power" could be good too but not absolutely necessary to understand at first...
This is what batteries can provide very well.
I understand the concept. I was asking for a cite about the seemingly-incorrect point about batteries. FWIW, that very search term doesn't produce the string "battery" anywhere on the first page.
Seems like pumped hydro offers a nice compromise.
Fwiw the hornsdale battery linked above cost AUD$172 million and can provide 2000MW of equivalent inertia. Link: https://www.energymagazine.com.au/sa-approves-world-first-ba...
That equivalent inertia can only be done for short periods but that's exactly what grids need in stability - there's generally no lack of total generation, just a need to jump in and smooth out spikes.
You can't build a dam for that price, nor could you do it in under 100 days from contract signing as that battery was built. Batteries are definitely the answer here. The 'more spinning mass' answers don't make sense since Australia literally solved the above problem in a much cheaper way already.
You don't need to build a dam, you just need the pipes and pumps for an existing dam (or elevated natural basin).
I thought you needed two dams. One higher than the other. You pump water back and forth between the two to generate or capture energy.
Is it that common that dams are already existing in nearby-ish pairs with the sufficient height difference? And that we haven't done this already?
Doing this is good where we can. But it has geographical limitations. Batteries don't so much.
My layman's understanding is that most locations that are suitable for pumped storage already have pumped storage built on them.
Yep, this is the issue. That and land cost. Also pumped hydro is most useful when you need very large capacity storage, whereas for preventing blackouts you need very high capacity fast generating to fix oscillations or to allow more generating capacity to come on line. They are basically acting as decoupling capacitors (except for AC) in this application.
You need two sufficiently large bodies of water close to each other at different elevations. You don't necessarily need two dams - for instance, the Ludington pumped storage plant adjacent to Lake Michigan uses the lake as the lower body.
Most economically-suitable locations for pumped hydro have been built out already.
You can always use a ton more concrete and force new locations, but the best locations have already been utilized and scaling law of batteries has brought them to the point where they're more competitive than new hydro for this kind of use.
Most economical locations for hydro generation have been built out already. Pumped hydro doesn't require flow like generation does, so there are thousands of times more suitable locations. Those haven't been built out.
At least in my country they have. They keep trying to build a facility but anything they come up with, the ROI just isn't there.
Sure, if you can site it.
What batteries CAN do however is go from 0W to their capacity watts in milliseconds. It is their instant-on ability that is disruptive.
Not only from 0W, but from negative capacity (max charging power) to positive capacity (max discharging power).
> Completely solved with lithium based grid storage at key locations btw.
That's because Australia has a moderate amount of renewables and prefers to burn fossil fuels. Right now, around 25% of the electricity in Australia is generated by solar or wind.
Spain is past 50% of renewable generation, and their problems are much bigger.
But Australia is tiny: 27M people. Just Spain is twice as big, and the European grid serves 500M people, we don't have the same problems, and probably can't solve them with the Australian solutions.
I wouldn't dismiss lessons from Australia. It has slightly higher electricity consumption than Spain (273 vs 265TWh/annually [1].) And there are limited interconnections to France [2]:
> That problem may not have been entirely Spain’s fault, El País said. “Interconnections with the rest of the continent continue to be much fewer than the European Commission recommends, not because Spain isn’t interested, but because France has for years resisted expanding them.”
[1] https://en.wikipedia.org/wiki/List_of_countries_by_electrici...
[2] https://www.theenergymix.com/massive-blackout-in-spain-shows...
[dead]
Seems the money shot starts at page 131:
> The ultimate cause of the peninsular electrical zero on April 28th was a phenomenon of overvoltages in the form of a "chain reaction" in which high voltages cause generation disconnections, which in turn causes new increases in voltage and thus new disconnections, and so on.
> 1. The system showed insufficient dynamic voltage control capabilities sufficient to maintain stable voltage
> 2. A series of rhythmic oscillations significantly conditioned the system, modifying its configuration and increasing the difficulties for voltage stabilization.
If I understand it correctly (and like software, typical), it was a positive feedback-loop. Since there wasn't enough voltage control, some other station had to be added but got overloaded instead, also turning off, and then on to the next station.
Late addition: It was very helpful for me to read through the "ANNEX X. BRIEF BASICS OF THE ELECTRIC SYSTEM" (page 168) before trying to read the report itself, as it explains a lot of things that the rest of the report (rightly) assumes you already know.
I’m a bit mystified as to how the grid controls voltage at all. The non-renewable plants follow the rules here:
https://www.boe.es/buscar/doc.php?id=BOE-A-2000-5204
7.1(b) seems to be saying that generators connected at 200kV adjust their reactive power generation/absorption in real time according to the voltage they observe, based on a lookup table provided by the grid operator.
This seems sort of sensible according to my limited understanding of the theory of AC grids. You can write some differential equations and pretend everything is continuous (as opposed to being a LUT with 11 steps or so), and you can determine that the grid is stable.
However, check out this shorter report from red eléctrica:
https://d1n1o4zeyfu21r.cloudfront.net/WEB_Incident_%2028A_Sp...
Apparently these 220kV plants are connected to the 400kV grid via transformers in substations that are not owned by the generator operators. And those transformers have “tap changers” that attempt to keep the 220kV secondary side at the correct voltage within some fairly large voltage range on the 400kV side. Won’t this defeat the voltage control that the 220kV generators are supposed to provide? If the grid voltage is high, then absorption of reactive power is needed [0], and the generators are supposed to determine that they need to absorb reactive power (which they can do), but if the tap changer changes its setting, then the generator will not react correctly to the voltage on the 400kV side.
In other words, one would like the generator to absorb reactive power according to P_reactive(primary voltage • 220/400), but the actual behavior is P_reactive(primary voltage • 220/400 • tap changer position), the tap changer position is presumably something like 400/primary voltage, and I don’t understand how the result is supposed to function in any useful way. Adding insult to injury, the red eléctrica repoet authors seem to be suggesting that a bunch of tap changers operators didn’t configure their tap changes well enough to even keep secondary voltages in range.
Does anyone with more familiarity with these systems know how they’re supposed to work?
[0] I can never remember the sign convention for reactive power.
The automatic voltage regulator (AVR) of the generator and the on-load tap changer (OLTC) operate on different timescales (AVR = quick; OLTC = slow). OLTCs are typically set up with a voltage deadband and a time delay to prevent 'hunting' (i.e., repeatedly tapping up and down).
I don't claim to know the details of reactive power management, but the primary mechanisms for grid stability in the EU is the "cascade" of services the TSOs procures:
- Fast Frequency Response (FFR), sub-second power adjustment following frequency table
- Frequency Containment Reserve (FCR), ~second power adjustment following frequency table
- Automatic Frequency Restoration Reserve (aFRR), ~second energy production following TSO setpoint signal
- Manual Frequency Restoration Reserve, ~minute energy production following TSO activation signals
My understanding is the primary failure in Spain was that 9 separate synchronous plants that had sold aFRR(?) to the TSO then failed to deliver, so when the TSO algorithms tried to adjust the oscillations, nothing happened. Everything else was kinda "as designed".
> 9 separate synchronous plants that had sold aFRR(?) to the TSO then failed to deliver, so when the TSO algorithms tried to adjust the oscillations, nothing happened.
Oof. This sounds like a classic of "it's only needed in emergencies, so it's only in emergencies that we find out it doesn't work".
I think your interpretation is correct. The voltage control is done at the high level of the grid, meaning the control covers bigger generation stations and major substations. Even if it’s small generator, rotating machinery, you won’t have strict voltage control other than its own AVR. The problem I see here is that we embed smaller individual generations at the lower level, where they pump the generated power to the grid at the medium voltage level. When you have majority of your generation at this level, you won’t have strict control over voltage and even frequency, I assume. I’m still digesting the report, but what I am after is whether they really neglected it and if it is not possible to do voltage control with 50% generation coming from renewable and through medium voltage level, aka lower level.
I’ve just read the whole shebang.
While the overall reason for the mass failure you cite is correct - a cascading failure - the interesting bit here are the oscillations that lead to it.
It looks very much like this was driven by algorithmic volatility trading of electricity spots - overproduction, price goes negative, buys placed, production ramps in response to rising price, price rises, sells placed, production falls due to falling price. The period of the oscillations in the grid seen before the blackout suggest a relatively slow cycle, and what they describe in the report sounds very much like this was an interaction between price-driven supply and real world supply.
It does speak to there being inadequate storage available on the grid to smooth demand and therefore pricing, but it also suggests that in certain conditions a harmonic can be set up between the market and price-driven production with catastrophic consequences.
It's a difficult read.
Cybersecurity and digital systems was not the issue but gets thirteen pages of proposed measures. I feel this could have been left out.
Electric System Operation was the issue and gets seven pages of proposed measures.
Check this shorter report by the operator:
https://d1n1o4zeyfu21r.cloudfront.net/WEB_Incident_%2028A_Sp...
Page 130 is where the actual human readable summary is. Although the previous pages were pretty detailed in explaining the cumulative instabilities.
Sadly, some news outlets are probably only going to look at the recommendations and read "cybersecurity" and (even though they are common sense recommendations) assume there might be more to say about the matter.
There's been a shit-ton of misinformation about cyberattacks within the first hour of the outage, and the public were unfortunately very receptive to it, so I guess they're trying to preempt those concerns?
Don't worry, some news outlets will summarize this as "renewables = bad" regardless of what the report actually says.
Oh wait, they already did: https://www.telegraph.co.uk/business/2025/06/18/renewable-en...
Ed: Do I need a /s tag here or something? My point was that we shouldn't worry too much about about the presentation of the report, its actual contents will be spun to suit any narrative regardless.
That doesn’t really say “renewables bad”.
The report actually says that there was a drop in solar generation.
Flame bait journalism is one of the things you can count on in any circumstance. If you replace that ‘renewables = bad’ with ‘politicians = idiots’ OTOH… sometimes the elected representatives should listen to unelected physicists and engineers. Grid stuff is one of those things.
So, the problem was a local voltage oscilation, where the high voltages caused generators to shut off.
How do these oscilations start? I understand that voltage isn't necessarily equal across the network, where frequency is. But that only allows oscillating, it doesn't cause it. Is this a basis inductor capacitor oscillation? Is it the small delay in inverters between measuring voltage and regulating their output? (seems unlikely, given that renewables aren't blamed) or is there some other source of (delayed) feedback.
And why do generators cut off at a high voltage? Is it a signal of 'too much power'? Is it to protect the generator from some sort of damage?
Sort of:
For the oscillations, the European grid in general is large enough that the time it takes for the energy to flow (at some fraction of the speed of light!) from one side of it to the other is not negligible: it's not a case of delays at the power plant, but delays in the network itself which can cause the various natural and artificial feedback loops in the circuit to start to become unstable and oscillate. In this specific incident, there's some implication in the report that the largest oscillation was unusual and may have been generated by single plant essentially oscillating on its own, for reasons unknown.
In either case, the oscillations were not the direct cause of the blackout: they were controlled, but the steps to control them put the system into a more fragile state. This is because of reactive power. The voltage in the system is due to both the 'real power', i.e. the power generated by the plants and consumed by consumers in the grid each cycle of the 50Hz AC, but also 'reactive power', which is energy that is absorbed by the consumers and the grid itself (all the power lines and transformers) and then bounced back to the generators each cycle. This is the basic 'inductor-capacitor' oscillation. This reactive power is considered to be 'generated' by capacitance and 'consumed' by inductance, though this distinction is arbitrary.
So, after the grid operator had stopped the oscillations, the grid was 'generating' a lot more reactive power, because damping down the oscillations generally involves connecting more things together so they don't fight each other as much. It also _lowered_ the grid voltage on average, so various bits of equipment were essentially adjusting their transformer ratio with the high-voltage interconnect to try to adjust for it.
Apart from these measures, the generators on the grid are generally supposed to contribute towards the voltage regulation, which helps with both damping these effects and reducing the change of the runaway spike that happened. But crucially, there's a difference between what they (by regulation, not necessarily technical capacity!) do. The traditional generators have active voltage control, which means they actively adjust how much reactive power they generate or absorb depending on the voltage on the lines. Renewable generators, by contrast, have a fixed ratio: they will be set to generate or absorb reactive power at a certain percentage of the real power (a few percent usually), they don't actively adjust this (they're not allowed to under the rules of the grid).
So, after the oscillation, the grid is generating a lot of reactive power and the power plants are absorbing it, but there's a lot of renewables around, which can't actively control voltage, they're just passively contributing a certain amount. Then there's a fairly rapid drop in real power output, which seems to be related to the energy market as some plants decide to curtail. This is expected, but renewables can do it pretty quickly compared to conventional plants. This means that the amount of reactive power being absorbed drops, i.e., counterintuitively a plant producing less power means the voltage rises.
In theory, there should be enough voltage control from conventional sources to deal with this, but in general they prove to not absorb as much reactive power as they were expected to, and the report calls out one plant which seems to just not be doing any control at all, it's more or less just doing something random. This means the voltage keeps rising, and, perhaps in part due to the adjustments in the transformer ratios, this means another plant trips off, at a lower voltage than it should (this is, basically, for protection: the equipment can only take so much voltage before it's damaged, but there's rules about what level of voltage it should withstand and, in extreme cases, for how long). This then makes the voltage rise more, and it's a fairly rapid cascade of failure from there, and many plants kick offline in a matter of seconds, and only then does the frequency of the grid start to drop significantly, but it's already too late because there's too much demand for the supply.
The recommendations of the report basically boil down to:
- Figure out why the plants (renewable and conventional) didn't have the capabilities the grid operator thought they did (or why they were actively causing problems), and fix them.
- Fix the regulations so renewable plants are allowed to contribute to active voltage control, and incentivize them to do so.
- Adjust the market rules so that plants have to give more notice before increasing or decreasing supply in response to prices
- Improve the monitoring of the grid and add other tools to help with voltage control (including better interconnects with the rest of Europe)
pg 132 "2.1 The characteristics of the first large oscillation (12:03) indicate that it is a phenomenon generated within the Iberian Peninsula. It is an atypical oscillation, with a frequency of 0.6 Hz, and its origin has been correlated with the operation of a facility. In addition, the system operator has identified an atypical behavior in an installation in the wind farm."
Could this have been a deliberate attempt to crash the grid?
Individual generators monitor Voltage, Frequency, and reactive power (≈ how much current is out of phase with voltage) to make decisions about injecting more or less power into the network. This is just historically how they've always been doing it.
Due to interactions between different generators, there can be instabilities causing voltage or frequency or reactive power to deviate outside of spec. A simple example might be two generators where one surges while the other drops back, then vice versa. The measurement (by the network operator) of these effects is poor for Spain - shown by the simple example that they have large oscillations that they couldn't explain.
There's path dependent healing and correction of problems by different generators, which overall leads to network stability. However the network operator here is not actually resolving cause and effect, and does not have the insight to manage their stability properly.
In this case you can see them trying a few things to inject changes that they hope will bring stability - e.g. tying many connections hoping that adding generators together into one network will resolve to a stable outcome.
Are there countries that have a better design for their electricity network control systems?
Disclaimer: I don't design electricity networks nor electricity markets. And the above is ignoring loads (loads are mostly less problematic for control than generation).
I suppose other system operators might have better a state estimator and wide area monitoring system. But real-time system operation is universally an engineer sitting behind a desk, looking at their screen, and trying to make the best decision with whatever data they have.
The actions that were taken did not strike me as out of the ordinary.
An interesting detail is that the instability had warning signs in the waveforms, that probably means that an algorithmic response is possible.
It doesn't look like this report really identifies the root causes...
I would like to see: "We have simulated the complete 200 and 400 kV grid of the iberian peninsula and western europe, and can reproduce the situation that occurred. Any one of the following changes would have prevented the issue, and we suggest implementing them all for redundancy. This simulation will be re-run every day from now on to identify future cases similar incidents could occur"
The engineers knew exactly what’s going to happen, the report is politically redacted. Very unfortunate, but completely expected.
What is curious to me is that there's a possibility that a single plant in conjunction with natural oscillations caused enough trouble to start a doom scenario.
Oscillation -> damping -> possibly faulty equipment and possibly lack of power plants to absorve the reactive load -> 0 voltage in two countries and some neighbouring regions
There's also the possibility that Portugal put too much demand on the market due to negative prices, but I'm not sure if it was explained how much that had an effect on the whole thing.
> Non-confidential version of the report of the committee for the analysis of the circumstances surrounding the electricity crisis of the April 28, 2025
Now I'm curious about what's in the confidential version of the report.
Based on the redacted bits, it is mostly company names and locations.
And more interestingly, the specific reasons various installations were unable to start up as they should have:
> Incidents detected during equipment start-up - Firstly, there is information consistent with the fact that several installations with the obligation of autonomous start-up were finally unable to provide this service in a stable manner, joining the system only once voltage had arrived from outside (from another of the "islands", normally anchored in one of the interconnections). This slowed down the start-up of the "skeleton" of the electricity system that would later make it possible to replenish the supply to demand.
The rest of the ~2 pages in that section is redacted.
It's a difficult read, so let me paste here what Claude Sonnet 4 summarised:
Based on this comprehensive report on the April 28, 2025 electrical blackout in Spain, I can summarize the key reasons why it happened:
## Primary Cause: Voltage Control Crisis
The blackout was fundamentally caused by *insufficient dynamic voltage control capacity* in the system, which led to a catastrophic "chain reaction" of overvoltages. Here's how it unfolded:
### The Perfect Storm of Contributing Factors
*1. Inadequate Voltage Control Resources* - Only 11 thermal power plants were coupled with voltage control obligations (the lowest number recorded in 2025) - One planned voltage control plant in the southwest failed the previous evening and wasn't replaced - Several connected plants didn't provide expected reactive power absorption during critical moments
*2. System Oscillations Weakened the Grid* - Multiple oscillations occurred throughout the morning (starting at 5:49 AM) - Two major oscillations at 12:03 PM (0.6 Hz) and 12:19 PM (0.2 Hz) significantly stressed the system - The first oscillation was traced to anomalous behavior at a specific photovoltaic installation - Measures taken to dampen these oscillations (increased grid meshing, reduced interconnection flows) inadvertently contributed to voltage increases
*3. The Fatal Chain Reaction (Phase 2-3)* Starting at 12:32 PM: - Voltages began rising rapidly across the transmission network - Generation facilities started disconnecting due to overvoltages, beginning with renewable plants - Each disconnection removed reactive power absorption capacity and reduced line loading - This caused further voltage increases, triggering more disconnections - The process accelerated into an unstoppable cascade
### Key Timeline - *12:32 PM*: Sustained voltage increases begin - *12:32:57*: First major generation loss (355 MW at Granada) - *12:33:16*: Second major loss (730 MW at Badajoz) - *12:33:17*: Third major loss (550 MW at Sevilla) - *12:33:30*: Complete system collapse to zero voltage
### Why Couldn't It Be Stopped?
Once the chain reaction began, stopping it would have required massive reactive power absorption capacity that simply wasn't available. The system's protective mechanisms (like demand disconnection) actually made the overvoltage problem worse by further reducing grid loading.
## Broader Context
The report emphasizes this was a *multifactorial event* - no single failure explains it entirely. Contributing factors included: - Low electrical demand creating capacitive effects in the highly meshed grid - Quarter-hourly market changes causing rapid generation adjustments - Spain's weak interconnection with Europe (only 3% vs. 15% target) - Complex renewable evacuation infrastructure with inadequate protection settings
The restoration process took until 7:00 AM the next day to reach 99.95% supply restoration, though it was considered exemplary by international standards.
When skimming through the report I got to think of the oscillation problem in RIP routing protocol. Although it isn't the same thing, but it shows the complexity of the problem to anyone who thinks there is a single solution to it.
Why so many pages of "Recommendation: implement multi-factor authentication" and other IT security irrelevancies? Did they need to pad out the number of pages?
> In the systems with network traffic evaluation probes, no records consistent with unauthorized activity have been observed, such as lateral movements, network traces or file movements for vulnerability exploitation or privilege escalation, among others.
> However, as is common in networks and information systems in any sector, other risks have been identified, such as vulnerabilities, deficiencies or inadequate configurations of security measures, which may expose networks and systems to potential risks, for which a series of measures are proposed.
Infrastructure in general has pretty terrible security practices, so I won't bemoan someone finding a useful soapbox to remind them to shape up a bit, even if it isn't the core cause of this particular issue (and it's probably also a reaction to various rumours/speculation about a cyberattack).
[flagged]
> but the fans of renewables just flagged it
As moderators we can only guess why people flag things, but there are other reasons why people may have flagged that comment.
It broke the guidelines due to its inflammatory style. On Hacker News we want to be able to discuss difficult topics involving arguments people may find counter to their assumptions, but you need to express things in a way that's persuasive rather than combative.
What I'm reading from that quote is that the issue wasn't renewables as such, but an issue of power generation reacting too quickly and too intensely to price fluctuations. "Renewables" only matter insofar as they're the sort of generation that, under the current regulatory regime, get to react to those pricing changes.
The report goes to great lengths to avoid certain words or phrases. The market failed here, it didn’t price in risk of grid collapse correctly.
That is a simple and great explanation.
It's also just unfounded conspiracism and factually incorrect.
> It's also just unfounded conspiracism
I don't see that.
> and factually incorrect.
Then substantiate your point.
Stability is one of the things grid operators pay for-- not just production.
Stability is one of the things grid operators pay for, sometimes, but whether they pay enough and validate that they're actually getting it is another thing. One core thing the report highlights is that many of the plants on the grid didn't do what they were required (and being paid) to do.
It's unfounded conspiracy to bang on about how "they are clearly hiding something".
Read the report. Power systems are complicated and not as easy as just "build another nuclear plant". It was (by regulation) conventional units that were required to provide voltage support, and it was in part their failure to operate according to their requirements that lead to the voltage excursions that caused the system collapse. The renewable systems that were capable of providing voltage stability were not allowed to do so.
> It's unfounded conspiracy to bang on about how "they are clearly hiding something".
It's also a bit of a straw-man to put your own version of what someone's saying in their mouth.
Buying insurance and other mitigations against rare circumstances, outside of well regulated and well understood products in stable marketplaces, is really hard. You need to do your homework with the counterparties and be sure that incentives align well enough to get what you want.
Unfortunately, when you set up an incentive system, you tend to get exactly what you pay for-- whether that's what you want, or not.
[flagged]
Except to point out that it's not nice that renewables arrive at random prices in the grid and start crapping all around it until they disappear when expensive again.
The stability of a nuclear plant vs the instability of a solar far when a cloud passes over.
That wasn't the core issue. It was the spark to the powder keg, so to speak.
See the sibling comment: https://news.ycombinator.com/item?id=44360082
[dead]
But the quote literally spells out it was market forces, not some instability in solar generation?
Your other comment probably got flagged because it started with a huge straw man and had multiple unwarranted jabs in it.
Temporary negative prices have been caused by the renewable generation which exceeded the grid demand at the time, which then evolved into the nasty feedback loop caused by the reaction of renewable generation to those conditions. You simply do not get such situation with traditional generation, it's the direct consequence of the intermittent nature of renewables and its high ratio in the total generation.
Also, have you read after the market part? Please watch this video https://www.youtube.com/watch?v=7G4ipM2qjfw if the last quote is gibberish to you. It discusses somewhat different issues, but the point still stands.
So an incredibly cheap source of supply exceeded the demand, and the market rules and some trips caused cascading failures.
Why is the problem the cheap source of supply rather than the market rules and incentives that made everything act the way it did?
Your comment suggests move back to good ol' expensive fossil generation instead of looking at how to bring the market rules up to date with evolving technologies.
>Why is the problem the cheap source of supply rather than the market rules and incentives that made everything act the way it did?
I explicitly mentioned this line of argument in the GP. The problem is that renewables only sometimes cheap and plentiful and often not when we want it. Even without accounting for the politically-driven preferential treatment covered in the sibling comment, from the purely technical point of view intermittency above certain threshold wreaks havoc in the traditional grid architecture designed for the traditional easily controlled "rotating" generation. It becomes really hard to manage the grid with existing tools when you have too much of intermittent highly distributed generation and in the extreme it leads to collapses like this.
As I wrote, yes, you could upgrade the grid, increase transmission redundancy, add battery/pumped/flywheel storage, introduce "smart" tools to manage the grid, and do a plethora of other things to accommodate renewables. Hell, you could even migrate the grid to DC!
But the cost of doing it is substantial. It's effectively a form of externalities of renewable generation, which are not accounted for in naive "cheap" $/kW metrics. Properly accounting for those externalities and adding them to the cost of renewable generation is possible, but politically unappealing.
>Your comment suggests move back to good ol' expensive fossil generation instead of looking at how to bring the market rules up to date with evolving technologies.
No, I believe we should remove the politically motivated shoehorning of renewables at the cost of grid stability. There should be a limit on how much intermittent generation we can have depending on the preparedness of the grid and we should pay less for power from such sources, not guarantee purchase from them!
As you say, we should have proper incentives structure which accounts for various externalities (including CO2 emissions!). We need to remove the existing subsudies on renewables which made sense in the early days, but not now. Let the generation sources play at the even field.
This makes sense to me to an extent, however, I do not entirely follow. You say:
> Properly accounting for those externalities and adding them to the cost of renewable generation is possible, but politically unappealing.
Implying this was/is not done and should be done. As a certified fan of looking out for (cost) dependencies, I agree with this to put it very mildly. I find it unlikely this wasn't done however, rather, I think renewables were likely onboarded harder than the externalities were taken care of to allow for it, possibly due to political pressure and/or mismanagement. Or at least, that rings all too familiar to me personally, not just from real world topics, but even from work. But then what you actually propose is:
> There should be a limit on how much intermittent generation we can have depending on the preparedness of the grid and we should pay less for power from such sources, not guarantee purchase from them!
Which is a different concern.
Also, this reads to me awfully like just flowery language for "hey, what if the obviously bad thing that happened wouldn't be allowed to happen anymore" with the logic retconned into it, but then I'll never have a way of proving or demonstrating that conclusively.
Finally,
> We need to remove the existing subsidies on renewables which made sense in the early days, but not now. Let the generation sources play at the even field.
This further doesn't follow from even your own explanation (i.e. "which made sense in the early days but not now" is not a substantiated claim). It's just your own political stance on the matter to the best I can tell.
> We need to remove the existing subsidies on renewables which made sense in the early days, but not now. Let the generation sources play at the even field.
This is also factually incorrect (unless Spain are now doing some country level subsidies on renewables). Fact is, new solar and new wind offer the lowes average power generation costs of any method. Regular market forces (without susidies) will favor renewables over anything else. Hydro being the most profitable.
Creating an level playing field is part of the complexity. The fact is we must migrate to mostly CO2-free power generation within the next decades and also replace fossil energy for transport and heating.
Market-based economies are great to follow technology trajectories and are efficient at capital allocation but even for them we need additional incentive structures to speed up the process.
I also think that most countries have massively reduced subsidies for new projects but existing subsidies will still be served for a long time.
We could also pay for renewables differently in order to disincentivize sudden drops or peaks, no?
Something like: the first 10Gw after start and the last 10GW before a stop make 50% of the revenue than the rest. That should disincentivize suddenly turning everything on or off depending on energy prices.
a couple of things to add to an excellent explanation.
I'm glad people are coming around to accepting that renewable energy has problems. We have some solutions to these problems but we do not have experience with them.
I agree entirely - the externalities of renewable energy are significant and are not paid for by the source of the problem - the renewable generators themselves.
Just as one example, what is the solution to an extended wind drought, say of a week or ten days? All the batteries in the world could not store enough energy for that.
A major challenge with renewable energy is that it is intermittent and variable but also unpredictable. it is impossible to predict wind speeds more than 24 or 36 hours out and even those predictions are often inaccurate. just building more wind turbines or solar panels won't cut it.
There is also the reluctance of grid operators to use the capacity available in renewable energy generators. The majority of wind turbines are capable of active and reactive power control but most grid operators either don't use this capacity or use it minimally.
A distribution connected wind turbine could do wonders for reactive power control but this is rarely done. More grid operators should pay for reactive power, like the UK is starting to do. This should also be sourced from EVs and small solar inverters.
> There is also the reluctance of grid operators to use the capacity available in renewable energy generators. The majority of wind turbines are capable of active and reactive power control but most grid operators either don't use this capacity or use it minimally.
I wonder how much is the near-complete inability for grid operators to communicate with smaller systems. My little solar inverter is capable of reactive power control over a respectable range of phase angles, and the grid operator has absolutely no ability to invoke this ability short of whatever formula the combination of PG&E, the various regulators, and the UL stuck into some standard for how small inverters are supposed to behave under various voltage and frequency conditions.
Never mind that inverters could also be fooled into thinking they’re islanded and therefore disconnect themselves if the grid frequency is too far out of range. This is usually designed to occur at above-nominal frequency, which is at least mostly not what happened in this event.
Because the only reason that solar is still generating power at negative prices is because they are getting subsidies to do it. Otherwise they would disconnect themselves apart from in extreme scenarios (control system down or something), and I bet when you are paying many tens of thousands of euro an hour you'd get someone to fix/manually turn it off sharpish.
Looks like there are a multitude of schemes of various vintages in Spain, which tl;dr basically give you a guaranteed price per MWh you generate. So imagine you get a 100eur/MWh subsidy for a (legacy) solar plant. The market price is €-20/MWh. You will still continue to produce power until the price reaches -100MW/h. Even worse are some contracts for difference (poorly thought through) which give you a guaranteed price regardless of what the market is at. So even if the price was -1,000eur/MWh the government or grid operator would still give you your €50/MWh (and the subsidy would be 1,050/MWh!).
The problem is if you reform this (and it is happening worldwide) solar is much, much less appealing. Because suddenly your solar plant which was getting (say) a guaranteed 70/MWh all year round suddenly does not make money for 6 months of the year at least at peak sun hours.
On top of all this, you have a lot of domestic solar in places like Spain. The grid operator _cannot_ control these assets in nearly all circumstances. They will continue to dump power into the grid regardless of the market price. This again will change but it requires an awful lot of work to retrofit invertors with remote control capability OR a lot of public backlash for charging end customers who bought solar in "good faith" now getting hit with peak time negative prices (so they change their behaviour).
I think my core message would be _any_ negative power prices is a sign of market failure. Acceptable in rare extreme occurrences, but the fact most of europe has highly negative prices very frequently is telling you the grid and market design is not able to handle what is going on.
> Because the only reason that solar is still generating power at negative prices is because they are getting subsidies to do it.
This massively simplifies reality.
E.g. in Finland where I live we also have issues with negative power prices. A few years ago we had some really low prices. It turns out, a fair bit of wind power producers never opted to add to their windmills any remote shutdown possibility, nor did they have the ability to monitor prices and react to them automatically. I.e. they just kept generating no matter the price, and had offers in at the network level at the lowest permissable price.
Since then, when they lost a non-insignifcant amount of money by running at negative prices, they've started installing control electronics in windmills and building IT systems and prediction algorithms to be able to react to this.
In the EU it is not as simple as "turning off when the prices are negative" since producers offer a certain capacity to the grid in an auction system the day before. You have to predict the weather + overall demand and set your offer accordingly.
Because the law mandates that renewables must be bought. Thats why prices fall negative.
> caused by the reaction of renewable generation to those conditions
No, that is not what the report says. It says, just like you say, that renewables reacted to market prices, causing a generation drop. It then says explicitly that synchronous generation caused oscillation, while PV plants showed a flat non-oscillating pattern.
From your comments I worry there are emotional factors clouding how you're reading the report - this was a systemic failure involving many separate technologies:
- Market signals - negative prices - caused a drop in PV generation (as frequently occurs)
- Synchronous plants caused oscillations as a side effect
- Plants procured to dampen exactly those oscillations did not deliver as requested
- TSO then took measures using interconnections to stabilize via other balance area
- This caused - presumed - overvoltages in distribution grids
- PV inverters then shut off, as mandatory by regulatory requirement in response to over voltage
You're absolutely right that PV played a large role here, but that point is diminished by making it out that PV is both the source of the initial generation drop and the source of the oscillations; it is neither.
The market design caused the generation drop, synchronous generators caused the oscillations, TSO action caused distribution overvoltages and regulatory requirements on PV firmware design in response to overvoltage caused the final blackout.
Where is the market for someone to get paid to pump water into a reservoir and let it fall down later for $$$?
That market exists, but the window of time here is like twenty minutes. Pumps have inertia and take time to spin up, you can't HFT load and generation.
I'm not fully informed about pumped storage, but twenty minutes is more than enough for most hydroelectric plants to go from 0 to full power, and that's probably the case for pumped storage as well. Eg,
In the case of Cruachan Power Station: “It takes just two minutes for a turbine to run up from rest to generate mode,” says Martin McGhie, Operations and Maintenance Manager at the power station. “It takes slightly longer for the turbines to run down from generate to rest, but whatever function the turbines are performing, they can reach it within a matter of minutes.”
https://www.drax.com/power-generation/in-energy-storage-timi...
If you read the report there was a significant amount of solar being produced at low prices and being pumped for storage. Further, the pumped hydro is the first load to be disconnected to balance demand on their system.
you are correct, but your analysis is not popular here. You will soon be presented with several reasons as to why renewable energy is not the problem and how batteries are the one true solution to these problems.
The reality is that electricity is complex and that renewable energy presents a new set of problems, problems to which we do not yet have complete solutions.
True, but the market moves fast because renewables (or, more precisely, wind & solar) move fast.
There is not much fast trading to be done on a nuke/gas/coal/hydro powerplant ramping up or down, but there is a lot of instability (and thus market volatility) to be found in fast varying solar/wind conditions.
That's inaccurate on the whole though, because while those big generators can't move fast, demand can move fast! Which is a difficult problem to manage in baseload grids.
Renewables just change one set of challenges for another set, at the end of the day it's all manageable.
It is a problem in baseload grid, but this is a global issue is shared with wind/solar – unless we find a way to sync demand peaks with wind/Sun peaks, that is solved by other means of energy buffering.
> because while those big generators can't move fast, demand can move fast! Which is a difficult problem to manage in baseload grids.
Don't forget rotational inertia. This gives the system a high-frequency response mode: it can resist sudden demand changes through stored kinetic energy, effectively acting as a low-pass filter with a fast dominant pole.
As you get a smaller share of generation with rotational inertia, you need a lot more buffering on short to medium timescales.
And, of course, it doesn't help for longer timescales that in many places renewable production slopes off in the late afternoon right when demand slopes upwards for cooling.
> in the late afternoon right when demand slopes upwards for cooling.
Demand rises because that's how people have their system set up. That cooling load can be shifted earlier in the day by using a slightly smarter thermostat to precook your house when the electricity is plentiful.
> Demand rises because that's how people have their system set up. That cooling load can be shifted earlier in the day by using a slightly smarter thermostat to precook your house when the electricity is plentiful.
You can do this a bit, but the insides of houses don't have that much thermal mass and the best insulated houses add a pretty large phase delay that makes the quickest rise in internal temperatures during the late afternoon as framing in the attic heats up.
I don't have a lot of luck in accomplishing meaningful precooling in my house. My best plan is to suffer until the late afternoon, turning on the AC at the end of the peak demand period when at least outside temperatures are lower, my AC units are shaded, and the cooling is more efficient.
Should’ve said ‘not enough spinning mass’ and it’d be perfectly fine for the politically correct and mean the same thing. This was highlighted as a risk for years and it finally materialized.
According to the operator report linked in another comment by leymed [1], the problem was not a lack of spinning mass (inertia) but voltage instability. From page 16 of the PDF:
The incident was NOT caused by a lack of system inertia. Rather, it was triggered by a voltage issue and the cascading disconnection of renewable generation plants, as previously indicated. Higher inertia would have only resulted in a slightly slower frequency decline. However, due to the massive generation loss caused by voltage instability, the system would still have been unrecoverable.
Obviously I’m as good of a grid operator as I was a stealth bomber expert on the weekend, but superficially that just doesn’t seem right. Maybe I’m underestimating how much spinning mass would be required, but that still qualifies as ‘not enough was present’.
You very much are underestimating it. Spinning mass helps even out very short term fluctuations in supply vs demand. Like on the timescale of tens of seconds, even when the whole grid is spinning mass. Even 10x the inertia in the grid would have maybe bought a few extra minutes, because the problem by the point the grid was collapsing was there were not enough plants online to provide the demand.
(Spinning mass on its own doesn't do much to deal with the voltage fluctuations. It's entirely something that's reactive to grid frequency, which is the most 'global' indicator of supply vs demand in a grid, since it can't fluctuate locally. But voltage and current can vary wildly in different parts of the grid, and required separate management)
please can you explain what doesn't seem right?
They admit they had plants offline when they should be available and then they say it wouldn't matter and the grid would've collapsed anyway..? Either they're in a very very bad spot and stop short of saying it outright in the report or it would've averted the disaster if the not-spinning mass was spinning as it should have.
People are having three different conversations at the same time:
– the concrete causes of this specific blackout; – how the existing grid is not prepared to deal with the current energy mix; – the energy policy of the past decades, from the nuclear moratorium in the 80s to the large subsidies for renewable generation of the past couple decades.
A person's strong opinion on any one of these issues will inevitably influence their opinion on the others.
It's worth pointing out that the worst part of the behaviour of renewables specifically in this incident (a fixed power factor for managing reactive power), is currently mandated by the regulations in Spain, even though many of them are already equipped to do voltage control.
You quoted
>the most plausible explanation is that it is due to market reasons (prices)
Seems to be market conditions or manipulations or inefficiencies in the market.