What broke Ethereum’s Fusaka upgrade? Prysm post-mortem reveals the cause

0
7

Prysm developers have released a post-mortem explaining what went wrong during the Fusaka upgrade on December 4 and how it briefly put the Ethereum network under pressure.

Right after Fusaka went live, a bug in Prysm caused serious problems for validators using that client. Validator participation dropped sharply to about 75%, and the network ended up missing 41 epochs. Because of that, around 382 ETH in proof rewards were lost.

The issue started almost immediately after Fusaka activated at epoch 411,392 on December 4. Prysm nodes began struggling because they were forced to repeatedly recompute old network states. This process used too much memory and computing power, which slowed nodes down or caused them to fail.

Prysm developers explained that these old historical states are very heavy to process. When many of them are replayed at the same time, nodes can get overwhelmed, creating something close to a denial-of-service situation.

Prysm validators make up roughly 15% to 23% of the network. When so many of them dropped offline or became unstable, overall participation fell from normal levels above 95% to just 75%. That pushed Ethereum dangerously close to losing finality.

If the same bug had hit a larger client, like Lighthouse, Ethereum could have lost finality completely. That would have caused major issues, including freezing Layer 2 rollups and stopping validator withdrawals until the problem was fixed.

The Fusaka upgrade itself wasn’t the issue. It rolled out smoothly with no downtime and introduced PeerDAS, a feature meant to greatly increase data capacity for Layer 2 scaling. The trouble only appeared afterward because of the Prysm bug.

What saved Ethereum was client diversity. While Prysm struggled, ten other consensus clients—such as Lighthouse, Nimbus, and Teku—kept running normally. That meant about 75% to 85% of validators stayed online, allowing the network to keep producing blocks and processing transactions.

The Ethereum Foundation quickly shared emergency instructions for Prysm users. Developers rolled out temporary fixes right away and later released permanent updates in versions 7.0.1 and 7.1.0.

By December 5, validator participation had bounced back to nearly 99%. Within 24 hours, the network was fully stable again, showing how important multiple clients and fast coordination are to keeping Ethereum secure.