Smart contract blockchain Solana nearly froze Wednesday, Marinade Finance says

Solana nearly froze on Wednesday morning when a routing glitch at a major data center provider knocked almost 29% of the network’s staked tokens offline, staking platform Marinade said.

Finality, the point at which blockchain transactions become irreversible, stops if more than a third of the coins staked go dark. Staking refers to act of locking coins in a blockchain network to secure it in return for rewards.

Marinade said that the network came within about 20 million tokens of that threshold. Roughly 90 validators were hit and together lost 333 SOL in rewards, a relatively small sum that will be covered by “validator bonds.”

“If delinquency had gone past a third, nothing finalizes for anyone holding SOL anywhere, and there’s no bond for that. The February 2024 halt took about five hours to restart,” Marinade said in an explainer post.

Solana is one of the leading smart contract blockchains, with assets worth $4.3 billion locked in DeFi protocols operating on the network. The blockchain has built a reputation as a faster and cheaper alternative to industry leader Ethereum, but has faced several outages in the past.

The latest issue started with a bad internet route from Teraswitch’s Miami facility that then spread to data centers across Europe and Asia, cutting off validators in London, Amsterdam, Frankfurt, Singapore and Tokyo. North America stayed online. The company fixed the issue in about 10 minutes, and traffic was flowing again by 4:16 a.m. UTC.

One single network operator, identified as AS2032, controlled more than a quarter of all the tokens people had locked up to secure the network, which was more than the Solana-prescribed safety limit. Almost all of those tokens went offline at the same time. Other companies lost another 14 million tokens in the same short period. Most of the affected validators, including the big one called Helius, stayed offline for the full 33 minutes because their backup systems never switched on.

This whole event is a clear warning: if more than one-third of the network’s tokens ever go offline at once, the entire blockchain freezes for every single person holding SOL, and there is no quick way to fix the bigger damage that would follow.