This week, the Solana mainnet briefly neared the loss of transaction finality. A routing failure by infrastructure provider Teraswitch caused approximately 28.83% of staked SOL to go offline. While the mainnet did not halt and blocks continued to be produced, this incident once again highlighted validators' reliance on a small number of infrastructure providers.
Close to the shutdown line but not triggered
According to Solana’s operational mechanism, if the offline staking ratio exceeds 33.34%, the network will cease finalizing transactions. Marinade Finance estimates that this incident has reached 86% of this threshold, leaving only about 20 million SOL away from triggering a full network halt.
Approximately 90 validators were affected, with the outage lasting about 33 minutes. The lost rewards for the affected validators amounted to approximately 333 SOL and are expected to be covered by validator staking at the end of this epoch.
Routing issues have affected multiple nodes across Europe and Asia.
The issue originated from Teraswitch's edge routers. Certain sites in Europe and Asia incorrectly treated a misrouted path as their local default route, giving it higher priority than the correct path, while the core infrastructure deemed this route invalid, ultimately causing 12 sites to lose valid forwarding paths.
Affected regions include London, Amsterdam, Dublin, Frankfurt, Singapore, and Tokyo; no similar outages occurred in North America. Teraswitch stated that the issue was identified approximately 10 minutes later, with network traffic restored around 04:16:15 UTC, though the Miami node remained temporarily disconnected from its backbone during troubleshooting.
Staking concentration amplifies the impact.
Marinade's analysis shows that AS20326 holds 118.89 million SOL, accounting for more than a quarter of the total staked supply. At the time of the incident, approximately 94% of these were simultaneously offline, becoming a key factor in the rapid escalation of this risk.
Additionally, approximately 14.1 million SOL are offline in environments related to Latitude.sh, Limestone, Butterfly Research, and Allnodes. Marinade stated that current data cannot confirm whether these outages stem from a common dependency or are merely coincidental in timing.
Marinade also found that 59 validators, collectively holding 80.2 million SOL, came back online within nearly the same recovery window in Amsterdam, Frankfurt, and Tokyo. These validators did not appear to trigger automatic failover but instead waited until routing reconverged before restoring service.
Of the 74 validators measurable by Marinade, only three successfully recovered through redundant arrangements: Laine, Cogent Crypto, and Lion3d. Marinade also acknowledged that its own delegation program suffers from centralization issues, with four ASNs accounting for two-thirds of its delegated staking, and AS395201 alone representing 36.94%.
The mainnet continues to operate, while the testnet was temporarily halted.
Jacob Creech, Vice President of Technology at the Solana Foundation, stated that among the 699 staked validators, 597 continued to participate in voting, accounting for approximately six-sevenths. The affected validators recovered within 40 minutes, and the validators delegated by the Foundation’s planned program were unaffected.
Anza added that during the same incident, Solana Devnet temporarily halted but automatically recovered after the routing issue was resolved. Since Devnet runs the same software as mainnet, some developers view this as an indication of improved network resilience compared to previous instances.
The last time the Solana mainnet completely went down was in February 2024, when it took nearly five hours to restart. Although this incident did not result in a full mainnet outage, it once again highlights that as long as infrastructure remains overly centralized, localized routing failures can still rapidly escalate into systemic network risks.
Additional information: Pyth was also affected. Since Teraswitch hosted two of its four NATS nodes, the loss of these nodes temporarily caused NATS to lose quorum, rendering the Router service, which depends on it, unavailable. Pyth stated that it will accelerate the adoption of new infrastructure providers and distribute critical services across independent failure domains.