August and September of 2021 are two months when Solana made envious strides. Its coin price soared from $35.47 on August 1 to $201.23 on September 9, an increase of 467% in just over one month. The price surge has boosted Solana’s market cap, getting the blockchain company into the top 10 digital currencies all over the world. At the moment of unparalleled success, Solana abruptly suffered a huge bug: at around 8 pm on September 14), its mainnet beta started to experience instability, with all applications on the blockchain unable to respond to requests. It was not until 2 pm on September 15 that the Solana team announced on Twitter that its network successfully upgraded into 1.6.25 and that the recovery of DApps, block explorers, and support systems would recover in a few hours. In all, Solana was down for over 18 hours in this incident. What happened during that period?

01 Solana's founding team

Solana has a very good founding team that encompasses technically skilled former engineers of Qualcomm, Intel, and Dropbox. The Solana project kicked off in 2017, and its first version was released in February 2018. The newly-established company started to take off, after the joining of Sam Bankman-Fried, a talented trader and the founder of FTX, one of the world’s major digital currency exchanges. In recent years, Solana has completed many rounds of financing. It raised $314 million in a recent private token sale round alone. Disclosed in June 2021, the round was led by Silicon Valley’s most prominent venture capital firms Andreessen Horowitz and Polychain Capital. To sum up, Solana is a well-connected, tech-savvy, well-funded public blockchain, which has built up a great reputation over just a few years.

Here is the main reason why Solana was created. After noticing Ethereum has been long plagued by transaction speed and scalability issues as an infrastructure platform, the founding team of Solana was determined to forge a high-performance, low-cost public blockchain that could carry large-scale applications, thus offering an ultimate solution to the impossible triangle of the blockchain concerning decentralization, security, and scalability. Solana is a decentralized blockchain built to enable scalable, user-friendly applications for the world, according to the definition released on its official website. In practice, it does make its name by virtue of ultra-high performance (50,000 transactions per second (TPS)) and ultra-low rates (single transaction costing only 0.000005 SOL), and ultra-fast confirmation speed (0.4 seconds). However, the bug-triggered network outage on September 14 has led many investors to re-examine the public blockchain and ponder what happened after all.

02 Solana experienced intermittent instability

Solana experienced intermittent instability at around 8 pm on September 14, which was confirmed by the Solana team.

Solana team told they experienced intermittent

At 8:38 pm on September 14, Solana tweeted: “Solana mainnet-beta is experiencing intermittent instability. This began approximately 45 minutes ago, and engineers are investigating the issue.” At 10:26 that night, the company tweeted again: “Resource exhaustion in the network is causing a denial of service, engineers are working towards a resolution.” In the early morning of September 15, Solana disclosed the network repair process in another tweet: “Sonala Mainnet Beta encountered a large increase in transaction load which peaked at 400,000 TPS. These transactions flooded the transaction processing queue, and the lack of prioritization of network-critical messaging caused the network to start forking. This forking led to excessive memory consumption, causing some nodes to go offline. Engineers across the ecosystem attempted to stabilize the network, but were unsuccessful.” So how did Solana manage to solve the problem? The solution was to call on the community and nodes to upgrade the Solana network version.

Solana tweeted Sonala Mainnet Beta upgrade

Since the updated version could solve the problem, why didn’t Solana do it at the very beginning? That is because in a huge, decentralized network of a public chain, it is not easy to require all nodes to upgrade to the new version together in a short time. The upgrade would fail without the support of 80% of network nodes. Or if the number of supportive nodes increased too slowly, which will also delay the bug fix for a long time, such an attempt could end up with huge risks. In fact, after calling on nodes to upgrade the network at 10:53 on September 15, Solana then tweeted at 2:01 in the afternoon that day:

“The Solana validator community successfully completed a restart of Mainnet Beta after an upgrade to 1.6.25. Dapps, block explorers, and supporting systems will recover over the next several hours, at which point full functionality should be restored.” The bug was unbelievably fixed within merely several hours. Solana didn’t take this measure until the eleventh hour, perhaps not because of worries about the community and nodes in completing the upgrade soon but the worry of exposing defects.

Someone has found that after Solana’s intermittent instability, its nodes were shut down simultaneously, and during those 10 hours, there was no transaction at all. Such unified collaboration of nodes was quite unusual among decentralized public chains. People can’t help but question the decentralization of this public chain with its ultra-high performance and ultra-fast transaction processing.

03 Users are firmly optimistic about Solana

Surprisingly, this incident didn’t cause too much volatility in Solana’s price. As of the writing (September 14), the token price is $148.81, a drop of less than 10% from $165.16 at 8 pm on September 14. It is more like a normal market adjustment to Solana’s surge for the past one or two months than a decline. Evidently, what underpins the token price is not just technology in most cases. There is no perfect public chain, and some chains even come with lots of bubbles. At least Solana still has much potential to catch up. This may explain why Solana always has a group of loyal users who are firmly optimistic about it and its price didn’t slump after the severe bug.

Does Solana’s 18-hour Network Outage Raise Doubts about Its Decentralization?