IT Infrastructure Built on a Toothpick: The Unseen Bottleneck

Author: Compiled by Hui Bian Xing Seng

Have you ever thought about how many layers of technology stack this article you're reading right now has gone through?

Mobile operating systems, browser engines, JavaScript engines, WebAssembly, CDNs, DNS, cloud servers, the Linux kernel, C runtime libraries, CPU instruction sets, semiconductor wafers... all ultimately come down to a single silicon chip consuming power.

Each layer sounds rock-solid. Each layer has a whitepaper, certifications, and backing from major companies.

But when you stack them together, you get a disturbing image—a wobbly tower reaching to the heavens, with the entire structure resting on a ridiculously thin red pillar.

Its name is the Von Neumann bottleneck.

And you, standing atop the tower, holding up your phone, feel that everything is perfectly normal.

John von Neumann, the father of modern computers. The von Neumann architecture is the foundational design of nearly all modern computers: the CPU handles computation, memory handles storage, and data is transferred between them via a single bus. This design has been in use since 1945, enduring the eras of mainframes, PCs, the internet, and mobile computing, and is now powering artificial intelligence.

But it has two inherent flaws: 1) CPUs have become increasingly faster, while memory and buses have failed to keep pace—data movement has become the system’s bottleneck, with the CPU spending most of its time waiting for data rather than performing calculations. 2) CPU performance heavily relies on cache capacity, but caches have inherent physical limitations and cannot be scaled horizontally like memory. Driven by modern high-level programming languages that prioritize human readability, programs have grown increasingly bloated and data structures more complex, surpassing cache capacity limits. This has further increased the frequency and volume of data movement between memory and the CPU, causing computers to run slower, become more sluggish, and crash more frequently. This is known as the "von Neumann bottleneck."

Ironically, this bottleneck is the most fundamental, core, and unavoidable issue in the entire IT industry, yet it is also the most neglected.

Why? Because solving it is too difficult and not easily profitable.

Big companies prefer to add more layers on top—faster GPUs, larger memory, more complex cache hierarchies, smarter scheduling algorithms. Each layer can be featured in a PowerPoint, each layer can be sold, and each layer avoids the red pillar rather than confronting and solving the core problem.

Like a building continuously adding floors while covering cracks in the foundation with increasingly thick wallpaper.

You ask the people living inside if they’re safe? They’ll say: Don’t worry, we just installed the latest smart lock and bought the most expensive disaster insurance.

Looking at this chart, you'll notice an interesting phenomenon: each layer, on its own, appears like a meticulously designed block.

Operating systems have a rigorous kernel architecture. High-bandwidth memory (HBM) employs precise stacking technology. CPUs and GPUs feature intricate designs with billions of transistors. Semiconductor fabs are the crown jewel of human industry.

The Linux and Apache foundations maintain the world's most reliable open-source infrastructure. LibcURL is invoked by nearly every program requiring network connectivity. The DNS system enables the global internet to find one another. Cloud providers promise 99.99% availability.

Each piece is perfectly square, with documentation, standards, and audits.

But when you stack them from bottom to top, the picture changes.

This is the truth about modern digital infrastructure: not a carefully designed whole, but a miracle built from countless makeshift solutions stacked together.

It works not because it was well-designed, but because enough people patched it enough times during enough late nights.

You think you're using a building. In reality, you're standing on a stack of patched-together blocks.

Look at the robotic arm extending from the side, labeled "AI Artificial Intelligence."

It is not supporting the tower but acting as a lever to accelerate the collapse of the block-like structure.

This is the most accurate satire of the current AI craze.

Everyone is saying AI is a productivity revolution, the next era, a silver bullet that will save everything. But from an infrastructure perspective, the first thing AI does is push an already unstable system even closer to the edge of danger.

Why? Because: 1) AI is a massive power hog. Training a large model consumes as much electricity as a small city uses in a year. Deploying inference requires vast numbers of GPUs, which still run on the von Neumann architecture—the red column is under more pressure than ever before. 2) AI users impose on data infrastructure a search load over ten thousand times greater than human users, and the loop-based solutions currently promoted will only further multiply the frequency of AI tasks involving data retrieval and computation geometrically.

More ironically, the output of AI.

Large volumes of AI-generated code are being submitted to open-source projects, with inconsistent quality and frequent vulnerabilities. AI-generated content floods the internet, overwhelming search engine results with spam. Countless AI-generated "solutions" are packaged as products and sold to customers, only to be revealed as illusions when problems arise.

The AI didn't fix the red column; it simply added another layer on top of the tower and told everyone: Look, we're taller now.

Whether it will collapse is something for next quarter to determine.

There’s a joke that says: It took humans 70 years to evolve computers from “calculating fast” to “able to make things up,” and we call that progress.

On the right side of the image, an angry bird is flying toward the tower. The label beside it reads: "What the big tech companies are doing."

This analogy is harsh, but accurate.

Large companies periodically release a "revolutionary" new product, framework, or standard. The launch events are grand, the presentations are stunning, and KOLs chorus their praise.

But these products often do not repair or strengthen existing industrial structures; instead, they undermine them.

When a new framework emerges, the ecosystem of the old framework dies. When a new standard is introduced, all devices compatible with the old standard become "outdated products." Big companies are well aware of the costs involved, but for them, the cost of disrupting the ecosystem is borne by developers and users, while the growth and valuation from new products belong to them.

On the left side of the chart, a Rust solo developer is flying away on a rocket, labeled: "Rust solo developers are doing their own thing."

This is another irony.

Independent developers are the most dynamic group in the IT ecosystem. Free from the constraints of big company KPIs, they can rapidly iterate and create high-quality applications that large corporations cannot produce.

However, the output of independent developers also has another side.

- Messy. The same feature has been implemented in ten different versions by ten independent developers, each with its own design philosophy, and each maintained only until the author lost interest.

- Unorganized. No standardized approach, no commitment to compatibility, no upgrade roadmap. Today it works; tomorrow, the author might delete the entire repository.

- Unprofessional. Many standalone projects lack testing, documentation, and security audits. The authors think "if it runs, it's fine," and then this project becomes a dependency for ten thousand other projects.

Individually, these issues may seem minor, but when combined, they become another form of market disruption. Users must sift through countless incomplete products, while developers stumble across numerous abandoned frameworks, causing the cost of trust across the entire ecosystem to rise steadily.

And those truly excellent independent developers—like the Rust developer in the image blasting off in a rocket—are focused on their own work, doing it brilliantly, but have nothing to do with this crumbling tower.

The tower won't become more stable because of them, and they won't stop because of the tower.

Everyone is busy with their own things, crashing their own things.

Let’s go through this entire chart from start to finish.

At the bottom is a large gray square labeled "Electricity (Computing Power, Water, Energy, etc.)"—the foundation of everything; without electricity, there is no computation. Yet in the IT industry’s narrative, electricity is the most overlooked layer. People discuss cloud-native architectures, microservices, and large models, but rarely ask: Where does the electricity for all this come from? How much does it cost? What is its environmental impact?

Above electricity lie four foundational pillars: the operating system, high-bandwidth memory (HBM), CPU/GPU, and semiconductor foundries. This is the hardware layer—the truly "solid and structured" part. Each represents the pinnacle of human industrial capability. But note: these are four separate stones, not a single continuous foundation—the connections between them inherently contain gaps.

Above it is a wobbly long plank labeled "Dynamic Arrays for C Language Developers." This plank spans four foundational stones but is inherently unstable. At its very center, a thin red pillar supports all the weight above it—the Von Neumann bottleneck. The entire weight of the IT world ultimately rests on this pillar.

Above that, the style goes off the rails. A "C99 project based on undefined behavior" is handling critical functions. Beneath the JVM, relational databases are crossed out. A robotic arm extends from the side, labeled "AI," pushing the tower.

Moving upward are the V8 engine, left-pad, network security, and WebAssembly—these are part of the web layer. Beside them, large text reads: "Web Developers Self-Sabotaging" and "Weird Things Happening in the Web."

At the very top, a tiny stick figure representing the user stands on the precarious peak of the tower, with the words "You are here" beside it. Zooming in, you see the figure holding up a phone—he is using this tower, which could collapse at any moment.

To the left of the tower, a Rust independent developer flies away on a rocket to pursue their own endeavors. To the right of the tower, an angry bird (a big tech company) crashes into the tower’s structure. Above the tower, a lone cloud drifts, labeled "Cloud Server Compliant with Specifications"—it has no connection to the tower, as if it were merely a concept.

The entire diagram contains not a single unnecessary word. Each annotation is a thorn, piercing the most sensitive areas of the IT industry.

It is not predicting anything. It is describing the present.

By this point, someone might say: "You're too pessimistic—doesn't the IT industry run just fine?"

Yes, it’s working.

But "functioning" and "stable" are two different things. Just because a person can walk doesn't mean their bones have no cracks. Just because a bridge can support vehicles doesn't mean it isn't overloaded.

The frightening aspect of the IT industry is that every layer is rapidly iterating, yet the fundamental problems at the bottom have never been truly solved. The von Neumann bottleneck persists, big companies still tear down buildings, AI still generates garbage, and independent developers still work in isolation.

Everyone is building upward, and no one is willing to look down.

Looking further down, you'll find the foundation is just a toothpick—and a toothpick will eventually fail.

Perhaps only on that day, standing at the top of the tower with our phones in hand, will we first ask: How exactly was the foundation of this building constructed?