From The Lab

Today’s Lesson

Software’s Growing Energy Problem

According to a recent International Energy Agency (IEA) report, data centers accounted for 460 terawatt hours (TWh) of energy consumption in 2022. They forecast that, based largely on growth in AI and cryptocurrency, that number could balloon to nearly 1000 TWh by 2026. For perspective, in 2022 the United Kingdom consumed about 320 TWh and the United States consumed roughly 4000 TWh.

Electricity consumption is only one of the resources the computing industry is gobbling up. The CPUs and GPUs that power these data centers require rare Earth metals to produce, driving fierce geopolitical competition and significant mining efforts.

Data centers are increasingly pulling water from the deserts of Arizona. Using a precious and dwindling resource to keep data centers cool in one of the hottest, dryest places in the US — in the middle of a long drought — seems foolish to this writer, but the state and some of its municipalities have created significant incentives for building data centers in the state.

Computing’s two most recent flavors of the week — cryptocurrency and machine learning — get a lot of deserved blame for the growth in energy consumption, land use, etc. But the entire field has lost sight of efficiency, embraced bloat, and leaned heavily on hardware improvements and data center expansion to hide the ugly reality that software just keeps getting bigger and slower.

It’s no wonder we see essays like Rina Diane Caballar’s recent We Need To Decarbonize Software. The situation is so dire that OpenAI CEO Sam Altman has said a “breakthrough” in electrical production will be needed to adequately power AI models in the near future, and that it’s driving him to invest in nuclear fusion companies.

How Did We Get Here?

There are four key factors driving software’s astounding resource consumption.

1. The widespread adoption of digital technology.

2. Layering ever more “levels of indirection” and software dependencies into new software.

3. Incredible hardware that hides poor performance.

4. Types of software that naturally require lots of computational power.

Widespread adoption is a force multiplier on the other three issues — the more people use our software, the more its inefficiencies matter.

Layering and Dependencies

I’ll call myself out here: My website’s home page — which is not complex at all — requires 1.85 MB of data to be transferred over the wire to load. That’s embarrassing, frankly. But I can’t do much to fix it because I built the site using Squarespace. This gives me tons of productivity benefits compared to implementing the site from scratch, but it makes me reliant on their inefficient, bloated system.

By comparison, the most complex page of my Conway’s Game of Life simulator — which is much more complex and interactive than my website’s homepage — only transfers 119KB, 100KB of which is the favicon.

This kind of tradeoff happens a lot in software. Python code runs slower than equivalent C code in no small part because the Python interpreter is its own whole program that has to run in order to execute the Python code, whereas the C code will be compiled and then executed directly as machine code. But, most people agree that writing, debugging, and deploying Python code is a lot easier.

From machine code to assembly to compiled code to interpreted code … all the way up through things like Squarespace, Electron, and React Native. This layering generally saves time and effort, but it almost always slows down software and adds to the memory footprint of applications.

The same is true of modern software’s extraordinary dependency chains. Whether it’s oversized JavaScript bundles sent across the wire or bloated native apps shipping with far more code than they need. This bloat is a major security vulnerability, too.

Finally, the vast web of tracking software and data collection pipelines also slow our software down. Unfortunately the hunger for big data and the success of “surveillance capitalism“ are only driving wider adoption of such tracking.

Great Hardware → Poor Software

The computer hardware industry is absolutely incredible. Advances in chip manufacturing and computer architecture have resulted in absurdly fast computers that fit in the palm of your hand. Programs that would once have been agonizingly slow are now lightning fast. The result is a software industry that doesn’t need to prioritize efficiency.

When computers were much slower and memory was highly constrained, programmers had to go to extraordinary lengths to optimize programs to run on constrained systems. An SNES cartridge could hold about 4 MB of data, so your whole game — sprites, music, and all — had to fit in that space or you literally couldn’t ship it.

Nowadays, the opposite is true. Computers are so amazingly fast and have so much memory that developers distribute packages in the tens to hundreds of gigabytes without a second thought. Similarly, slow code goes unnoticed because CPUs are super fast.

Compute Intense Software

The advances in computer hardware also make certain types of computationally intense software feasible. Neural networks were invented in the 1950s, but they weren’t broadly useful until the 2000s. That’s because older computers just weren’t fast enough to train the models effectively. State of the art LLMs take months to train on modern supercomputing clusters.

Cryptocurrency, LLMs, and other compute-intense systems can only be useful because of modern hardware. Now that they’re useful and becoming popular, they generate mode demand for high-performance computer hardware, and we’re caught in a positive feedback loop.

What Can Be Done?

What gets measured gets managed — the first step for most programmers is to start measuring performance characteristics. Start monitoring the size of your software bundle and profiling your code. Once you know what’s making your code slow, you can start making it faster.

If you’ve never taken a course on data structures and algorithms, consider doing so. The fundamentals of algorithmic complexity, as well as common optimization strategies like Dynamic Programming and Divide and Conquer, show up across programming domains. Oz Nova’s CS Primer is an incredible place to start this journey.

Follow domain-specific trends. A fascinating example in the machine learning world right now is quantization. This is a way to take higher precision parameters — say 32-bit floats — and reduce the precision after training to something like 8-bit or 4-bit. This makes the inference computation even more parallelizable without sacrificing much in terms of model performance metrics.

Evaluate your dependencies, maybe you don’t need to rely on a 400kb 3rd party package just to use one function or class. Consider using a lower-level or compiled language. As much as I love Python code written in Rust or C is just faster.

Overall, the most important message is to care about efficiency for its own sake not just when it causes problems. As a little bonus, your server costs will come down, too. 😉 

Themes in the News

Are Phones and The Internet Bad For Kids?

Jonathan Haidt’s new book, The Anxious Generation, was published in late March. In the broadest possible terms, the premise of the book is that smartphones are bad for children and society should really do something about it. His promotional team deserves special kudos: For a few days it seemed like he was everywhere, appearing on podcasts, writing op-eds, etc.

In the book, Haidt highlights growing rates of depression and suicide among young people, declining rates of play and in-person socialization, increasing rates of attention disorders, and more, laying substantial blame directly on smartphones and the various internet services kids access using them. Here’s a representative snip from one of those promotional editorials in The Atlantic:

Haidt has long been a polarizing figure, and The Anxious Generation was not an exception. It resonated with some reviewers:

And … not as much with others:

“Protect the children” is — of course — a common political rallying cry. Some of the book’s detractors worry about what happens when that mantra is taken too far. For example, The Markup recently published a fascinating piece about just how much of the internet has been banned by public school districts. The list included nasa.gov, merriam-webster.com, and a bunch of other perfectly child-appropriate sites.

404 Media explored the substantial risks of the recent spate of age verification laws. Arguing, for example, that “people–including minors–will go to unmoderated, actively harmful alternatives that don’t require handing over a government-issued ID to see people have sex. Meanwhile, performers and companies that are trying to do the right thing will suffer.”

Meanwhile, there’s no doubt that some of what’s available to kids via their smartphones is truly horrible. Stories about kids using “nudify” apps to make realistic AI-generated pornography of their classmates keep popping up. And Instagram is even advertising those apps!

Similarly, the narrow case that allowing kids to access their phones in school is terrible for learning outcomes seems quite solid. Slow Boring did a

long, well-researched piece, and the Norweigan Institute for Public Health recently published overwhelmingly positive results from a review of schools that implemented cellphone bans.

The XZ Backdoor Attempt

An attempted “supply chain” attack was thwarted. The attack involved an open-source compression utility called XZ. The utility is a mainstay of the Linux operating system, which powers roughly 90% of all cloud computing. Had the malicious code not been found the backdoor would have allowed the attacker to execute any code they wanted on the infected machines via an interaction with XZ and SSH.

The attack appears to have been years in the making. Someone — or possibly a group — going by the name Jia Tan (JiaT75 on GitHub) started making pull requests to open-source packages in 2021. A pull request represents a set of changes to a codebase, and if a library’s “maintainer” approves the PR, those changes will be applied to the code. Here’s “Tan’s” first known pull request.

Over time, Tan kept working in the open-source community. Submitting PRs, participating in code review, and ultimately courting the maintainer of XZ to become a maintainer themselves. Once Tan became a maintainer, they had the authority to make changes to XZ without any approval. This February, they used that authority to insert the backdoor.

Andres Freund, a software engineer at Microsoft, “accidentally” discovered the vulnerability while investigating a performance issue and benchmarking a database technology called Postgres. Apparently, the issue was related to the malicious code injected into XZ. Freund wrote on Mastodon that investigating the issue and discovering the vulnerability, “really required a lot of coincidences.”

The episode highlights both the strengths and weaknesses of open-source software. The fact that so many companies (many unwittingly) rely on packages like XZ in absolutely critical means they are exposed to attacks like this, originating from software and software engineers they have not vetted. But, the fact that open-source is all done publicly also led to a speedy and transparent investigation.

Teb’s Tidbits

• The FTC is banning noncompetes nationwide.

• Congress actually banned TikTok, what now?

• LLMs are starting to become decent little hackers.

• How to make a horrible AI-driven fake-news click farm for $105.

• Meet the hacker who single-handedly took North Korea’s internet offline.

• New York City’s chatbot lied to New Yorkers about the law, oops.

• Gamers are selling their GPU’s idle time, and it’s probably being used to make nonconsensual deepfake porn.

Remember…