Everyone Hates systemd
Exploring one of Linux's most heated holy wars
If there’s one thing that techies love to engage in, it’s holy wars.
Nothing matches the self-righteous feeling of spending half of your workday telling internet strangers that they’re idiots.
Whether it’s mocking the newest Vim user still stuck in their terminal; sharing the latest meme about malware to your Windows friends; or chiding MacOS as the poor man’s FreeBSD while simultaneously deriding the Apple fanboys for overpaying for it.
There’s a holy war for every technical niche.
However, there’s one holy war that you may not have heard about. It’s not new by any means, having started back in 2010. But in terms of contention, it ranks near the top.
It’s the war over systemd.
If you’re an avid Linux user, then you’re already well aware of systemd’s lore. In which case, feel free to skip this article and go straight to the comments with your spicy take. I know you want to.
But if you are a software engineer with only a surface-level understanding of Linux, like me, then you’re likely unfamiliar with systemd’s sordid past.
It wasn’t until a couple of years ago when having drinks with some sysadmins from work that I became aware of the drama.
“Man, screw systemd. It’s the worst thing to happen to Linux,” one of them spouted. His numerous empties had given him the courage to preach to the choir, as most of the others nodded in agreement.
I nodded along too, though I had no idea what he was talking about. When I got home that night, I did a quick Google search to give me the rundown. And oh boy, did it take me down a rabbit hole. Or a cesspool, depending on your perspective.
While a lot has already been written on the matter, I couldn’t resist the urge to toss in my two cents. This post will be recounting the history of Linux init systems, the rise of systemd, and why it has become so infamous in the various Linux communities.
The goal of this piece isn’t to rile up old feuds but to educate newer software engineers and other technical folks about a core component of the Linux ecosystem and the history behind it.
I will lay out the facts as unbiased as I can while still making it relatively brief and entertaining. I do have some thoughts on the matter, but I will save those for the end.
Note: While I did a lot of research into this topic, I’m not an expert. If you catch a glaring error or inaccuracy, please let me know and I will update it. I also didn’t spend a ton of time editing, so their’s likely typos and grammer errors.
Before systemd — a history of Linux init systems
To understand systemd’s present, we must understand its past and why it was created in the first place. In order to do that, we’ll need to learn about init systems.
When you turn on your computer, a lot of things happen. One of them is the creation of the first process, also known as PID1. It’s called PID1 because it’s the process with the id of 1.
PID1 is very important to an operating system. It is the parent process for the entire system. Generally, when you work on your computer, you use a lot of processes.
You can see them all when you run ps aux in a terminal.
All of those processes that aren’t PID are directly or indirectly a descendent of it.
In the past, PID1 has had a simple yet crucial role. It was a simple daemon that ran in the background and executed a series of startup scripts that booted the rest of the services in the system.
While it was simple, it also needed to be stable. Since PID1 is the parent of all other processes, if it either crashes or exits, it will take down the rest of the system with it. For much of Linux’s history, the various distributions used an init system based on Unix’s System V, known as sysvinit.
sysvinit was well suited for computers in the 1980s to early 2000s as it was meant to operate in a static environment. It issued the bootup scripts synchronously in a pre-defined order. It didn’t execute the next script until the current one had finished. It was inefficient and slow, but that was fine for the time.
However, as time went on, hardware became more dynamic and the Linux kernel more asynchronous. Devices would boot at unpredictable times, leading to distribution developers having to create hacks and workarounds. Also, users wanted to be able to plug external drives after the machine had already booted. sysvinit wasn’t designed to handle this.
Additionally, sysvinit wasn’t a fully-functioning service manager. It wasn’t fully capable of tracking processes, leading to orphaned processes. It also didn’t handle dependencies requirements well.
It was apparent to the maintainers of the various Linux distributions that sysvinit was outdated and needed to be replaced.
The rise of systemd
Due to sysvinit’s shortcomings, people began to look for a spiritual successor. Several projects were created to take the mantle. One of the most notable of these projects was Upstart.
Upstart was a big improvement over sysvinit. It used an event model that was able to start and stop processes based on events in the system. This allowed it to take advantage of the asynchronous nature of modern hardware without hacky workarounds.
At its peak, Upstart was incorporated into a variety of Linux distributions, such as Ubuntu, Fedora, RHEL, and openSUSE. But it had its shortcomings. Upstart didn’t utilize parallelism, making the startup process slower than it needed to be. It also had some notable bugs (review the article).
There was still room in the market for competition. Enter systemd.
Started by Lennart Poettering and Kay Sievers in 2010, systemd was created to improve upon the init process — much like Upstart. However, unlike Upstart, systemd goes about it in quite a different way.
In Lennart Poettering’s post Rethinking PID1, he states:
For a fast and efficient boot-up two things are crucial:
- To start less.
- And to start more in parallel.
How systemd works is a matter of confusion. It’s sometimes referred to a dependency or event-based model for system booting. And while the distributions that use systemd may use it in such a fashion, systemd was designed around socket activation.
While a deep dive into socket activation is outside the scope of this piece, I can give a brief synopsis.
When a computer boots, it spawns off various services and processes. These processes are interconnected and talk to one another via a socket, which is essentially a buffered channel for communication. If a service tries to connect or communicate with another service that isn’t active yet, it will fail. However, what systemd does is activates all service sockets at the same time.
Therefore, if a service attempts to communicate with a service that isn’t active, the message will get buffered in the socket until the recipient is ready. This allows systemd to boot all the various services and processes in parallel, leading to faster boot times.
As stated in Rethinking PID 1:
Starting more in parallel means that if we have to run something, we should not serialize its start-up (as sysvinit does), but run it all at the same time, so that the available CPU and disk IO bandwidth is maxed out, and hence the overall start-up time minimized.
All of this makes systemd an efficient init process. However, systemd didn’t stop there.
Poettering believed that PID1 had a special role with unique privileges that were being underutilized. Along with efficiently booting up the system, he envisioned systemd as a foundation and platform for building operating systems.
Influenced by Poettering’s vision, systemd’s scope has increased over the years. It has expanded from an init system to an entire software suite with 69 separate binaries. From its own website:
systemd is a suite of basic building blocks for a Linux system. It provides a system and service manager that runs as PID 1 and starts the rest of the system.
[systemd includes] a logging daemon, utilities to control basic system configuration like the hostname, date, locale, maintain a list of logged-in users and running containers and virtual machines, system accounts, runtime directories and settings, and daemons to manage simple network configuration, network time synchronization, log forwarding, and name resolution.
While systemd’s feature set has increased since 2010, so has its popularity and influence.
Due to Poettering’s relation to Red Hat, Fedora adopted systemd in 2011 shortly after its creation. Arch and openSUSE quickly followed in 2012. Debian held out for a while, but eventually, it and Ubuntu followed suit in 2015.
As of this writing, 13 Linux distributions run systemd by default. While I couldn’t find the exact percentage, it’s safe to say that the vast majority of Linux users use systemd.
systemd is unarguably influential, but that popularity hasn’t come without a price.
The drama surrounding systemd
It’s obviously hyperbolic to say that everyone hates systemd. The majority of Linux users either like or don’t care about it.
However, its adoption was rife with drama and turmoil. A vocal minority has grown that curse its name. They’ve created multiple websites pointing out its various flaws and systemd-free Linux forks. Some even personally insult its creators. It’s even gotten to the point where death threats have gotten involved.
This is more than a simple disagreement over software. And it’s worth asking why and how it has come to this.
Through my research, I’ve found many criticisms of systemd, but they all seem to be split into three different categories.
The Creators
The creators and community around systemd are a common source of ire. Linus Torvalds, the creator of Linux, famously called Kay Sievers a “f*cking primadonna” during a heated exchange on the Linux mailing list.
While Kay Sievers is one of systemd’s co-creators, Lennart Poettering is the face of it. And as such, he receives the brunt of the backlash. One of the main reasons Poettering has faced the wrath of many in the FOSS crowd is his public persona.
As the lead developer of systemd, Lennart Poettering is a public figure. He has his own Wikipedia page, a modest Twitter following, and a blog where he shares his thoughts.
I’ve read a number of his posts. They are very well constructed and thought out. If your only exposure to him was from those pieces, you might wonder where all the hate comes from.
However, Lennart is also very vocal when it comes to GitHub issues, Hacker News comment threads, and social media. And it’s on those venues where he tends to be more unfiltered.
Lennart’s commonly criticized for his attitude towards bugs and his unwillingness to accept blame. So much so that he received a Pwnie award in 2017 for Lamest Vendor Response.
But there’s one piece of discourse that tops even that. When you search Poettering’s name on Google, one of the first things that pop up is a post he uploaded to Google+ back in 2014. In it, he discussed how the overall Linux community was a very toxic environment. So much so that he constantly receives hate messages and even death threats.
His advice for newcomers was to “grow a thick skin, or run away”.
This caused quite the stir in the various Linux and open-source communities. While many agreed with his sentiment, others accused Lennart of behaving in similar ways as Torvalds.
It’s debatable whether the antics and personal life of a creator should impact the public opinion on their works. After all, while Torvalds has had his run-ins with the systemd developers in the past, he still likes the software.
In an interview with Slashdot, Torvalds stated:
I have to say, I don’t really get the hatred of systemd. I think it improves a lot on the state of init[.]
Yeah, I’ve had some personality issues with some of the maintainers, but that’s about how you handle bug reports and accept blame (or not) for when things go wrong. If people thought that meant that I dislike systemd, I will have to disappoint you guys.
However, for systemd’s detractors, Poettering’s controversial statements have made an already polarizing project all the more divisive.
Complexity and Design
systemd is often referred to as bloatware by its cynics. It’s easy to see why. Its code base clocks in at over 1 million lines of code. For most other projects, that wouldn’t be a problem. But systemd is a unique case.
As I said earlier, PID1 has an important yet precarious role. Since it is the parent of all other processes, if it crashes, the entire system will crash.
The way that sysvinit and Upstart solved this issue in the past was to be simple in scope. Their simplicity helped increase their stability. And herein lies the problem.
systemd is the furthest thing from simple. It’s an entire software suite with 69 binaries. Its scope has evolved from being just an init system to taking over many other parts of the operating system. Parts that people argue didn’t need fixing.
And this was by design. Poettering has expressed on multiple occasions his desire for a more unified and integrated Linux ecosystem. From a user experience perspective, this makes sense. Speaking from personal experience, being able to log in to any systemd-integrated operating system and understanding how to start and stop services is a great feature. But it comes with tradeoffs.
The combination of systemd’s complexity and its mass adoption in the Linux ecosystem have proved to be a gruesome twosome in the past. Any change that it makes, especially if it’s not backward compatible, impacts nearly everyone running a Linux machine.
Some notable examples are:
Introducing a change that kills background processes when a user logs out by default.
GNOME making
systemda hard-dependency and preventing any non-systemdsystems from using it. (It’s possible to run GNOME withoutsystemdby requires lots of workarounds.)
However, many in the Linux community see systemd’s complexity as a necessity. According to an ArchLinux developer:
What most systemd critics consider “bloat”, I consider necessary complexity to solve a complex problem generically.
Change From Convention
The last, and perhaps largest, the reason I see for complaints about systemd is that it is a significant change from its predecessors. Both in implementation and philosophy.
Some of the loudest complaints came from sysadmins. When you’ve spent years and possibly decades developing scripts, tools, and knowledge for the older system, it’s a shock when all of those things get deprecated and outdated in a single release.
Sysadmins spend a lot of time messing with and configuring operating systems. They do it as a job and for their livelihood. When you mess with their livelihood, they tend to form strong opinions about it.
Also, they hate binary logs.
The other change that had many up in arms was the break from the UNIX philosophy.
The UNIX philosophy states that everything tool should do one thing and do it very well. While Poettering is quick to point out that systemd is a collection of binaries, others are quick to point out that many of the binaries depend on one another, not making it as decoupled and modular as advertised.
Additionally, systemd doesn’t abide by the POSIX standard — no doubt influenced by Lennart’s personal opinions of POSIX. This ties systemd to the Linux ecosystem and alienates its BSD brethren.
For many, this is the straw that breaks the camel’s back. While they can forgive the other issues I’ve listed, breaking from the UNIX and POSIX standards is a step too far. They see it as the holy grail and no amount of benefits or improvements that it brings to the system.
Don’t mess with UNIX.
My Personal Thoughts
After reading through numerous blog posts, mailing lists, and forum threads, I have developed some of my own thoughts on the matter.
As a software engineer who uses only uses Linux’s surface features on a daily basis systemd is perfect. It was made for my use case.
On a recent project, my team had to create our own images with custom agents and daemons baked in. Writing the service files for those agents to ensure that they ran on startup and restarted on failure was a breeze. We haven’t had an issue with them in production (at least not yet.)
With that said, I’m sympathetic to those that find a major issue with systemd. The group that seems to have the biggest problems are power users with complicated use cases, such as sysadmins of large systems. Because for them it isn’t just a hobby. It’s their livelihood.
When you’ve spent years, and possibly decades, developing scripts, tools, and knowledge for a system, it’s a shock when all of those things get deprecated in a single release. It takes time and effort to rewrite and port those tools in order to upgrade to a new version.
There’s a reason so many systems still run version 2 of Python despite version 3 being available for 13 years.
One of the reasons why systemd has remained a controversial topic for the past 10 years is because its adoption wasn’t a singularity but a series of unfortunate events.
Due to the nature of FOSS, each Linux distribution adopted it separately at different times. And each adoption sparked the same debate over and over again. Hardening each person’s opinion and resolve.
From an engineering and architectural perspective, systemd has made a lot of smart decisions. At the same time, consolidating large swaths of the Linux ecosystem comes with its own shortcomings.
As with everything in software, there are no perfect solutions — just tradeoffs.
In conclusion
To me, the history of systemd represents the best and worst of the tech industry. On the positive side, there are a number of well-thought-out presentations, articles, and discussions that have taught me a lot about Linux and init systems. More than I would’ve learned in my day job.
But on the negative side is the online harassment and death threats toward the systemd creators — joke or not.
There’s a lot that tech professionals can learn from the drama surrounding systemd.
systemd’s biggest failing, in my opinion, was its communication. I’m sure the systemd creators had the best of intentions. But if a significant number of people are calling you primadonnas, including the creator of Linux, there’s room for growth.
On the other hand, systemd knew its audience. It provided a free and maintained service that was better and more modern than its predecessors. This made it a no-brainer for many Linux distributions to make the switch.
For one piece of software to take over 99% of the Linux ecosystem in less than a decade is rather remarkable in my opinion. Whatever your opinion of Poettering, I aspire to have a project even just 1/10 as successful.