> In addition to /etc/systemd/system, the drop-in ".d/" directories for system services can be placed in /usr/lib/systemd/system or /run/systemd/system directories. Drop-in files in /etc/ take precedence over those in /run/ which in turn take precedence over those in /usr/lib/. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.
> ...
And this is just an excerpt which doesn't cover the details. Read the manpage for all the hilarity.
>Over-elaborated hierarchies where configuration files may reside, and their precedence schemes.
>Read the manpage for all the hilarity.
If you find it hilarious it's because you don't understand the reason. /usr belongs to the distribution and /etc belongs to the user. The distribution doesn't touch /etc and the user doesn't touch /usr.
Upstream ships a unit with its own opinion of the defaults. That unit goes in /usr.
Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file.
User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.
Some units are created at runtime on demand and thus ephemeral, like mount units generated by generators. These go in /run. These are functionally "system-provided units" because users must be able to override them, so they're below /etc in the hierarchy.
The "defaults" and "overrides" we're talking about are both at the level of the individual setting as well as at the level of the whole file. If a user wants one of the settings to be different, they can add a new 99-foo.conf file that unsets / changes it. If they want a whole file to be different, they can add a new file with the same name that is empty / has different values for its settings.
BTW `systemctl cat whatever.unit` will print all the files that systemd considered in the order that it considered them.
>And this is just an excerpt which doesn't cover the details.
The only detail your excerpt doesn't include is the consideration of `service.d/`, `socket.d/` etc directories which provide overrides for all units of that type. It's just one point, not commonly used, and not that big of a deal either.
> If you find it hilarious it's because you don't understand the reason.
I think it's hilarious that you think users are supposed to understand "the reason" at this level of detail.
When we make a thing, we make choices about who it's for. Sometimes those choices are explicit. But this is a good example of an implicit choice. Clearly it grew into this shape for reasons, but those reasons are about the people doing the making, not the people using it.
Even if we grant that all of this hierarchy is necessary, it still imposes a large burden on the person trying to figure out "why is X doing Y?" And that might be fine if the makers said, "Well this is going to be hard, so what can we do to make it easy?"
But denying that users will reasonably find this hilarious (or frustrating or impossible) is just digging the hole deeper. Telling people that their understandable reactions are wrong doesn't change their reactions, it just gets them to stop mentioning the problems around you.
> it still imposes a large burden on the person trying to figure out "why is X doing Y?"
Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.
Personally I have found the systemd hierarchy to be very useful. It allows me to create overrides for certain properties in a Unit without having to worry about keeping the rest in sync with upstream, or my package manager overwriting my configuration. And it allows me to have different independent packages (or chef recipes or ansible playbooks) that create different override files for the same unit without stepping on each others toes.
Yes it is a little complicated, but for most end users, you don't really need to worry about most of the complexity, you just need to know where you put your overrides. And for distributors and power users, that complexity serves a useful purpose.
> Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.
Sure, it could be that systemd does have amazing tooling for helping users with the burdens they have created. Or, given the very mixed feelings about systemd, maybe they've tried to be user focused but not hit the mark. I don't know enough to say.
But my point is that either way responses of the form "it's so easy, you just have to remember [9 paragraphs of detail users don't care about]" are part of the problem, not the solution.
Systemd didn't create this problem. But it solves the issue of distributions shipping config files as part of packages and then on every package upgrade having to reconcile between the distribution's config and your modifications. Now with the drop in system you don't need to do that.
I am not an expert but I think in general systemd has a lot of complexity but it's to handle existing issues in a better way. Some of the older init systems might be simpler to describe or get started but lead to more confusing situations in the long run.
"Make it as easy as possible, but not simpler". I think this applies perfectly here. Systemd has very complicated requirements due to its very unique position in a Linux distribution. And as a (power) end-user of it, you basically need you basically needs to touch only /etc
> Systemd has very complicated requirements due to its very unique position in a Linux distribution.
But are all those requirements actually necessary for the job it's doing? Or have its developers imposed artificial constraints through their choices in designing systemd that make it more complex than it could otherwise be?
The combination of distributions existing and believing they “own” parts of the system, and users needing to override settings means it is as simple as possible, and no simpler.
A distribution 'owning' a directory means that the package manager is allowed to change files in there, and allowed to assume nothing major was changed from what it installed.
A distribution that doesn't own any directories like this effectively can't make any changes. It's like writing a class without private fields.
That's because some features are targeted at different audiences. End users want simplicity and ease if use. Admin also want those, but they are secondary to determinism, consistency, and ease of tracking changes.
I find the best way to think about systemd is as the result of some people that probably knew the most about init systems remaking init to be consistent, documented, and handle a bunch of new features that different newer init systems had used to good effect.
Then, because regardless of how much those people knew the vast mofority of information about needs was sequestered in 1000 different teams and projects, they had to add a bunch of new features. Repeat that 100 times, and you get what we have today, which is a system that mostly works (and works well if you actually adopt their preferred patterns), but one that also has the craziest and hardest to remember directive names I have ever seen, where slight nuance about how they function means entirely different params.
In a perfect world it would be rewritten with the same features but with all the knowledge of what features are actually needed known and consistently implemented with sane naming from the beginning, but nobody wants to go through that again.
That last bit is especially what interests me about systemd. From the way you describe it, it sounds like they made it mainly to please themselves, and were then forced to accommodate the actual users and actual users. But that they did so grudgingly and partially, again favoring their interests.
Assuming that's how the history went, fine. Everything has a history, and we have to deal with the now.
But my point is that we are not going to deal well with the now, let alone the future, unless we recognize the impact the history has on people. So if people defensively shout "that's not funny" and imply users are wrong for being bothered by the mess, then we're on track to build another mess. We can't do better until we are honest about why it didn't turn out as well as we would have liked.
> it sounds like they made it mainly to please themselves, and were then forced to accommodate the actual users and actual users
I think it's less that than that there were real advances they were after, which they delivered (Poetterings original essays on what it can and should do and where the ideas came from, such as OS X's LaunchD and some things from Windows, etc) referred to numerous things that every sysadmin I knew at the time that read it was excited if could be delivered (but none of use really appreciated what it meant to deliver them).
The issue is not that they initially built it to their own desired, but that the way SysV init worked with just launching bash scripts to start and control the process that eventually should be running meant that a lot of crazy behavior was put into scripts along the years, because when you can write a program to control your program, sometimes it's easier to just put special behavior into your launch script than to patch the real code in question in some way.
When this crashed against systemd's desire to know what's actually running and control the environment perfectly (which is beneficial, it was always a pain to have a process that started fine from your root user shell but not on startup), and strongly disincentivize programs between systemd and the process in question (which a bash script is) for reasons of process and resource tracking (Linux kernel cgroups were just starting to widely be used beneficially around this time), there started to be a mass influx of feature requests for behavior that used to be handled by dedicated bash scripts previously, but now systemd needed to account for.
Combine that with systemd adding real dependency tracking (as well as start order tracking, which is related but not quite the same, leading to the difference in Wants and Requires), and what we have is a system that's just trying to do something vastly more complicated than what SysV init did previously (and that's before we get to the scope creep which caused them to want to take over some other early boot processes to they could be deterministic or whatever, which is also a PITA for many people).
If you're actually interested in the history if this, I recommend reading "systemd, 10 years later: a historical and technical retrospective" at https://blog.darknedgy.net/technology/2020/05/02/0/. That's where most my info on the details come from. I was around in the beginning and was excited to see systemd start, but I didn't actually use it until far later since most the systems I administer don't get changes like that immediately (and it was in an extremely slow release cycle at that time).
If you read the above you may come away with a different perspective than me, but I will say that systemd is very consistent in its operation and very robust in how it tracks processes and whether there's been a problem and restarts are needed, etc, as long as you actually adopt their preferences (which is to say run everything foreground and let systemd deal with STDIN/STDOUT and environment, and have a good pid to start with). If you do that, writing a new service file is extremely easy. Far more easy than it ever was with SysV init. It's a different paradigm, and sometimes it can be a pain when some process doesn't want to play nice, but mostly it just works once you stop fighting it, which is nice, and cuts down on admin time figuring out problems (especially since it just grabs all STDOUT and STDERR for you so random error messages that don't go to logs aren't lost either).
Thanks for the details. I will check out the link.
I should say that "real advances they were after" is not at all incompatible with "made it mainly to please themselves". The things that pleased them may well have been "real advances"; they often are. The question is the extent to which they were prioritizing the needs, capabilities, and interests of the whole audience being served, not just the things they personally thought important or preferable.
Users are only supposed to know about `/etc`. The rest of the directories are useful for distribution maintainers, people who package software for a distribution (but may not be maintainers), and some developers.
> Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file.
> User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.
Larry Wall elegantly solved these problems 38 years ago with the "patch" utility. Debian is a good example of how its use has been standardized to solve these problems.
The patch approach to managing distribution-provided configuration files is superior to the systemd unit approach for at least three reasons:
- Non-experts can read the configuration files from top to bottom in linear order, so the documentation can be inlined by the distribution provider, which means they can add maintainers notes as necessary.
- It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.
- The implementation of all this is factored out and standardized across 1000's of packages, so it only needs to be implemented once and understood once.
Rebasing patches is pretty hard I'd say. Much harder than just explaining someone the logic behind /lib, /etc and /run. I mean, this directories are seperate for a reason. Knowing the idea behind the directory split (a.k.a. Linux Filesystem Hierarchy Standard) is more essential than knowing how to use the patch command line tool. There are more and more people who don't want to use the command line. And that's completely fine.
> - It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.
Anecdata, but still: Every single time the package manager asked me what to do, it was not because of a breaking change, but simply because the base file changed in a way that the patch didn't apply cleanly anymore. That is, I, the user, had to work around limitations of the patch format. If the distro configuration and the user configuration had been separate files, things would have just worked fine.
You don't have to be an expert to read multiple files. If you do need expert assistance with something like that please contact me and we can work something out.
So it's not hilarious statements like this make it hard to understand the reason? I almost cannot tell whether your whole post is sarcasm, or just showcasing irony.
You mean that the top-level Linux directory are hilariously badly named?
If so, I don't think anybody will disagree.
But if you meant that's a large problem, then well, no it isn't. There are 7 of those badly named ones. It's a matter of reading half a page of text to see what they do, and after 2 or 3 times you read it as reference, you have them memorized.
If would be nice to solve the issue, but it isn't anybody's priority.
Yes, "The user (syadmin) doesn't touch /usr, except /usr/local", where compiled in applications reside with full path so /usr/local/{bin,sbin,share,...}.
If you're deploying an especially big application, you may also prefer /opt/$APPNAME/{bin,etc,usr,...} (A full filesystem tree, basically).
We manage a fleet of systems at work, and I don't remember editing or touching a file under /usr, unless the period I was developing .deb or .rpm packages to deploy on said fleet.
There are many written and unwritten rules in Linux world, and sometimes people disregard or ignore these rules so blatantly, so I can't understand whether the post I'm replying to is a sarcasm or irony.
I think there is a gap here between those who routinely (exclusively?) used UNIX machines along with 50, 100, 1000 other people all logged in at once (generally pre-Linux era) and those who only ever got to see and enjoy a time when everyone could have a UNIX workstation on their desktop (Linux, Mac, or other x86 flavour, take your pick).
Yeah, I really feel like we're in a "horseless carriage" or "radio with pictures" era for computing. We have incredibly powerful machines available. But the dominant paradigm for managing that power is to slice it up into things that simulate much less powerful processors in software environments designed for a shared 1970s university department minicomputer.
In many ways I love it, of course. It's what I grew up on, and I'm sure I'll have a small Linux cluster in the retirement home, because that's how computers are "supposed to be". But I think the old paradigm has only lasted so long because we have so much computing power to burn.
I'm not sure I understand. When the "old paradigm" was new, computing power was much more limited. They couldn't afford to "burn" any computing power. But it's only stuck around, because now we have more computing power? That doesn't make sense
You might read some Kuhn. Paradigms change when the old one becomes unsupportable. But the vast increase in computing power per dollar means we can waste a lot of it without it being economically uncomfortable.
He works for a financial trading firm, where the competitive nature of it means that every processor cycle counts. For him, older paradigms are perplexing and irritating, because they are major barriers to getting the most out of the hardware.
Back in the shared era, the machines were better-suited to individual customization, and were more easily understood by system administrators. Now, they're optimized too... waste more computational resources when idle?
If you want to use /usr for yourself, be prepared to throw the FHS and compability with almost everything away and rebuild your whole userland with something like NixOS/Guix.
In particular, in our own version of the system, there is a directory "/usr" which contains all user's directories, and which is stored on a relatively large, but slow moving head disk, while the othe files are on the fast but small fixed-head disk. — https://www.bell-labs.com/usr/dmr/www/notes.html
Isn’t that just a backronym? I read recently (here, I think) that on some original specific UNIX system, the user directories were there originally, and then other stuff that belonged elsewhere went in there because the other physical volume (where things like /bin were) ran out of space. Later another dish was added and profiles were moved there, to make /home.
It’s still hilarious. The fact that there’s a need for this hilarity doesn’t negate that. What a mess humanity have created… (Let’s write a suckless OS!)
systemd configuration lookups are lightweights and straightforward compared to ansible variables lookups rules :
--
Understanding variable precedence
Ansible does apply variable precedence, and you might have a use for it. Here is the order of precedence from least to greatest (the last listed variables override all other variables):
1 command line values (for example, -u my_user, these are not variables)
2 role defaults (defined in role/defaults/main.yml) 1
3 inventory file or script group vars 2
4 inventory group_vars/all 3
5 playbook group_vars/all 3
6 inventory group_vars/* 3
7 playbook group_vars/* 3
8 inventory file or script host vars 2
9 inventory host_vars/* 3
10 playbook host_vars/* 3
11 host facts / cached set_facts 4
12 play vars
13 play vars_prompt
14 play vars_files
15 role vars (defined in role/vars/main.yml)
16 block vars (only for tasks in block)
17 task vars (only for the task)
18 include_vars
19 set_facts / registered vars
20 role (and include_role) params
21 include params
22 extra vars (for example, -e "user=my_user")(always win precedence)
IIRC, Ansible had this exact variable precedence long before they were acquired by RH.
I get that hating RH is in vogue currently, but let's keep it grounded in reality please.
EDIT: And for the record, these Ansible precedence rules are not a bad thing. Once you are somewhat familiar with them, it allows you to write incredibly flexible and extensible playbooks and roles.
As an Ansible person, I couldn’t agree more. Being able to manage variables at basically any level of precedence is extremely useful.
I am also ever more deeply concerned about IBM’s hell-bent purge of all the good left at Redhat. I had thought it would take them less time to destroy Redhat, but I’m sure they wanted to make sure they spent time “listening” before making all these ecosystem-eroding changes.
Chef is worse. It's discouraged to use `override` in Chef cookbooks but nothing says that you can't, and introducing spooky action at a distance is just part of your life now.
I love how there are 21 places to look if you're trying to figure out why a CLI parameter is being ignored. Does it at least print a warning or something?
According to the comment above, `cmd -u my_special_user` will only use the value "my_special_user" if none of the other 21 places have another configuration for the user, right?
".d" directories are a well-established pattern to split a complicated config file into logical units. In the case of something like systemd this is absolutely essential, as you wouldn't want the config for your web server in the same file as the mail server.
/usr/lib is the location where the package manager will drop config files. This is the default, distro-provided configuration. /etc/ is where the administrator can place config files, providing system-specific overrides. /run/ is for files "created at runtime", I am not quite sure what the point of those is.
/run is useful for configuration generation. Sure, you could maybe poke an API and have it do the right thing, but if "everything is a file" then why not put your configuration in a file? That way it's easier to see what's actually happening.
Systemd is also a useful demonstration of how to show people where configuration comes from -- `systemctl status <unit>` will include a list of all the files found during lookup, and `systemctl show <unit>` will give you the resolved unit configuration including all the defaults.
Drop in files (config in .d folders) are a core extension point of systemd and not a symptom of an overly engineered config system. These are how services are meant to be custom tailored to specific machines and even user's needs.
Let's say you need to pass a special flag to sshd on your system, a bad approach would be to go edit the distro shipped sshd.service file. Your change will work until the distro ships an updated file and you forget to hack in your change again.
Instead place a sshd.service drop-in config file in the appropriate place to customize the service as necessary. Your config tweak takes precedence over the distro shipped config, but doesn't live inside it or conflict with upstream changes.
Over-elaborated hierarchies where configuration files may reside, and their precedence schemes.
E.g. from systemd.unit manpage ( https://www.freedesktop.org/software/systemd/man/systemd.uni... ):
>...
> In addition to /etc/systemd/system, the drop-in ".d/" directories for system services can be placed in /usr/lib/systemd/system or /run/systemd/system directories. Drop-in files in /etc/ take precedence over those in /run/ which in turn take precedence over those in /usr/lib/. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.
> ...
And this is just an excerpt which doesn't cover the details. Read the manpage for all the hilarity.