For me, systemd is the best thing since sliced bread.
As a programmer, I now don't need to care about dropping privileges, managing logging, daemonization (the moment I need to do the double-fork dance again, chairs will be flying, I swear), dropping into a chroot, and do half-arsed guesses "is it up and running yet?" from a convoluted mess of shell code that looks like a bunch of hair stuck down a drain for a month.
I just write an easy-to-debug program which I can launch from command line and see it run, and when I'm satisfied, a systemd unit from a cookie-cutter template is going to make it run. Service dependencies are now a breeze, too.
If I need to limit resources, I can just declare the limits in the unit. If I want a custom networking/mount namespace, it's taken care of.
I agree. It puts the system administrator in control of a lot of these things. Sometimes that can be annoying (the developer knows that systems calls they need) but often it is a huge benefit. I think socket-passing especially is a huge win, it shifts a huge amount of complexity out of each application and gives huge benefits as the administrator can configure the sockets however they want without needing each application to support each feature independently. Furthermore it removes one of the most common reasons why applications need to be started as root.
It's great... until something unexpected happens. Like your NIC doesn't have link on the ethernet cable and systemd waits for minutes without allowing you to abort waiting on the network because other units depend on it.
Or how about adding new buggy DNS code that doesn't work in common scenarios? Oh, sorry, here's another CVE because we didn't create enough test cases for the corner cases that are actually important.
Or oops, "nobody uses ntsysv", right? Or "We don't need to implement chkconfig even though we broke it".
systemd is a monolithic beast that is absorbing everything else in the system without considering that some of its decisions should be able to be disabled, and a lot of the design decisions are half baked. I don't believe in the philosophy of design its author has embraced. Progress is good, but please, stop breaking shit that has worked for decades. Anyone can write new code that partially implements a feature, but it takes real effort to responsibly migrate users from tools that worked to your new shiny half assed kitchen sink.
It may be great for less-skilled people, but for anyone running anything where it's too critical to outsource support it's then necessary to have a systemd expert inhouse (and such a person has proven extremely hard to find).
I have never encountered the need for this and I honestly doubt you have either. Systemd fails mysteriously far less often than poorly written init files I have seen and it is honestly not that hard to debug almost every failure I've ever encountered. And I have worked on some very large scale systems with systemd. A systemd file I helped write has run many millions of times over without issue. I honestly find this comment impossible to believe
I’m a big fan of systemd, but we’ve definitely run into actual bugs in it. Especially the early RHEL7 days (7.0-7.3). Some were fixed upstream, some we’ve worked around, etc.
In the beginning, the road was quite bumpy, and my irk was that people's use cases were being dismissed in a very cavalier fashion: like, if I didn't think of a workflow, or a use case, or a piece of software that doesn't play nice which you cannot change, it's not valid and let it burn. Oh, and whoever points these things out, or ridicules me for this attitude, is a troll or a hater.
In 10 years, though, reading recent bug reports, I can see that the project's leads have grown out of it, largely.
Case in point, they had recently put a bug into a 24x release that made a lot of machines, including mine, unbootable, but the fix was just as quick.
There is still no way to filter application output at collection time, which can functionally make journald useless if you have an application that is too chatty, and will likely never be given that (AFAICT) there is hefty ideological opposition from the systems developers¹.
Isn't this something that, to a degree, might be mitigated by rate limiting journald has?
IMO Lennart is right in that collection should be optimized to a point where it is (almost) never the bottleneck. I, personally, would put a filtering program between the chatty application and journald.
- Legacy application, with no access to source code.
- Company that wrote the application hasn't existed for a decade.
- It logs several useless lines every second. This is non-configurable, and includes a timestamp which makes each linue distinct.
- It also logs some very important lines that _must_ be responded to.
I don't think this is a unique situation to find yourself in, and currently this application is permitted - explicitly - to spam the journal to a point that it is effectively useless. Yes, you can use a filtering application to sit in between systemd but that's a workaround and not an actual solution (as stated in the issue linked above):
- it could not be "plugged in" via overlay (i.e. modification of Exec... is needed - this gets ugly especially if you have ExecStartPre/ExecStopPost)
- SyslogIdentifier has to be set to produce proper name in journald
- The filter itself has to be quite reliable (to not hang) and a bit sophisticated to properly handle signals, process shutdown, low disk space etc.
What you need is a pipe through grep. It’s as reliable as it gets. I don’t understand, though, why your log filter should all of a sudden grow disk space monitoring and signal handling. This is what systemd and journald should be doing, and in their recent versions, they are quite capable.
How about to open this exceedly verbose legacy program with IDA and fill with NOPs all the places where this program logs something useless? I mean, this is an activity which really should be named as "hacking", in opposition to playing with init scripts.
I generally agree, but will say that’s not always easy if it’s something system level that’s being induced to be too chatty —- you’re stuck replacing the upstream unit’s Exec, which frankly kind of sucks. I’m also trusting the filter to be as reliable, which may or may not be true.
Rate limiting works, but it’s a pretty blunt instrument and can lead to losing your actually valuable entries vs capturing garbage.
Might depend on your environment too, I generally find the newer you can get for systemd the better off you are (which is really quite high praise!). Unfortunately the life cycle for some of our environments is quite long for things like RHEL7 -> RHEL8.
As a systems guy with a focus more on ops, I agree. It's not all roses - journald/journalctl and binary logging can go die in a pit of fire for example - however setting LimitNOFile= in a unit is just really, really nice (as well as CPU limits and all sorts of other cgroup/namespace needs). But let me just mention again that journald/journalctl can go die in a pit of fire - if it wasn't for everyone adding rsyslog to create regular text files we would be a world of hurt. But in return we get almost complete, painless cgroup level handling right in the unit file with a simple key=value structure (so really, you don't have to know anything at all about cgroups or namespaces to be very effective). Most options have a doc for them - it's usually easier for me to find an obscure systemd setting with a nice blurb about what it does (in non-programmer speak) than it is to dig up an obscure sysctl setting, e.g.
I can/could do without timesyncd and resolved (it's easy - just use chrony e.g.) but I like udevd being now part of systemd. It would be nice to not write /etc/udev/rules.d/ and instead have a foo.udev unit type, perhaps that is in our future (we do have .device units, but it's not the same - yet? the future). In this ballpark I think it's more on each distro picking and choosing - Ubuntu for example drank the kool-aide much deeper than RHEL - RHEL for example uses chrony out of the box, not timesyncd. However udevd and logind seem to be common across all distros now, as another user commented the KillBackground=yes setting in logind is just horrible to have as a default. The whole "homed" thing makes me sad that it's even being coded, I hope nobody adopts that (I dislike it for the same reason I dislike automount); someone out there wants it though.
The ability to dynamically edit a unit (systemctl edit) and to dynamically alter the running service constraints (systemctl set-property), all PID file type needs are handled in /run (getting rid of the nasty SysV stale unexpected crash reboot pid problem which many scripts failed to handle properly). Users having the ability to use their own private init items (systemctl --user) is great - timers, socket activation, custom login units, all very well extended down into the user's control to leverage. I'm sort of 50/50 on cron vs. timers, that's more of a use case by use case decision (example: tossing a https://healthchecks.io "&& curl ..." is just a lot quicker and easier in cron, but running a dyndns script on my laptop with a timer is nicer).
Touching on systemctl edit, it's really easy now to show folks (think a DBA team who only has the fundamental ops skill) how to quickly chain their After= and Before= needs for start/stop of their (whatever) without having to go down a rabbit hole - it's simple to use, the words and design are accessible and familiar, the method by which it works is a little obtuse (it's rooted in understanding the "dot-d" sub-include design pattern). On RHEL at least it uses nano as the default editor, annoying to me but good for casual non-vim users and easy enough to override using $EDITOR.
I used SysVinit for all the same years as everyone else (Solaris to Debian to Red Hat, ops touches it all) and wrote many my fair share of complex init units to start DB2, Oracle, java appservers (anyone remember ATG Dynamo?); systemd handles natively what 75% of that work was/is (managing PID files, watching/restarting failures, implementing namespaces/cgroups, handling dependency chains, etc.); for those complex scenarios (looking at you, Tomcat) you can still just have a unit launch a very complex shellscript "like in the old days". I haven't looked in awhile, but last time I knew in RHEL7, Red Hat did exactly that with Tomcat - just had the systemd unit launch a script.
It is, however, a real bear to debug sometimes - it's far easier to "bash -x /etc/init.d/..." and figure out what in the world is going wrong than it is to debug systemd unit failures. But, the same holds true for trying to debug DBus (if you've never tried / had to, it's not fun at all without deep dbus knowledge). I would like to see the future add more ops-oriented debugging methodology - if you've every used "pcs" (the commandline tooling for Pacemaker offered by RHEL), we could really use "systemctl debug-start" type of interfaces to the commandline offer the same experience as "bash -x" days of old. There are debug settings, they're just not ergonomically dialed in for the ops user, IMHO - systemctl debug-start would save people a lot of headaches.
I generally love it but had some weird bug a month or two ago where something in the distro (AWS Linux 2) added a log trimmer config for squid that made systemd restart the squids, not reload but restart, every thirty minutes, on all the hosts, so all the clients got connection reset every thirty minutes. The signals to restart came from PID 1 but the fix was commenting out the log trimmer config for squid. Hard to debug - turning high level of debugging on and PID 1 logged restarting the squid but not why.
I feel your pain - I'm curious, was it a drop-in /etc/logrotate.d/ config which was sending a HUP? I don't run squid but Google'd up that it will take a USR1 to rotate logs which should not close the HTTP connections (allegedly HUP closes them). Perhaps the AL2 folks chose the wrong signal? https://wiki.squid-cache.org/SquidFaq/InstallingSquid#squid_...
I'm wondering if this new config made squid exceed any limits, causing it to hard stop. I'd expect the log trimmer to have/cause a bug in squid itself.
> journald/journalctl and binary logging can go die in a pit of fire for example
This is, of course, not a problem, because as systemd folk are wont to point out, systemd is not, in fact, monolithic, meaning they use well-defined interfaces and can be swapped out for an alternative.
That's true, but you can get all that without taking over the entire system. Upstart was a more lightweigh contender to systemd which would give you all that but none of the "Enterprise Linux Userspace Daemon" crap.
1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses.
2. Some of the systemd logging is binary, so good luck with that if there's a problem.
3. Have you tried non-systemd init systems other than sysvinit?
4. Yes, it is convenient when everything below your development is cenrtalized by a single entity. It can easily provide a consistently useful underpinning. But there's a price - overly strong coupling of the init system, the kernel and part of the user-space, centralized control of, well, almost all of how things work on the system, and
stagnation of the ecosystem due to there being only one game in town.
In addition to s6 itself, the author of s6 also wrote a survey <https://skarnet.org/software/s6/why.html> that provides a good large-scale view of the design space.
(My gripe with how systemd does a traditional init’s job is mostly the unit and dependency types, all of which AFAICT are specified in terms of actions on state transitions, not consistency conditions on states, so for all that it has a positive boatload of them I can’t actually figure out how to specify which configurations are permissible for my system.)
I stopped reading when they started attacking bind. Sheer ignorance. Also cathedral is good for him and his elite team but not in general?
Unix has always been worse is better. You may disagree but it is the secret sauce. YAGNI for that potentially ideal system. Sadly. I like mathematically solid systems but the people with good enough systems explore the solution space much quicker.
I’m not sure what you are referring to. Do you mean some other page on that website? There’s a fair bit of design philosophy there, and I vaguely remember reading it and agreeing with some parts while disagreeing with others, but that was years ago. I’ve no interest in defending an arbitrarily chosen subset of the author’s views in this thread.
Indeed I don’t even necessarily agree with everything said on the page I linked to (which I did reread before posting the comment), but regardless of what I think about the particular point in design space (s6) it advocates, I still consider it a good overview of the space itself and prior art in general, and that’s the only thing I claimed to offer. I do have some thoughts about init systems, but I don’t feel they’re ready to put them up for discussion here, so I haven’t.
> Software that does more instead of less is, simply put, badly designed software. Trying to come up with an all-encompassing solution is always a sign of developer hubris and inexperience, and never a sign of good engineering. Ever. Remember sendmail, BIND, INN,
Yeah from the skarnet page above. BIND had many troubles over the years but mostly because we were all learning about secure code practices in C etc. bind was just a name server. Never heard of it reading mail. There was a lot to learn about network exploits, for sure. But attacking bind as doing too much seems disingeneous. And the whole tone reminds me of the same tone as https://suckless.org/ who cares if some browser takes a Gig of RAM or my window manager is ginormous. My laptop is hard pressed to use all the RAM it has and if systemd and the kernel is running sixty daemons but the unit files are easy to write as a here doc in cloud init, then win win win.
(Running postfix and bind on my personal cloud VMs; have run Apache, haproxy, nginx and lighttpd, as well as built in Python web servers).
Looks quite neat, though perhaps just a little more complex than systemd service files, I guess they essentially boil down to similar things.
Where systemd has one file with a bunch of settings in, this is split into a directory of single-purpose files in s6. I'd hesitate to call it "better", but from a surface reading it seems roughly equivalent from a usability perspective.
According to the author, Laurent Bercot, the code for s6 is much cleaner than the systemd code. For anyone curious I guess the best way to find out is to compare and make a judgment of one’s own.
While I appreciate clean code when I have to work on it, it's somewhat orthogonal to my requirements when we're talking about running systems rather than building them.
I guess it could be preferred as a second-order factor.
In some ways yes, in other ways no. I've looked at both, and there is always a certain amount of "ugliness" that comes with writing low level C software for Unix.
I'm not familiar with s6 myself. But - the choice between a single file or multiple files in a directory is a relatively minor issue in the overall scheme of things. i.e. if s6 is "better" or "worse", it's not because of this fact.
I really don't understand the scenario where binary logging is a problem. journalctl is a command just like, I dunno, gzip, and people are fine with gzipped logs. If something goes horribly wrong with your system, you're not looking at logs with an oscilloscope, you're looking at logs by mounting the disk on some other working OS - whether it's the initramfs, or a live CD, or whatever. You can run gunzip < /mnt/brokensystem/var/log/messages.1.gz from that other working OS to read text logs; you can also run journalctl --root /mnt/brokensystem to read journald logs.
(Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)
Binary logging would be fine if it Just Worked, but my experience is that journald/journalctl does not just work, and when it breaks I don't have logs.
For example, if the system clock is not monotonic, my text logs are still written sequentially in order, and it's easy to figure out what happens. It is the stated view of the journald maintainers that, if your clock is not monotonic, you get to keep both pieces. https://github.com/systemd/systemd/issues/662 (The specific issue described there sounds complicated, but the fundamental problem behind it, which I have hit multiple times, is "if your system can't maintain a monotonic clock at all times, including early boot, then your logs will be mangled.")
Similarly, a text log line is complete the moment it's written, even if the process that wrote it goes away immediately afterwards. Journald, by contrast, performs asynchronous metadata lookups for _each_ log line at some later time, which it apparently does not cache. This means that, when a process dies suddenly, the final messages will not appear in "journalctl -u", because they failed to get tagged with metadata. (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.)
So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.
> (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.)
I am so happy you mentioned this, as I am in a position of developing a daemon used by people who insist on using systemd, and I keep asking them for logs when things fail and, in fact, the most critical logs -- arguably the only ones that ever truly matter: the ones that come immediately before the daemon terminates for some reason -- are often missing. Now I know why :/.
> So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.
Yeah... I hadn't even gotten to the end of your message before immediately going "omg I have to look into this" and the most apt description that was coming to my mind for a logging system that fails to actually store logs -- and particularly the most critical death throws to the log before termination -- is 100% "incompetent" :/.
> You can run gunzip < /mnt/brokensystem/var/log/messages.1.gz from that other working OS to read text logs; you can also run journalctl --root /mnt/brokensystem to read journald logs.
It's subtle, and you've accidentally missed it like many do - you made an assumption that everything logs through journald. On my personal system where I do not have rsyslog (aka trying to live the journald life) I have non-journald text logging for: httpd, sa (sar/sysstat), lightdm, audit, atop, Xorg, cups, fdsync and samba. If we just stick to httpd, sa and samba (most folks know how those work) it shows how logging is way more complex than what is captured in the journal - these apps by design maintain their own logs.
So now you launch your rescue ISO (let's assume modern sysrescuecd which has journalctl) and get your filesystems mounted, you have to employ two different techniques - journalctl for that single-use format and then your traditional find/grep skills for everything else. You don't know why it crashed, how it crashed and are on a fishing expedition. Was it an RPM upgrade? (logged to yum.log or rpm.log, not journald) Was it update-initramfs running out of disk space truncating your initrd? (sometimes logged to journald depending on distro, sometimes not). Don't know until you start following breadcrumbs, find/grep is the superior toolset (much like the trebuchet is the superior siege weapon).
It's not that you cannot do it, it's that journald-only (no rsyslog) forces using a specific method with specific tools to access what should be extremely easy to access data. Keeping logging all in the same format (text) is what I want, it's not that I don't know how to use journalctl; I don't want to use journalctl, it's a pig in lipstick.
> (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)
I agree with this, they belong over in /var/lib/ somewhere and those files bug me, always have. "Just because these other guys did it" is however a logical fallacy, they are (IMO) just as wrong because they're more like database files than they are logs in my opinion. (the secure/auth.log is more a "log")
If you want to use journald for those programs you could just configure them to pipe the logs there, or you could just disable journald logging and have it pipe its logs to the syslog. If your distro didn't configure all those programs to log to the same place, that's more of a distro configuration problem than a problem with any specific syslogger. I personally dislike having a bunch of services that try to implement their own log rotation, I would rather have that handled by the system.
System Ops (at scale, large company, lots of teams) in a nutshell: you did not build the system or choose what kind, it was most likely installed using vendor defaults by the systems owner and you were called in because it's misbehaving. Most likely 50 other people have touched various parts of it, some with skill some without.
"Give me an IP, username and password - what OS is it?" are about all you start with and go from there. It's probably a critical system to someone, and everyone swears on a stack of bibles that nobody did anything, touched anything or made a change. You have very specific domain knowledge (kernel, grub, SAN/storage, systemd, dbus, etc.) and typically ask a lot of questions to the systems owner as your fingers are flying ruling out reasons (low hanging fruit common issues).
Ok so complain to the ops department that they need to unify their logs. That's their problem, not yours. If the company is big I would expect them to be doing that anyway, either they coalesce around a journald-type thing that aggregates the logs locally, or they'll use another centralized service like datadog, splunk, etc. Edit: If you are ops then this is your entire wheelhouse, you should be able to solve it at scale without messing everything up.
Sure, but that's entirely the problem those centralized logging services were made to solve. You make it really easy for everyone in the company to put their logs in the right place.
1. journalctl is painfully slow. Most frequent commands I use: journalctl -u myunit --since=today and journalctl -u myunit -n 1000 take minutes on loaded servers with large amount of logs. 1st command is instant for daily rotated plain text logs and 2nd is almost instant for any plain text logs.
In theory it is possible to make a binary log database which will work fast for many queries including these, but journalctl is not fast at all.
2. People are fine with gzip because log compression is usually done in a way which makes log corruption/loss highly unlikely:
gzip log.ystd, fsync log.ystd.gz, rm log.ystd
At any time you can reset a server and you will either have uncompressed or compressed logs, not some partially written binary blob.
Say the computer writing the log was destroyed, you were able to recover the storage, but the only computer available did not have journalctl.
Or if you're writing to network storage and would like to analyze the logs from your haiku box.
Text is not perfect, but it's the one thing that is always available.
Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption well. Which is something kind of important for logs.
The journalctl binary format seems to handle corruption pretty well. That was a design criteria.
Everyone forgets or tries to ignore that text files ARE A BINARY FORMAT. It is encoded in 7-bit ASCII with records delimited by 0x0a bytes.
Corruption tends to be missing data, and so the reader has to jump ahead to find the next synchronization byte, aka 0x0a. This also leads to log parsers producing complete trash as they try to parse a line that has a new timestamp right in the middle of it.
Or there's a 4K block containing some text and then padded to the end with 0x00 bytes. And then the log continues adding more after reboot. Again, that's fixed by ignoring data until the next non-zero byte and/or 0x0a byte. This problem makes it really obvious that text logs are binary files.
And here, this isn't perfect but if you had to hack out the text with no journalctl available you could try this:
grep -a -z 'SYSLOG_TIMESTAMP=\|MESSAGE=' /var/log/journal/69d27b356a94476da859461d3a3bc6fd/system@4fd7dfdde574402786d1a1ab2575f8fb-0000000001fc01f1-0005c59a802abcff.journal | sed -e 's/SYSLOG_TIMESTAMP=\|MESSAGE=/\n&/g'
The journalctl binary format seems to handle corruption pretty well. That was a design criteria.
Thanks for pointing that out. I guess thats why they came up with their own format instead of just using sqlite, or something else that is already a standard.
Everyone forgets or tries to ignore that text files ARE A BINARY FORMAT
That's a bit pedantic, even for HN standards :-)
But yes, I know all about fragile log parsers and race conditions of multiple processes writing to the same file. I was just thinking about a scenario where you end up having to read raw logs when things go haywire.
> the only computer available did not have journalctl
This is exactly what I don't understand. This is a world where no other computer exists? I have bigger things to worry about (even if you scope the problem down to to "no other computer with journalctl installed exists on my network").
> Or if you're writing to network storage and would like to analyze the logs from your haiku box.
I don't have any Haiku boxes, but I do have Windows boxes. At my day job, where I'm a Linux sysadmin for a large finance company, my workstation is Windows, and I don't have admin on it. So even with conventional UNIX logs, a much more common case is - as I mentioned - that I'd want to read them from a computer that doesn't have gzip installed.
But we're fine with gzip logs, because the way I'd actually do this is to get a Linux computer running.
(Also, keep in mind that the filesystem itself is a binary format. If you're really worried about reading logs from Haiku, you wouldn't put /var/log on NFS because that sounds like a terrible idea, you'd log to a FAT filesystem. But nobody actually does that. Everyone's logs are on ext4 or XFS or btrfs or whatever, and nobody says those formats are a bad idea.)
Note that not every journald/journalctl is created equal. It's easy to end up with logs written on one computer that journalctl on another computer can't read, even if the latter is a newer version, depending on which settings each one was compiled with (which is mostly a distro question.)
All right, then that seems like a reasonable objection to the format, not simply "It's a binary format." gzip files created anywhere can be read by any version of gzip.
A standalone (and cross-platform) journalctl file reader seems like a useful thing to have around and not a terribly difficult thing for someone to build.
This is exactly what I don't understand. This is a world where no other computer exists?
What about a windows user that has a smart device of some sort that is acting up. They're able to dump the files, but now they have to read them. It's a more realistic scenario than a haiku box, but its the same idea.
And for the record, I am against gzip logs too. I think if you need to save that many logs you should be exporting them to another system instead of archiving them locally.
I think the argument is really about transferablilty of skills. I already know how to manipulate compressed files because I have to do that in other places. And once you realize logs are just text files, I can immediately transfer all my skills of dealing with text files to dealing with logs.
But now I have to learn another set of tools (or at least another command to convert it to text files). It's not really a huge issue (I'm not really a systemd hater), but there can be a bit of dread when all the tools move off of standard formats, like simple text files, to custom formats, requiring you to learn idiosyncrasies of lots of different packages.
(And the reverse is true. Knowing how to use journalctl only helps me with systemd, and nowhere else. It's a piece of knowledge helpful in only one area that I cannot transfer anywhere else)
I think this is exactly it. Journalctl doesn't resemble any tools I already know, so I have to look up how to use it each time. Binary logs aren't the problem; we have had those for ages (e.g lastlog). I just wish journalctl were a little more familiar.
It's also just a really good tool for reading logs. Sure, it's always possible to cobble something together that merges a bunch of text logs and orders entries by date, but with journalctl that's just what it does and it's as simple as `journalctl -u spam -u eggs -u spam`.
> I know that it can be achieved without binary logs, but does any popular distribution implement compression out of the box?
Are you saying that since distros don't optimally configure a program out of the box, we should scrap and replace the whole program instead of just fixing its default configuration?
> we should scrap and replace the whole program instead of just fixing its default configuration?
Loads of distributions assisted that upstream works out of the box with systemd. That way the work is shared across distributions. Not sure why you make such a strange suggestion. Loads of work has been saved thanks to systemd. So much more is shared across distributions it's kind of crazy to look back.
It's not at all clear what you're talking about, honestly. First you talk about distributions, then a vague "we", then some weird "scrap and replace" nonsense. No idea who you mean with a sentence such as "we scrap and replace rsyslog". It really makes no sense, if there's a developer that works on rsyslog the work will continue. Distributions might change a default, but it didn't seem to be about distributions.
To make things a bit clearer I talked about all the benefits of having systemd across distributions, plus how the work is shared. This because you seemed to not understand the benefits that systemd has for distributions, in response to the "since distros don't optimally configure a program out of the box". Again, with systemd work it often allowed things to be done upstream. Configuring things in a distribution is a waste of time, previously distributions often wrote their own init scripts.
One person specifically mentions that it's nice when stuff is good by default. Complaining that it's some fault of the distribution completely misses the point: systemd allows stuff to be shared across distributions! No need to complain that either the default wasn't ok, the init script had a bug, etc. It's shared by default. Similarly, once a mistake is found, the bugfix can be made in one place.
That's the nice bit. Your weird comment isn't clear at all, plus completely misses what's nice about systemd for distributions.
Stagnation of the ecosystem should not be an argument here as systemd is trying to replace the sysvinit that has its roots in Unix System V released almost 40 years ago.
The choice isn't systemd or roll-your-own-init-system. There are alternatives like runit, openrc etc.
You can do the same painless setup (arguably even easier) with runit as the base template requirement is literally just
#!/bin/sh
the_executeable &
Granted, logs in runit are optional but there are problems with default logging too, e.g. Docker will keep filling logs until the disk is full unless you explicitly tell it not to in either its configuration or your own custom log rotation rules. Neither of which are default.
As a programmer, I now don't need to care about dropping privileges, managing logging, daemonization (the moment I need to do the double-fork dance again, chairs will be flying, I swear), dropping into a chroot, and do half-arsed guesses "is it up and running yet?" from a convoluted mess of shell code that looks like a bunch of hair stuck down a drain for a month.
I just write an easy-to-debug program which I can launch from command line and see it run, and when I'm satisfied, a systemd unit from a cookie-cutter template is going to make it run. Service dependencies are now a breeze, too.
If I need to limit resources, I can just declare the limits in the unit. If I want a custom networking/mount namespace, it's taken care of.