Shepherd is a wonderful little service manager. The project has been integrating the good features of systemd (socket activation, timers, etc.) while retaining the hackability of using a fully featured programming language for configuration. Been happily using it with Guix for over a decade!
NLnet has funded a really cool project to turn Shepherd into a distributed system that enables a fleet of Shepherds to cooperate. https://nlnet.nl/project/DistributedShepherd/
Also a note since there are a number of comments on here about this being a rival for Systemd: Shepherd precedes Systemd by quite a bit! Shepherd was previously known as "dmd", the "daemon for managing daemons", which was made in 2003! So in that sense, if anything was imitating anything (and not claiming anything was), it would be the reverse :)
You've been using Guix for over a decade? Is there somewhere I can read more about your experience? I think Guix is a cooler version of nix but I've been put off by the lack of packages and smaller community.
> Shepherd is a wonderful little service manager
I believe you
But what practical use is it?
I cannot tell.
> practical use
service manager.
Where by you can create a service of your own application in userland without needing to be PID 1 to execute the service. With all the bells and whistles of monitoring and trigger.
Eh, I'd say systemd still has far better dmcrypt support. The bad (I used to think it was terrible, but I tried installing Debian today, which lowered the bar even further) support for full-disk encryption is what keeps me off Guix, as a matter of fact.
okay
Taking a quick look at this, the first thing I notice is that the configuration file syntax is obtuse and nigh-illegible unless you already know what things are, and even then it's cumbersome and looks frustrating to write.
e.g. instead of systemd's simple:
Exec=/path/to/ntpd -n -c /etc/ntpd.conf -u ntpd -g
#:start (make-forkexec-constructor
(list "…/bin/ntpd"
"-n" "-c" "/…/…-ntpd.conf" "-u" "ntpd" "-g")
#:log-file "/var/log/ntpd.log")
I get that there are benefits to being able to do more advanced scripting in a service file, but this seems like a classic example of programmers writing things for programmers and not for users (or at least, not for users who don't 'get it').
Going to chalk this up as another 'GNU reimplemented something badly because of NIH syndrome' incident and move on.
I think this may be just due to unfamiliarity. Like, if that mess you shared was written as follows:
{
start: makeForkexecConstructor(["…/bin/ntpd", "-n", "-c", "/…/…-ntpd.conf", "-u", "ntpd", "-g"]),
logFile: "/var/log/ntpd.log"
}
And I think it's also hard to claim that one is better than the other. I personally have gotten my feet wet with Guix and I would love the time to become more familiar it.
While I agree it's hard to claim one is outright better than the other and there are a lot of factors that go into it, I do feel it's important to toss my chip in as someone who's not familiar with Lisp, Scheme, or any other functional language that I find it's syntax and layout particularly difficult to parse, let alone get started with compared to say, the INI-inspired systemd conf files. I can detail why but the mere fact that I'd have to learn an entire functional programming language (no matter how "easy/simple" it's claimed to be) to be able to competently edit a service file is a huge immediate turn-off for me.
I would definitely bat an eye at "make fork exec constructor", regardless of what syntax sugar you sprinkle on that abomination.
Eh, I'm a Schemer (not Guile though) but the above is dramatically less readable and, at a glance, understandable than the equivalent systemd .target file. The problem with Lisps (but especially Schemes) is there's magic everywhere that makes no sense without having a sizeable context of the program you're working in.
IMO in this example there seems too be to little magic.
I vaguely learned lisp years ago and i already prefer the guile scheme over systemd - guile correctly takes an argument vector instead of a command string which needs to be split using some obtuse shell quoting rules.
Having the shell taken out of the equation is what Lennart promised but never delivered.
Interesting take. As a hobbyist, I get special characters and escapes wrong in systemd all the time, because they try to look like regular shell but aren't. And systemd-escape (or whatever it's called) just makes an ugly mess that makes it hard for me to read.
I don't know lisp, but the idea of having each argument be part of a list (kind of like launchd's xml format) resonates with me.
> the actual guile scheme documentation didn't seem to explain that syntax at all.
(quote data)
'data
'x ⇒ x
'(1 2 3) ⇒ (1 2 3)
'#(1 (2 3) 4) ⇒ #(1 (2 3) 4)
(quote x) ⇒ x
(quote (1 2 3)) ⇒ (1 2 3)
(quote #(1 (2 3) 4)) ⇒ #(1 (2 3) 4)
— <https://www.gnu.org/software/guile/manual/html_node/Expressi...>
I agree systemd's is easier to understand, and I'm a happy systemd user.
But some of Shepherd's strengths are in other examples, and it makes sense to evaluate the whole.
A few examples:
* if we want multiple similar services, I'd prefer writing a Guile function than using systemd templates.
* the above Shepherd guile code lives lexically alongside your wider Guix guile code. AST-aware syntax highlighting and refactoring tools would treat them the same. That's pretty neat and beats having two different languages. Guix calls this g-expressions.
This "one language for all your system" is pretty compelling, but Guix's feature set is a bit too far behind NixOS at the moment for me.
You don't have to be an expert, but knowing the basics of Lisp helps. The full definition of the example "ntpd" service is clear to read.
> the actual guile scheme documentation didn't seem to explain that syntax at all
It does, here: https://www.gnu.org/software/guile/manual/html_node/Expressi...
It's a basic concept of Lisp. Everything is evaluated, unless quoted. Code is data is code!
> Code is data is code!
Which nowadays is 'yaml is code is yaml is code is yaml' in the devops world. The wheel turns.
I tell ya, it was sometimes difficult to be a lisp guy doing devops full-time.
...only with more parsing gotchas :(
The system is integrated into Guix, so makes sense to use s-exp for the configuration.
Systemd has done things that previously were trivial very difficult. I would like to know how it fares in that arena. The syntax can just be learned.
> NIH syndrome
I would be using nonGuix if it booted on my machine (libre seemingly has its claws too deep), instead of Nix, precisely because Guix has far less NIH: Nix is homegrown everything. Guile was a pre-existing project (for both Guix and The Shepherd), for a pre-existing language family.
I mean, they're pretty much the same right? You can't abstract away the services and you need to pass arguments to the executable (ntpd in this case).
I don't see more or fewer issues, but I also don't see the value to using Scheme instead of a conf. file like format.
At the end, there's talk about adding the possibility to live reconfigure services, which is pretty cool but I don't really see what kind of users are targeted by that use-case, at least not regular users like me who's running Linux on a single laptop.
The link at the end (https://spritely.institute/news/spritely-nlnet-grants-decemb...) is IMO much more interesting since it would make it possible to orchestrate several machines and their services as well. If you have live reloading as well, I think it would make for good developer experience.
My main gripe with Guile is its ergonomics and tooling: no LSP, no linter, no step-debugger that I know of and the docs are often unhelpful imo.
Maybe I'm not Scheme brained enough but I think it's a shame because there are some really cool projects out there (anything https://spritely.institute/ really).
P.S: I know that some people will tell me to get used to REPL based development which I think is fine for small-ish projects but I can't understand the flow one uses for larger projects where many components might need to talk to each other.
To be fair the Guile one is properly quoted and specifies the log file. make-forkexec-constructor is a really terrible name though, and Guile is a poor choice.
Guile has been the scripting language of choice for GNU before the Internet was called Internet. It hasn't even been disliked, it's been simply ignored, for the most part.
I know that's true, having heard it talked about a lot over the years. But why, though? What is it about Guile that made GNU pick it over CL or Scheme (or TCL or Lua or...)?
NM, answered my own question. It’s literally Scheme.
> TCL
I actually know the answer to that one!
> The principal lesson of Emacs is that a language for extensions should not be a mere "extension language". It should be a real programming language, designed for writing and maintaining substantial programs. Because people will want to do that!
> Another lesson from Emacs is that the way to make sure an extension facility is really flexible is to use it to write a large portion of the ordinary released system. If you try to do that with Tcl, you will encounter its limitations.
> Tcl was not designed to be a serious programming language. It was designed to be a "scripting language", on the assumption that a "scripting language" need not try to be a real programming language. So Tcl doesn't have the capabilities of one. It lacks arrays; it lacks structures from which you can make linked lists. It fakes having numbers, which works, but has to be slow.
From Why you should not use Tcl by Richard Stallman [1].
Brilliant. That's exactly the kind of information I was missing. I don't know enough about Tcl to opine on whether he was right, but if that's what RMS thought about it, I can see why he'd push back hard against it..
For those that don't know, GNU Shepherd is an init system (like systemd) written in Guile (a LISP), the unit files are written with Guile too.
That’s actually a pretty badass name for an init system.
My main remark is: please do consider DESKTOP more than HPC etc, Guix System have a great potential to surpass NixOS for good but can't until the desktop will be the main driver, because before working on a system we enjoy it personally, and personally means desktops and homeservers...
"give a man an inch and they'll take a mile" and what not, but,
I wonder how hard this would be to port to Hoot, a web assembly/wasm-ified Guile Scheme. https://spritely.institute/hoot/
It would be neat to have some real management APIs for the things running on a page, to be able to manage long running processes & workers.
A bit more niche than for general webapps, but systemgo comes to mind; a re-impl of some of systemd in go, specifically designed to be run on the web in Browsix. https://github.com/plasma-umass/systemgo
Amusingly enough, the main direction Shepherd is exploring right now is integrating Spritely Goblins (the folks that are also writing Hoot), so that it would have a distributed, capability-based process manager. I wouldn't say running Shepherd with Hoot is a direct goal of this, but it may fall out naturally from it.
Shepherd may actually work in Hoot right now -- I know the Spritely team were working on getting Guile's fibers library to work in WASM, and that's probably the biggest lift.
It may be possible to get Shepherd to work with Hoot eventually! I'm not sure what on earth it would mean though. What daemons would you manage in the browser? But it's indeed a fun idea!
If you could configure a fleet of machines from a browser window, over the network; that would be pretty cool. No clue how that would work in practice here though.
Since everything would be speaking OCapN you could use the browser interface as a convenient command center. This idea is becoming quite appealing to me actually... would be cool to have a little dashboard showing the health of my servers and be able to restart a failed service if I needed to.
Wow, that's a very fun reply! Nice.
Fingers crossed that the Stack Switching spec gets adopted & serves Guile Hoot's needs for fibers. https://github.com/WebAssembly/stack-switching
Stack switching in wasm would be great (as long as they aren't nerfed which they might be!) but note that Hoot has fibers right now: https://gitlab.com/spritely/guile-hoot/-/tree/main/lib/fiber...
> I wonder how hard this would be to port to Hoot
It would be interesting, though I don't know if it would make sense. In any case, the way to do it would be to extract all the code that does POSIX things into an abstraction layer. Shepherd is built on the Fibers async system which Hoot supports so that part shouldn't be an issue.