Linux Administration
Linux Processes and systemd: ps, Signals, Units, journalctl and Timers
Intermediate9 min readUpdated
A running Linux system is a tree of processes supervised by systemd. This guide covers the process side first — states, ps and top, signals, background jobs and scheduling priority — then moves to systemd: what a unit is, how to write and harden a service, how dependencies and targets order the boot, and when a timer is a better answer than a cron entry.
What you will learn
- Interpret process state, memory and CPU columns in ps and top output
- Send the right signal to stop, reload or inspect a process and understand what cannot be caught
- Control foreground and background jobs and detach long-running work from a terminal
- Write, enable and harden a systemd service unit and read its logs with journalctl
- Express boot ordering with dependencies and targets, and schedule work with systemd timers
Everything running on a Linux machine is a process, and every process except the first was created by another one. That single fact explains a surprising amount: why a shell variable does not survive a script, why closing a terminal kills what it started, and why service management is really about parentage and lifetimes rather than about starting programs.
For its first few decades Linux managed that lifecycle with shell scripts in /etc/init.d. Every current distribution now boots with systemd, which supervises services directly, tracks their processes in control groups, collects their output into a structured journal and starts things in parallel according to declared dependencies. This guide covers the process concepts you need at the shell, then the systemd layer that sits on top of them.
The life of a process
A process is created when an existing one calls fork(), producing a near-identical copy, and the copy usually calls execve() to replace its memory image with a different program. The child inherits the parent's environment, open file descriptors, working directory and umask — that inheritance is what makes shell redirection work at all, as described in shell basics.
When a process ends it becomes a zombie until its parent calls wait() to collect the exit status. If the parent dies first, the orphan is re-parented to PID 1, which reaps it. PID 1 on a systemd host is systemd itself; inside a container it is whatever your image's entrypoint is, which is why containers that run a shell script as PID 1 accumulate zombies and ignore signals unless the script is written for the job.
The state column you see in tooling maps to what the kernel is waiting for:
| State | Meaning |
|---|---|
R | running or runnable — on a CPU or in the run queue |
S | interruptible sleep — waiting for input, a timer, a socket |
D | uninterruptible sleep — usually blocked on disk or NFS I/O; cannot be killed |
T | stopped, typically by Ctrl-Z or SIGSTOP |
Z | zombie — finished, exit status not yet collected |
A pile of processes in D is the classic signature of a storage problem, not a CPU problem. That distinction matters when you are reading load average, and it is explored further in logs and troubleshooting.
Looking at what is running
ps prints a snapshot. Two invocations cover most needs: the BSD-style ps aux for a wide overview, and ps -ef for the same information in the UNIX style. The forest view is the one worth memorising, because parentage is usually the question.
$ ps -eo pid,ppid,user,stat,pcpu,pmem,rss,etime,cmd --sort=-pcpu | head -5
PID PPID USER STAT %CPU %MEM RSS ELAPSED CMD
1842 1 postgres Ss 7.4 9.1 748120 3-04:11:52 /usr/lib/postgresql/16/bin/postgres
2310 1842 postgres Ss 2.1 1.4 118004 3-04:11:50 postgres: checkpointer
9921 1 www-data S 1.8 0.6 52208 02:14:07 php-fpm: pool www
$ ps -ef --forest | grep -A3 sshd
RSS is resident set size, the physical memory actually mapped in; VSZ (virtual size) counts address space and is routinely enormous and meaningless for a JVM or a database. Compare RSS between processes, not VSZ.
For a live view, top refreshes every few seconds; press 1 to split CPU usage per core, M to sort by memory, P by CPU, k to signal a process and f to choose columns. htop, if installed, adds a scrollable tree, mouse support and per-process signalling from a menu. pidstat 2 from the sysstat package samples per-process CPU, memory and I/O at a fixed interval, which is far more informative than a single snapshot when you are chasing an intermittent spike.
Finding a process by name is pgrep, and pgrep -a shows the full command line:
$ pgrep -a -u www-data php-fpm
9921 php-fpm: pool www
9922 php-fpm: pool www
Signals
A signal is a small asynchronous notification delivered to a process. Most can be caught by a handler, ignored, or left to the default action.
| Signal | Number | Default action | Typical use |
|---|---|---|---|
SIGHUP | 1 | terminate | terminal hang-up; by convention, "reload configuration" for daemons |
SIGINT | 2 | terminate | Ctrl-C |
SIGQUIT | 3 | terminate + core | Ctrl-\; a JVM prints a thread dump instead |
SIGKILL | 9 | terminate | uncatchable, last resort |
SIGTERM | 15 | terminate | the polite default of kill and systemctl stop |
SIGSTOP / SIGCONT | 19 / 18 | stop / continue | uncatchable pause and resume |
SIGUSR1 / SIGUSR2 | 10 / 12 | terminate | application-defined, often log rotation or debug toggles |
kill 9921 # SIGTERM by default
kill -HUP $(pgrep -o nginx)
pkill -u bob -TERM # every process belonging to bob
kill -QUIT <jvm-pid> # thread dump into the JVM's stdout
kill -0 9921 && echo "still alive" # signal 0 tests existence and permission only
Reaching for kill -9 first is the most common bad habit in this area. It denies the process any chance to flush a write buffer, commit a transaction or clean up a lock file, and it leaves stale sockets and PID files behind. Send SIGTERM, give it a few seconds, then escalate. If a process ignores SIGTERM repeatedly, that is a bug worth reporting rather than a reason to keep using -9.
SIGKILL also cannot help with a process stuck in D state: it is blocked inside the kernel, and the signal is only delivered when it returns to user space. Fix the storage or network mount it is waiting on.
Job control at the shell
Interactive shells group the processes of one command line into a job so you can suspend and resume them.
$ ./import-data.sh
^Z
[1]+ Stopped ./import-data.sh
$ bg # resume it in the background
$ jobs -l
[1]+ 21507 Running ./import-data.sh &
$ fg %1 # bring it back to the foreground
& starts a command in the background straight away; wait blocks until background jobs finish. None of this survives the session: when the terminal closes, SIGHUP goes to the job. nohup cmd & and disown %1 both dodge that, but on a modern system the honest answers are a terminal multiplexer (tmux, screen) for interactive work and a transient unit for anything unattended:
systemd-run --unit=import-once --collect /usr/local/bin/import-data.sh
journalctl -u import-once -f
Scheduling priority
The kernel's scheduler weights processes by their nice value, from -20 (most favourable) to 19 (least). Only root can lower a nice value.
nice -n 10 tar -czf /backup/site.tar.gz /var/www # start a batch job politely
renice -n 5 -p 21507 # adjust a running one
ionice -c 3 -p 21507 # idle I/O class: only when nothing else wants the disk
Nice values are a hint, not a quota. When you need a real ceiling, use the control-group limits systemd exposes on any unit — CPUQuota=, MemoryMax=, IOWeight= — because those are enforced by the kernel rather than negotiated by the scheduler.
systemd: units and the unit file
systemd manages units, each described by an INI-style file. The suffix determines the type: .service for a daemon or one-shot job, .socket for socket activation, .timer for scheduled work, .mount and .automount for filesystems, .target for a grouping point, .path for filesystem triggers, .slice for resource partitioning.
Unit files come from three places, in increasing priority: /usr/lib/systemd/system/ (shipped by packages, never edit), /etc/systemd/system/ (yours), and drop-in directories such as /etc/systemd/system/nginx.service.d/override.conf for overriding a few settings of a packaged unit. systemctl edit nginx creates a drop-in for you; systemctl cat nginx shows the effective, merged configuration.
Here is a complete unit for a Java application, of the kind you would deploy alongside the packaging described in packaging and deployment:
# /etc/systemd/system/orders-api.service
[Unit]
Description=Orders API
Documentation=https://example.com/runbooks/orders-api
After=network-online.target postgresql.service
Wants=network-online.target
[Service]
Type=exec
User=orders
Group=orders
WorkingDirectory=/opt/orders-api
EnvironmentFile=/etc/orders-api/env
ExecStart=/usr/lib/jvm/java-21-openjdk-amd64/bin/java -XX:MaxRAMPercentage=70 -jar app.jar
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
TimeoutStopSec=30s
# Hardening: each line removes something the service does not need
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/orders-api
MemoryMax=2G
[Install]
WantedBy=multi-user.target
Three fields deserve comment. Type= tells systemd when the service counts as started: exec as soon as the binary is executed, simple as soon as it is forked, notify when the program signals readiness via sd_notify, forking for old-style daemons that background themselves, oneshot for a task that runs and exits. Getting this wrong is why a dependent unit sometimes starts before the service is really listening; notify is the accurate option when the program supports it.
Restart=on-failure restarts after a non-zero exit or a signal but not after a clean stop. Combine it with StartLimitIntervalSec= and StartLimitBurst= in [Unit] so a crash loop eventually gives up instead of hammering the machine.
WantedBy=multi-user.target is what systemctl enable acts on: enabling creates a symlink in multi-user.target.wants/, which is why enable and start are separate operations — one affects the next boot, the other affects now.
Driving systemd
sudo systemctl daemon-reload # after adding or editing any unit file
sudo systemctl enable --now orders-api # start it and start it at boot
systemctl status orders-api # state, PID, cgroup, last log lines
systemctl is-active orders-api # scriptable: exit code plus one word
sudo systemctl restart orders-api
sudo systemctl reload orders-api # runs ExecReload, no downtime if supported
systemctl list-units --type=service --state=running
systemctl list-unit-files --state=enabled
systemctl show orders-api -p MemoryMax -p Restart
Two commands earn their keep when something is slow or broken at boot: systemd-analyze blame ranks units by start-up time, and systemd-analyze critical-chain shows the dependency path that actually determined the boot duration. systemctl list-dependencies orders-api prints the tree in the other direction.
systemctl mask is stronger than disable: it links the unit to /dev/null so nothing, not even another unit's dependency, can start it. Remember to unmask before wondering why a package's service will not run.
Reading a unit's logs
systemd captures each unit's standard output and error into the journal, tagged with the unit name, the PID, the boot id and priority. journalctl queries it.
journalctl -u orders-api -f # follow, like tail -f
journalctl -u orders-api --since "1 hour ago" -p err
journalctl -b -1 -u orders-api # previous boot
journalctl -u orders-api -o json-pretty | head -40
journalctl --disk-usage
Journal retention and whether it survives a reboot are configured in /etc/systemd/journald.conf; the storage question and the relationship with /var/log are covered in the troubleshooting guide.
Targets, dependencies and ordering
A target is a unit that groups others, replacing the old runlevels. multi-user.target is a fully booted system without a graphical session; graphical.target pulls that in and adds a display manager; rescue.target is single-user mode. systemctl get-default and set-default control which one boots.
Dependencies and ordering are deliberately separate concepts:
Requires=andWants=pull another unit in —Requiresfails your unit if the other fails,Wantsdoes not.After=andBefore=only order units that are already being started; they pull nothing in.
A unit with After=postgresql.service but no Wants= will start after the database if the database is starting anyway, and immediately if it is not. Almost every "my service starts too early" report is a missing Wants= or the use of network.target (interfaces configured) where network-online.target (an address is actually usable) was meant.
Timers instead of cron
A timer unit activates another unit on a schedule. It needs two files: the work and the trigger.
# /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly database dump
[Service]
Type=oneshot
User=postgres
ExecStart=/usr/local/bin/db-backup.sh
# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run the nightly database dump
[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true
[Install]
WantedBy=timers.target
sudo systemctl enable --now db-backup.timer
systemctl list-timers --all
systemd-analyze calendar "Mon..Fri 09:00" # verify an expression before trusting it
sudo systemctl start db-backup.service # run it once, now, to test
Persistent=true runs a missed job after the machine comes back up — the property cron lacks and the reason backups silently skip a day on laptops and spot instances. RandomizedDelaySec prevents a hundred machines hitting the same endpoint at 02:30:00. OnUnitActiveSec= expresses "every 15 minutes since the last run" for jobs whose duration varies. And because the work is an ordinary unit, its output lands in the journal instead of being emailed to a mailbox nobody reads.
Cron still exists and is fine for a handful of personal jobs; crontab -e edits your own table and /etc/cron.d/ holds system entries with a user column. Just remember that cron runs with a minimal environment and no login files, so scripts need absolute paths and an explicit PATH.
Common mistakes
- Editing a packaged unit in
/usr/lib/systemd/system. The next upgrade overwrites it. Usesystemctl editor a file in/etc/systemd/system. - Forgetting
daemon-reload. systemd caches unit files; a change is invisible until you reload, andstatuswill not tell you why. - Confusing
enablewithstart. Enabling only affects boot. Useenable --nowwhen you mean both. - Using
Type=simplefor a program that forks. systemd tracks the wrong PID,MAINPIDis wrong, and reload and stop misbehave. After=network.targetfor anything that binds a public address. Usenetwork-online.targetwith a matchingWants=.- Killing with
-9by reflex. You lose the clean shutdown path the service already implements. - Relying on nice to protect a busy machine. Set
CPUQuota=andMemoryMax=on the unit instead.
Key takeaways
- Processes form a tree; children inherit environment and descriptors, orphans are adopted by PID 1, and a zombie is an uncollected exit status rather than a running program.
ps -eowith explicit columns,top/htopandpgrepanswer most "what is running and why" questions; compare RSS, not VSZ.- Send
SIGTERMand wait beforeSIGKILL;SIGHUPconventionally reloads configuration, and nothing reaches a process stuck in uninterruptible I/O. - A systemd unit file declares how to start a program, when it counts as started, how to restart it and what it is allowed to touch; drop-ins override packaged units safely.
Wants=/Requires=pull units in,After=/Before=only order them, andnetwork-online.targetis what network services actually need.- The journal holds every unit's output;
journalctl -u … -p err --sinceis the fastest route to what went wrong. - Timers give scheduled work the same units, logging, dependencies and catch-up semantics as everything else systemd runs.
Sources and further reading
Frequently asked questions
- Why does my service die when I close the SSH session?
- A program started from a login shell belongs to that session. When the terminal disappears the kernel delivers SIGHUP to the foreground group, and most programs exit. Prefixing with nohup or backgrounding with an ampersand only papers over it. Anything meant to survive belongs in a systemd unit, or at minimum in systemd-run --user --scope, which places it outside the session cgroup.
- What is the practical difference between SIGTERM and SIGKILL?
- SIGTERM is a polite request: the process can catch it, flush buffers, close sockets, remove its PID file and exit cleanly. SIGKILL cannot be caught, blocked or ignored — the kernel destroys the process immediately, so open transactions are lost and temporary files remain. Always send SIGTERM first, wait, and reserve SIGKILL for a process that has genuinely stopped responding.
- Why is a process shown as defunct or zombie, and how do I remove it?
- A zombie is a finished process whose exit status has not yet been collected by its parent with wait(). It holds no memory or CPU, only an entry in the process table. You cannot kill it because it is already dead. Fix the parent — restart it or send it SIGCHLD — and the entries clear. If the parent exits, init adopts the children and reaps them.
- Should I use a systemd timer or a cron job?
- Timers integrate with the rest of systemd: the work runs as a unit, its output goes to the journal, dependencies and resource limits apply, Persistent=true catches up after downtime, and RandomizedDelaySec spreads load across a fleet. Cron is fewer files and is fine for a simple periodic script on a machine you manage by hand. On a server you also manage with units, prefer timers for consistency.