Linux Administration

Linux Processes and systemd: ps, Signals, Units, journalctl and Timers

Intermediate9 min readUpdated

A running Linux system is a tree of processes supervised by systemd. This guide covers the process side first — states, ps and top, signals, background jobs and scheduling priority — then moves to systemd: what a unit is, how to write and harden a service, how dependencies and targets order the boot, and when a timer is a better answer than a cron entry.

What you will learn

  • Interpret process state, memory and CPU columns in ps and top output
  • Send the right signal to stop, reload or inspect a process and understand what cannot be caught
  • Control foreground and background jobs and detach long-running work from a terminal
  • Write, enable and harden a systemd service unit and read its logs with journalctl
  • Express boot ordering with dependencies and targets, and schedule work with systemd timers

Everything running on a Linux machine is a process, and every process except the first was created by another one. That single fact explains a surprising amount: why a shell variable does not survive a script, why closing a terminal kills what it started, and why service management is really about parentage and lifetimes rather than about starting programs.

For its first few decades Linux managed that lifecycle with shell scripts in /etc/init.d. Every current distribution now boots with systemd, which supervises services directly, tracks their processes in control groups, collects their output into a structured journal and starts things in parallel according to declared dependencies. This guide covers the process concepts you need at the shell, then the systemd layer that sits on top of them.

The life of a process

A process is created when an existing one calls fork(), producing a near-identical copy, and the copy usually calls execve() to replace its memory image with a different program. The child inherits the parent's environment, open file descriptors, working directory and umask — that inheritance is what makes shell redirection work at all, as described in shell basics.

When a process ends it becomes a zombie until its parent calls wait() to collect the exit status. If the parent dies first, the orphan is re-parented to PID 1, which reaps it. PID 1 on a systemd host is systemd itself; inside a container it is whatever your image's entrypoint is, which is why containers that run a shell script as PID 1 accumulate zombies and ignore signals unless the script is written for the job.

The state column you see in tooling maps to what the kernel is waiting for:

StateMeaning
Rrunning or runnable — on a CPU or in the run queue
Sinterruptible sleep — waiting for input, a timer, a socket
Duninterruptible sleep — usually blocked on disk or NFS I/O; cannot be killed
Tstopped, typically by Ctrl-Z or SIGSTOP
Zzombie — finished, exit status not yet collected

A pile of processes in D is the classic signature of a storage problem, not a CPU problem. That distinction matters when you are reading load average, and it is explored further in logs and troubleshooting.

Looking at what is running

ps prints a snapshot. Two invocations cover most needs: the BSD-style ps aux for a wide overview, and ps -ef for the same information in the UNIX style. The forest view is the one worth memorising, because parentage is usually the question.

$ ps -eo pid,ppid,user,stat,pcpu,pmem,rss,etime,cmd --sort=-pcpu | head -5
  PID  PPID USER     STAT %CPU %MEM   RSS     ELAPSED CMD
 1842     1 postgres Ss    7.4  9.1 748120  3-04:11:52 /usr/lib/postgresql/16/bin/postgres
 2310  1842 postgres Ss    2.1  1.4 118004  3-04:11:50 postgres: checkpointer
 9921     1 www-data S     1.8  0.6  52208    02:14:07 php-fpm: pool www

$ ps -ef --forest | grep -A3 sshd

RSS is resident set size, the physical memory actually mapped in; VSZ (virtual size) counts address space and is routinely enormous and meaningless for a JVM or a database. Compare RSS between processes, not VSZ.

For a live view, top refreshes every few seconds; press 1 to split CPU usage per core, M to sort by memory, P by CPU, k to signal a process and f to choose columns. htop, if installed, adds a scrollable tree, mouse support and per-process signalling from a menu. pidstat 2 from the sysstat package samples per-process CPU, memory and I/O at a fixed interval, which is far more informative than a single snapshot when you are chasing an intermittent spike.

Finding a process by name is pgrep, and pgrep -a shows the full command line:

$ pgrep -a -u www-data php-fpm
9921 php-fpm: pool www
9922 php-fpm: pool www

Signals

A signal is a small asynchronous notification delivered to a process. Most can be caught by a handler, ignored, or left to the default action.

SignalNumberDefault actionTypical use
SIGHUP1terminateterminal hang-up; by convention, "reload configuration" for daemons
SIGINT2terminateCtrl-C
SIGQUIT3terminate + coreCtrl-\; a JVM prints a thread dump instead
SIGKILL9terminateuncatchable, last resort
SIGTERM15terminatethe polite default of kill and systemctl stop
SIGSTOP / SIGCONT19 / 18stop / continueuncatchable pause and resume
SIGUSR1 / SIGUSR210 / 12terminateapplication-defined, often log rotation or debug toggles
kill 9921                 # SIGTERM by default
kill -HUP $(pgrep -o nginx)
pkill -u bob -TERM        # every process belonging to bob
kill -QUIT <jvm-pid>      # thread dump into the JVM's stdout
kill -0 9921 && echo "still alive"   # signal 0 tests existence and permission only

Reaching for kill -9 first is the most common bad habit in this area. It denies the process any chance to flush a write buffer, commit a transaction or clean up a lock file, and it leaves stale sockets and PID files behind. Send SIGTERM, give it a few seconds, then escalate. If a process ignores SIGTERM repeatedly, that is a bug worth reporting rather than a reason to keep using -9.

SIGKILL also cannot help with a process stuck in D state: it is blocked inside the kernel, and the signal is only delivered when it returns to user space. Fix the storage or network mount it is waiting on.

Job control at the shell

Interactive shells group the processes of one command line into a job so you can suspend and resume them.

$ ./import-data.sh
^Z
[1]+  Stopped                 ./import-data.sh
$ bg                    # resume it in the background
$ jobs -l
[1]+ 21507 Running       ./import-data.sh &
$ fg %1                 # bring it back to the foreground

& starts a command in the background straight away; wait blocks until background jobs finish. None of this survives the session: when the terminal closes, SIGHUP goes to the job. nohup cmd & and disown %1 both dodge that, but on a modern system the honest answers are a terminal multiplexer (tmux, screen) for interactive work and a transient unit for anything unattended:

systemd-run --unit=import-once --collect /usr/local/bin/import-data.sh
journalctl -u import-once -f

Scheduling priority

The kernel's scheduler weights processes by their nice value, from -20 (most favourable) to 19 (least). Only root can lower a nice value.

nice -n 10 tar -czf /backup/site.tar.gz /var/www   # start a batch job politely
renice -n 5 -p 21507                               # adjust a running one
ionice -c 3 -p 21507                               # idle I/O class: only when nothing else wants the disk

Nice values are a hint, not a quota. When you need a real ceiling, use the control-group limits systemd exposes on any unit — CPUQuota=, MemoryMax=, IOWeight= — because those are enforced by the kernel rather than negotiated by the scheduler.

systemd: units and the unit file

systemd manages units, each described by an INI-style file. The suffix determines the type: .service for a daemon or one-shot job, .socket for socket activation, .timer for scheduled work, .mount and .automount for filesystems, .target for a grouping point, .path for filesystem triggers, .slice for resource partitioning.

Unit files come from three places, in increasing priority: /usr/lib/systemd/system/ (shipped by packages, never edit), /etc/systemd/system/ (yours), and drop-in directories such as /etc/systemd/system/nginx.service.d/override.conf for overriding a few settings of a packaged unit. systemctl edit nginx creates a drop-in for you; systemctl cat nginx shows the effective, merged configuration.

Here is a complete unit for a Java application, of the kind you would deploy alongside the packaging described in packaging and deployment:

# /etc/systemd/system/orders-api.service
[Unit]
Description=Orders API
Documentation=https://example.com/runbooks/orders-api
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=exec
User=orders
Group=orders
WorkingDirectory=/opt/orders-api
EnvironmentFile=/etc/orders-api/env
ExecStart=/usr/lib/jvm/java-21-openjdk-amd64/bin/java -XX:MaxRAMPercentage=70 -jar app.jar
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
TimeoutStopSec=30s

# Hardening: each line removes something the service does not need
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/orders-api
MemoryMax=2G

[Install]
WantedBy=multi-user.target

Three fields deserve comment. Type= tells systemd when the service counts as started: exec as soon as the binary is executed, simple as soon as it is forked, notify when the program signals readiness via sd_notify, forking for old-style daemons that background themselves, oneshot for a task that runs and exits. Getting this wrong is why a dependent unit sometimes starts before the service is really listening; notify is the accurate option when the program supports it.

Restart=on-failure restarts after a non-zero exit or a signal but not after a clean stop. Combine it with StartLimitIntervalSec= and StartLimitBurst= in [Unit] so a crash loop eventually gives up instead of hammering the machine.

WantedBy=multi-user.target is what systemctl enable acts on: enabling creates a symlink in multi-user.target.wants/, which is why enable and start are separate operations — one affects the next boot, the other affects now.

Driving systemd

sudo systemctl daemon-reload            # after adding or editing any unit file
sudo systemctl enable --now orders-api  # start it and start it at boot
systemctl status orders-api             # state, PID, cgroup, last log lines
systemctl is-active orders-api          # scriptable: exit code plus one word
sudo systemctl restart orders-api
sudo systemctl reload orders-api        # runs ExecReload, no downtime if supported
systemctl list-units --type=service --state=running
systemctl list-unit-files --state=enabled
systemctl show orders-api -p MemoryMax -p Restart

Two commands earn their keep when something is slow or broken at boot: systemd-analyze blame ranks units by start-up time, and systemd-analyze critical-chain shows the dependency path that actually determined the boot duration. systemctl list-dependencies orders-api prints the tree in the other direction.

systemctl mask is stronger than disable: it links the unit to /dev/null so nothing, not even another unit's dependency, can start it. Remember to unmask before wondering why a package's service will not run.

Reading a unit's logs

systemd captures each unit's standard output and error into the journal, tagged with the unit name, the PID, the boot id and priority. journalctl queries it.

journalctl -u orders-api -f                # follow, like tail -f
journalctl -u orders-api --since "1 hour ago" -p err
journalctl -b -1 -u orders-api             # previous boot
journalctl -u orders-api -o json-pretty | head -40
journalctl --disk-usage

Journal retention and whether it survives a reboot are configured in /etc/systemd/journald.conf; the storage question and the relationship with /var/log are covered in the troubleshooting guide.

Targets, dependencies and ordering

A target is a unit that groups others, replacing the old runlevels. multi-user.target is a fully booted system without a graphical session; graphical.target pulls that in and adds a display manager; rescue.target is single-user mode. systemctl get-default and set-default control which one boots.

Dependencies and ordering are deliberately separate concepts:

  • Requires= and Wants= pull another unit in — Requires fails your unit if the other fails, Wants does not.
  • After= and Before= only order units that are already being started; they pull nothing in.

A unit with After=postgresql.service but no Wants= will start after the database if the database is starting anyway, and immediately if it is not. Almost every "my service starts too early" report is a missing Wants= or the use of network.target (interfaces configured) where network-online.target (an address is actually usable) was meant.

Timers instead of cron

A timer unit activates another unit on a schedule. It needs two files: the work and the trigger.

# /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly database dump

[Service]
Type=oneshot
User=postgres
ExecStart=/usr/local/bin/db-backup.sh
# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run the nightly database dump

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target
sudo systemctl enable --now db-backup.timer
systemctl list-timers --all
systemd-analyze calendar "Mon..Fri 09:00"    # verify an expression before trusting it
sudo systemctl start db-backup.service       # run it once, now, to test

Persistent=true runs a missed job after the machine comes back up — the property cron lacks and the reason backups silently skip a day on laptops and spot instances. RandomizedDelaySec prevents a hundred machines hitting the same endpoint at 02:30:00. OnUnitActiveSec= expresses "every 15 minutes since the last run" for jobs whose duration varies. And because the work is an ordinary unit, its output lands in the journal instead of being emailed to a mailbox nobody reads.

Cron still exists and is fine for a handful of personal jobs; crontab -e edits your own table and /etc/cron.d/ holds system entries with a user column. Just remember that cron runs with a minimal environment and no login files, so scripts need absolute paths and an explicit PATH.

Common mistakes

  • Editing a packaged unit in /usr/lib/systemd/system. The next upgrade overwrites it. Use systemctl edit or a file in /etc/systemd/system.
  • Forgetting daemon-reload. systemd caches unit files; a change is invisible until you reload, and status will not tell you why.
  • Confusing enable with start. Enabling only affects boot. Use enable --now when you mean both.
  • Using Type=simple for a program that forks. systemd tracks the wrong PID, MAINPID is wrong, and reload and stop misbehave.
  • After=network.target for anything that binds a public address. Use network-online.target with a matching Wants=.
  • Killing with -9 by reflex. You lose the clean shutdown path the service already implements.
  • Relying on nice to protect a busy machine. Set CPUQuota= and MemoryMax= on the unit instead.

Key takeaways

  • Processes form a tree; children inherit environment and descriptors, orphans are adopted by PID 1, and a zombie is an uncollected exit status rather than a running program.
  • ps -eo with explicit columns, top/htop and pgrep answer most "what is running and why" questions; compare RSS, not VSZ.
  • Send SIGTERM and wait before SIGKILL; SIGHUP conventionally reloads configuration, and nothing reaches a process stuck in uninterruptible I/O.
  • A systemd unit file declares how to start a program, when it counts as started, how to restart it and what it is allowed to touch; drop-ins override packaged units safely.
  • Wants=/Requires= pull units in, After=/Before= only order them, and network-online.target is what network services actually need.
  • The journal holds every unit's output; journalctl -u … -p err --since is the fastest route to what went wrong.
  • Timers give scheduled work the same units, logging, dependencies and catch-up semantics as everything else systemd runs.

Sources and further reading

Frequently asked questions

Why does my service die when I close the SSH session?
A program started from a login shell belongs to that session. When the terminal disappears the kernel delivers SIGHUP to the foreground group, and most programs exit. Prefixing with nohup or backgrounding with an ampersand only papers over it. Anything meant to survive belongs in a systemd unit, or at minimum in systemd-run --user --scope, which places it outside the session cgroup.
What is the practical difference between SIGTERM and SIGKILL?
SIGTERM is a polite request: the process can catch it, flush buffers, close sockets, remove its PID file and exit cleanly. SIGKILL cannot be caught, blocked or ignored — the kernel destroys the process immediately, so open transactions are lost and temporary files remain. Always send SIGTERM first, wait, and reserve SIGKILL for a process that has genuinely stopped responding.
Why is a process shown as defunct or zombie, and how do I remove it?
A zombie is a finished process whose exit status has not yet been collected by its parent with wait(). It holds no memory or CPU, only an entry in the process table. You cannot kill it because it is already dead. Fix the parent — restart it or send it SIGCHLD — and the entries clear. If the parent exits, init adopts the children and reaps them.
Should I use a systemd timer or a cron job?
Timers integrate with the rest of systemd: the work runs as a unit, its output goes to the journal, dependencies and resource limits apply, Persistent=true catches up after downtime, and RandomizedDelaySec spreads load across a fleet. Cron is fewer files and is fine for a simple periodic script on a machine you manage by hand. On a server you also manage with units, prefer timers for consistency.