Linux Administration
Linux Logs and Troubleshooting: journalctl, Rotation, OOM, Load and strace
Advanced10 min readUpdated
Under pressure, most people troubleshoot by guessing and restarting. This guide replaces that with a method: establish what changed, read the right log, then measure one resource at a time. It covers journald and the files under /var/log, log rotation and retention, and the recurring failures — full disks, the OOM killer, high load with idle CPUs, hung processes and DNS — with the command that identifies each.
What you will learn
- Apply a repeatable triage sequence instead of restarting services and hoping
- Query the journal by unit, priority, boot and time, and know what still logs to /var/log
- Configure log rotation and journal retention so a disk never fills with history
- Distinguish CPU saturation, I/O wait, memory pressure and OOM kills from one another
- Inspect a running or hung process with lsof, strace, /proc and dmesg
Troubleshooting on Linux is a skill with a shape. The people who are quick at it are not remembering more commands than everyone else; they are following a sequence, ruling out one layer at a time, and reading the log before forming a theory. The people who are slow restart the service, see the symptom return, restart it again, and only look at the logs after an hour.
This guide is about that sequence. It covers where log output actually goes on a systemd host, how to interrogate it, how to keep it from filling the disk, and then the specific failures that recur on every server: no space, no memory, load without CPU, a process that has stopped responding, and a dependency that has gone away. It assumes familiarity with processes and systemd and with the diagnostic order in networking basics.
Triage before tools
Before typing anything, answer three questions. What is the actual symptom — a specific error, a timeout, a wrong result — as opposed to "the site is down"? When did it start, and does it correlate with a deployment, a configuration change, a certificate expiry, a scheduled job or a traffic increase? What is the blast radius — one endpoint, one host, one customer, everything?
Then work through the layers in a fixed order, stopping at the first thing that is wrong:
- Is the service running, and has it restarted recently?
systemctl status,systemctl show -p NRestarts. - What does its log say around the start time?
journalctl -u … --since. - Is a resource exhausted? Disk, inodes, memory, file descriptors, connections.
- Are its dependencies healthy? Database, cache, upstream API, DNS.
- Did anything change?
journalctl -u … --since, package history, configuration in version control.
Write down what you check and what you find as you go. Ten minutes into an incident nobody remembers whether they already looked at the disk, and a shared note is the difference between two people investigating and two people duplicating each other.
Where the output goes
On a systemd host there are two log worlds, and you need both.
The journal collects everything a unit writes to standard output and error, plus kernel messages and anything sent through the syslog API. It is a structured binary store: each entry carries fields such as _SYSTEMD_UNIT, _PID, _UID, PRIORITY, _BOOT_ID and MESSAGE, which is what makes precise filtering possible. It lives in /run/log/journal (volatile, lost at reboot) or /var/log/journal (persistent) depending on Storage= in /etc/systemd/journald.conf. If journalctl -b -1 reports no entries, persistent storage is off; create /var/log/journal and restart systemd-journald to turn it on.
Plain files under /var/log are written by programs that manage their own logging: /var/log/apache2/, /var/log/nginx/, /var/log/mysql/, /var/log/audit/audit.log. Distributions that still run rsyslog also receive a copy of the journal, producing /var/log/syslog and /var/log/messages, plus /var/log/auth.log or /var/log/secure for authentication events.
Knowing which is which saves time. A failing PHP page is in the FPM pool's error log and in the Apache virtual host's error log, not in the journal; a service that will not start is in the journal, not in any file the application controls.
Querying the journal
journalctl -u nginx.service -n 100 --no-pager
journalctl -u orders-api -f # follow
journalctl -u orders-api --since "2026-09-07 14:00" --until "2026-09-07 14:30"
journalctl -p err -b # this boot, error and worse
journalctl -b -1 -p warning # previous boot
journalctl -k --since "30 min ago" # kernel ring buffer, time-filtered
journalctl _UID=33 --since today # everything by one user id
journalctl _PID=1842
journalctl -u orders-api -o json-pretty | head -40
journalctl --list-boots
Priorities run from emerg (0) to debug (7), and -p err means "err and more severe". Time expressions accept "10 min ago", yesterday, today and absolute stamps. -o cat strips the metadata down to bare messages, which is what you want before piping into the pipelines from grep, sed and awk:
journalctl -u orders-api --since today -o cat \
| grep -oE 'status=[0-9]{3}' | sort | uniq -c | sort -rn
Two flags are worth knowing for incidents. journalctl -u a -u b --since … merges several units into one time-ordered stream, which is how you see a request arrive at the proxy and fail at the backend in the same view. And journalctl -f -u a -u b -p warning gives a live filtered feed during a deployment.
Non-root users see only their own entries unless they are in the systemd-journal group — worth arranging for anyone expected to debug without full sudo.
Rotation and retention
Unbounded logs fill disks, and a full disk takes down every service on the machine at once. Both log worlds need a limit.
For the journal, set the ceiling in /etc/systemd/journald.conf:
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=1G
MaxRetentionSec=30day
Compress=yes
sudo systemctl restart systemd-journald
journalctl --disk-usage
sudo journalctl --vacuum-size=1G # reclaim now
sudo journalctl --vacuum-time=14d
For files, logrotate runs from a systemd timer and processes drop-in configurations in /etc/logrotate.d/:
/var/log/myapp/*.log {
daily
rotate 14
missingok
notifempty
compress
delaycompress
create 0640 myapp adm
sharedscripts
postrotate
systemctl reload myapp.service > /dev/null 2>&1 || true
endscript
}
The mechanism matters. logrotate renames the current file and creates a new one, but a process that already has the old file open keeps writing to it — the renamed file grows while the new one stays empty. Two solutions exist: postrotate signals the service to reopen its files (what reload does above), or you use copytruncate, which copies the contents and truncates the original in place. copytruncate loses any lines written during the copy, so prefer the signal when the application supports it.
Test a configuration before trusting it:
sudo logrotate -d /etc/logrotate.d/myapp # debug: say what would happen
sudo logrotate -f /etc/logrotate.d/myapp # force a rotation now
cat /var/lib/logrotate/status | grep myapp
The first sixty seconds on a sick machine
A quick sweep that costs almost nothing and frequently ends the investigation:
uptime # load average: 1, 5, 15 minutes
free -h # available memory, swap in use
df -h; df -i # space and inodes
systemctl --failed # units that gave up
journalctl -p err -b --since "1 hour ago" | tail -50
dmesg -T --level=err,warn | tail -30 # kernel complaints, human timestamps
sudo ss -s # socket counts by state
vmstat 1 5 # r, b, si/so, us/sy/id/wa over five seconds
Read vmstat columns as a diagnosis in themselves: a large r means processes queued for CPU, a large b means blocked on I/O, non-zero si/so means active swapping, and a high wa in the CPU group means the processors are idle because they are waiting for storage.
Disk and inode exhaustion
df -h reports bytes and df -i reports inodes; a filesystem can be full in either dimension, and a partition at 100 per cent inodes with 40 per cent space free confuses people every time. Millions of small session, cache or mail-queue files are the usual cause.
When bytes are the problem, find the growth rather than the size — most of a system's disk is legitimately occupied, so what you want is what changed:
sudo du -x --max-depth=2 /var 2>/dev/null | sort -n | tail -15
sudo find /var/log -type f -size +200M -printf '%s\t%p\n' | sort -rn | head
sudo find / -xdev -type f -mtime -1 -size +100M -printf '%TF %s %p\n' 2>/dev/null | sort -rn | head
Reclaiming space safely means truncating rather than deleting anything a service holds open — sudo truncate -s 0 /var/log/myapp/huge.log returns the blocks immediately and leaves the descriptor valid. Then remove the cause: a debug log level left on after a deployment, a rotation rule that was never written, a core dump pattern writing gigabytes into the working directory, or a journal with no SystemMaxUse.
Keep a few per cent of headroom deliberately. Databases, package managers and the journal all behave badly at exactly zero bytes free, and recovering a machine with no space to write a temporary file is far harder than it sounds.
Memory pressure and the OOM killer
free -h has one column that matters: available. It estimates how much memory a new process could obtain, counting reclaimable page cache. Linux deliberately uses all spare memory as cache, so a low "free" figure with a healthy "available" figure is a system working correctly, not a problem.
When available memory really runs out and swap cannot absorb it, the kernel invokes the OOM killer, which selects a victim by a badness score weighted towards large, recently started, non-privileged processes, and kills it with an uncatchable signal. The evidence is in the kernel log:
$ sudo dmesg -T | grep -i -A2 'out of memory'
[Sun Sep 7 03:14:22 2026] Out of memory: Killed process 4821 (java) total-vm:6291456kB,
anon-rss:3874112kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:8192kB oom_score_adj:0
journalctl -k --since today | grep -i 'killed process'
systemctl status orders-api | grep -E 'Active|Main PID|status'
cat /sys/fs/cgroup/system.slice/orders-api.service/memory.peak
The signature is unmistakable: a service that vanishes with no shutdown messages in its own log, and a kernel line naming it. systemctl status reports the unit as failed with signal KILL.
The fixes are ordered by how well they work. Cap the offender with a unit limit — MemoryMax=2G and MemoryHigh=1600M in the service file — so the kernel throttles and kills within that cgroup instead of choosing a victim system-wide. Fix the application's own limits, such as a JVM heap larger than the machine or an FPM pool with too many workers. Add swap, which converts a hard kill into slow degradation and buys time to react. Adjust oom_score_adj for a process that must survive, such as the database on a database host. Adding more RAM without understanding which of these applies usually just moves the failure later.
Load, CPU and I/O
Load average counts processes that are runnable and those in uninterruptible sleep. That second group is why load is not a CPU metric on Linux.
uptime
nproc # compare load against this
top -b -n1 | head -15
pidstat -u -p ALL 2 3 # per-process CPU over three samples
iostat -xz 2 3 # per-device utilisation, queue depth, await
ps -eo pid,stat,wchan:24,cmd | awk '$2 ~ /^D/' # what the blocked processes are waiting on
Interpret the combination:
- Load high,
ushigh,walow: genuine CPU saturation. Find the process withpidstat; profile or scale out. - Load high,
wahigh, processes inD: storage or a network filesystem.iostat -xshowsawaitand%utilper device; a saturated disk or a stalled NFS mount produces exactly this. - Load high,
syhigh: kernel time — often an interrupt storm, a pathological number of context switches, or a program calling into the kernel far too often.vmstat 1showscsandin. - Load low, application slow: the bottleneck is elsewhere. A lock, a database query, an upstream service. Application metrics answer this, not host metrics — which is the argument for the instrumentation described in actuator and observability.
Looking inside one process
When a specific process is misbehaving, four tools reveal what it is doing without a debugger.
sudo lsof -p 4821 | head -30 # open files, sockets, libraries
sudo lsof -i :8080 # who holds a port
sudo lsof /var/lib/mysql # who is using a path
ls -l /proc/4821/fd | wc -l # file descriptor count
cat /proc/4821/limits | grep 'open files' # against the limit
cat /proc/4821/status | grep -E 'State|Threads|VmRSS'
sudo cat /proc/4821/stack # kernel stack of a hung task
Descriptor exhaustion is a common and confusing failure: the application logs "too many open files", accepts no new connections, and looks healthy in every host metric. Compare the count against the limit, then raise it in the unit with LimitNOFILE= rather than in a shell profile, which a service never reads.
strace shows the system calls a process makes. Attach to a running process, and prefer the summary first because raw output is overwhelming:
sudo strace -c -p 4821 # counts and time per syscall; Ctrl-C to report
sudo strace -f -e trace=openat,connect -p 4821
sudo strace -f -T -o /tmp/trace.txt -p 4821
A summary dominated by futex suggests lock contention; one dominated by openat returning ENOENT usually means a search path is wrong; repeated connect calls returning ECONNREFUSED or ETIMEDOUT name the dependency that has gone away. Attaching pauses the target briefly and slows it noticeably, so be careful on a busy production process and prefer a short capture. ltrace does the same for library calls; perf top samples where CPU time is actually spent.
Kernel messages
dmesg reads the kernel ring buffer, where hardware and driver events land before any log daemon sees them. dmesg -T adds human-readable timestamps, and dmesg -w follows.
Things worth recognising: EXT4-fs error and I/O error, dev sda mean failing storage — check smartctl -a /dev/sda immediately. nf_conntrack: table full, dropping packet means the connection-tracking table is exhausted, which presents as random network failures under load. TCP: request_sock_TCP: Possible SYN flooding means the accept backlog overflowed. segfault at … ip … records a crashing binary. Under memory pressure and oom-kill are the memory story above.
Because the buffer is a fixed size, old messages are overwritten. journalctl -k -b -1 gets kernel messages from the previous boot when the ring buffer no longer has them, which is essential after an unexplained reboot.
Checking a service end to end
A structured pass over one service, from the outside in:
systemctl status orders-api # running? restarts? since when?
systemctl show orders-api -p NRestarts -p ExecMainStartTimestamp
journalctl -u orders-api -p warning --since "1 hour ago"
sudo ss -ltnp | grep 8080 # listening, and on what address
curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' http://127.0.0.1:8080/healthz
getent hosts db.internal && nc -vz db.internal 5432 # dependency reachable
sudo lsof -p "$(systemctl show -p MainPID --value orders-api)" | wc -l
journalctl -u orders-api --since "1 hour ago" -o cat | grep -c 'ERROR'
Each line rules something out. A rising NRestarts says the unit is crash-looping and the interesting log is just before each restart. A healthy local curl with failing external requests moves the investigation to the proxy or the firewall. A dependency that does not resolve or does not accept a connection ends the search immediately.
After the incident
Two habits pay for themselves. First, capture evidence before restarting: a thread dump, the last few hundred journal lines, ps output, free, df. A restart usually clears the symptom and destroys the explanation, and the next occurrence is then equally mysterious.
Second, write a short blameless note: timeline, symptom, root cause, what made it hard to diagnose, and one concrete change. The most valuable outcome of an incident is often not the fix but the missing alert, the log line that should have existed, or the retention setting that meant yesterday's evidence was already gone.
Common mistakes
- Restarting first. You lose the evidence and learn nothing; the fault returns at 03:00.
- Reading only the last log line. The interesting entry is usually the first error in the sequence, before the cascade of consequences.
- Deleting a large open log file. No space is returned. Truncate, then fix rotation.
- Treating load average as CPU usage. Check
waand theD-state processes before concluding you need more cores. - Panicking at low "free" memory. Read the
availablecolumn; cache is doing its job. - Leaving debug logging on after an investigation. It fills the disk quietly a week later.
- Not enabling persistent journal storage. Everything about the failure that caused the reboot is gone.
- Running
strace -fon a busy production process for minutes. The slowdown becomes the new incident.
Key takeaways
- Follow a fixed order — symptom, timing, blast radius, then service, log, resource, dependency, change — and record what you ruled out.
- The journal holds unit and kernel output; applications that manage their own files still log under
/var/log. Check both. journalctl -u … -p err --since, merged across units and with-o catfor piping, answers most log questions.- Bound both log stores:
SystemMaxUsefor the journal, alogrotaterule with a reload inpostrotatefor files. - Distinguish full bytes from full inodes, and truncate open files rather than deleting them.
- An OOM kill is visible in
dmesg; contain it with cgroup limits on the unit rather than by adding RAM blindly. - High load with idle CPUs and processes in
Dstate is a storage or network problem —iostat -xandwchanwill name it. lsof,/proc/<pid>/,strace -canddmesg -Texplain what one process is doing; capture evidence before you restart anything.
Sources and further reading
Frequently asked questions
- Where do logs actually live on a modern system — the journal or /var/log?
- Both. Anything started by systemd sends its standard output and error to the journal, which is a binary, indexed store read with journalctl. Applications that open their own log files still write plain text under /var/log, typically web servers, databases and mail systems. Many distributions also forward the journal to rsyslog, so the same message appears in /var/log/syslog. Check both before concluding a message does not exist.
- Why does my disk fill up again the day after I delete a large log file?
- Deleting a file that a process still has open frees no space until the process closes it, and the service keeps writing to the same descriptor. The correct operations are to truncate the file in place, then fix the underlying cause with log rotation and a retention limit. If the growth is in the journal, set SystemMaxUse in journald.conf and reclaim immediately with journalctl --vacuum-size.
- The load average is 20 but top shows the CPUs mostly idle. What does that mean?
- On Linux the load average counts processes in uninterruptible sleep as well as runnable ones, so blocked disk or network I/O inflates it without using CPU. Check the wa column in top or iostat, look for processes in D state, and measure device latency with iostat -x. A load of 20 with 90 per cent idle CPU is a storage or network filesystem problem, not a shortage of processing power.
- How do I tell whether the kernel killed my process or it exited on its own?
- Look for an Out of memory line in dmesg or journalctl -k around the time it disappeared; the kernel prints the chosen victim, its score and a table of memory consumers. systemctl status also reports the exit reason, showing signal 9 for an OOM kill of a unit's main process. A clean non-zero exit code with no kernel message means the application decided to stop, so its own log holds the answer.