Linux Administration
grep, sed and awk: Text Processing Pipelines That Answer Real Questions
Intermediate9 min readUpdated
Three tools cover almost all command-line text processing: grep selects lines, sed edits them, awk computes over fields. This guide builds the shared regular-expression foundation, gives each tool the depth it deserves, adds the smaller utilities that glue pipelines together, and then works through realistic questions about a web server log from first match to a ranked report.
What you will learn
- Write basic and extended regular expressions and know which tool accepts which dialect
- Select, count and extract matching text with grep and its most useful options
- Edit streams with sed using addresses, substitution flags and in-place editing
- Compute per-field summaries with awk using patterns, actions, arrays and BEGIN/END blocks
- Combine cut, sort, uniq, tr and join into pipelines that answer questions about log files
Command-line text processing rewards a small investment enormously. Every log file, CSV export, configuration file and command output on a Linux system is text, and three programs — grep, sed and awk — cover nearly everything you will want to do with it. They have been stable for decades, they are on every machine including the most stripped-down container, and once you can compose them you stop writing throwaway scripts for questions that a one-line pipeline answers.
The division of labour is worth stating up front, because reaching for the wrong tool is what makes these pipelines ugly. grep decides which lines. sed changes what a line says. awk treats a line as fields and can count, sum and remember. This guide builds the regular-expression foundation they share, then takes each in turn, and finishes by answering realistic questions about a web server access log.
The regular expressions underneath
A regular expression describes a set of strings. The syntax is mostly shared, with one significant fork.
| Construct | Matches |
|---|---|
. | any single character |
[abc], [a-z], [^0-9] | one character from, or not from, a set |
[[:digit:]], [[:space:]], [[:alpha:]] | POSIX character classes, locale-aware |
^, $ | start and end of the line |
* | zero or more of the preceding item |
\+, \? (BRE) or +, ? (ERE) | one or more; zero or one |
\{2,5\} (BRE) or {2,5} (ERE) | between two and five repetitions |
\(…\) (BRE) or (…) (ERE) | a capture group, referenced later as \1 |
| `\ | (BRE) or|` (ERE) |
\b, \<, \> | word boundaries (GNU extension) |
Basic regular expressions (BRE) are the default for grep and sed; extended ones (ERE) are selected with -E and are what awk uses natively. The only difference is which characters need a backslash, and ERE — where (a|b)+ means what it looks like — is the dialect to standardise on. Perl-compatible expressions with lookahead and non-greedy quantifiers are available as grep -P on GNU systems, but they are absent on BSD and macOS, so avoid them in portable scripts.
Two habits prevent most confusion. Always single-quote a pattern so the shell does not expand *, $ or \ before the tool sees it. And remember that these engines are greedy: .*" matches to the last quote on the line, not the first. Where that bites, match a negated class instead — [^"]*" stops at the first one. Building an expression interactively against sample text, in the regex tester or with grep --color, is far faster than reasoning about it in your head.
grep: choosing lines
grep -i 'timeout' app.log # case-insensitive
grep -n 'ERROR' app.log # prefix each match with its line number
grep -c 'ERROR' app.log # count matching lines, not matches
grep -v '^#' /etc/ssh/sshd_config # invert: everything that is not a comment
grep -w 'id' schema.sql # whole word only: not "uuid" or "identity"
grep -F 'a.b.c' hosts.txt # fixed string: no regex interpretation
grep -r --include='*.java' 'TODO' src/ # recursive, filtered by name
grep -l 'password' /etc/myapp/* # list only the file names that match
grep -o '[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' access.log # print only the matched part
Four options carry more weight than the rest. -E gives you the readable dialect. -o prints only the matching text, which turns grep into an extractor rather than a filter. -C 3 (or -A/-B) prints context lines around each hit, which is how you read a stack trace found by its exception name. And -q prints nothing and simply sets the exit status, which is what a script wants:
if grep -q '^PasswordAuthentication no' /etc/ssh/sshd_config; then
echo 'password login is disabled'
fi
grep -f patterns.txt data.txt reads many patterns from a file, which is a fast way to filter one list by another. And on a large file, prefixing with LC_ALL=C can be several times quicker, because the matcher then works on bytes rather than multibyte characters — safe whenever your data is ASCII.
sed: editing a stream
sed reads input line by line, applies a script to each line and prints the result. The overwhelmingly common command is substitution, s/pattern/replacement/flags.
sed 's/localhost/db.internal/' app.conf # first occurrence per line
sed 's/localhost/db.internal/g' app.conf # every occurrence
sed 's/DEBUG/TRACE/gi' app.log # g and i combine
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/' dates.txt # reorder with groups
sed 's#/var/www/old#/var/www/new#g' nginx.conf # any delimiter: # avoids escaping slashes
sed -E 's/(password=)[^ ]+/\1REDACTED/' app.log # mask a secret before sharing a log
In the replacement, & stands for the whole match and \1 … \9 for capture groups. GNU sed adds \U, \L and \E for case conversion.
Every command can be limited to an address — a line number, a range or a pattern:
sed -n '10,20p' large.log # -n suppresses output; p prints: lines 10-20
sed '1d' data.csv # delete the header row
sed '/^$/d' notes.txt # delete blank lines
sed '/^#/d; /^$/d' /etc/nginx/nginx.conf # comments and blanks: two commands, semicolon
sed -n '/BEGIN CERT/,/END CERT/p' bundle.pem # everything between two patterns
sed '/\[database\]/,/^\[/ s/^port =.*/port = 5433/' app.ini # substitute only inside a section
sed '$a\ExtraLine' file.txt # append after the last line
sed -n '/exception/I{=;p}' app.log # print line numbers and lines, case-insensitive
That last group — a range address with a substitution — is the pattern that makes sed genuinely useful for configuration files, because it lets you change a key only within the section it belongs to.
In-place editing is -i, and the safe form supplies a backup suffix:
sed -i.bak -E 's/^#?(PermitRootLogin) .*/\1 no/' /etc/ssh/sshd_config
diff /etc/ssh/sshd_config.bak /etc/ssh/sshd_config
Always run the command without -i first and look at the output. And be aware that sed -i writes a new file and renames it over the old one, so the inode changes: ownership can be reset, hard links are broken, and a process holding the old file open keeps reading the old content. For files a running service reads, that is usually fine; for anything more delicate, edit a copy and install it explicitly.
sed has a wider command set — y for transliteration, N to pull in the next line, hold-space commands h, H, x, G for multi-line work. Those exist and occasionally save the day, but a multi-line sed program is write-only code. Once you need the hold space, you almost certainly want awk.
awk: fields, patterns and arithmetic
awk splits every input line into fields on whitespace (or a separator you choose) and runs a sequence of pattern { action } rules against it. $1 is the first field, $NF the last, $0 the whole line, NR the record number and NF the field count.
$ awk '{ print $1, $9 }' access.log | head -3
203.0.113.7 200
198.51.100.44 404
203.0.113.7 200
$ awk -F: '$3 >= 1000 { print $1, $6 }' /etc/passwd
alice /home/alice
bob /home/bob
Omit the pattern and the action runs on every line; omit the action and the default is print $0, which makes awk '/error/' a working substitute for grep.
BEGIN runs before any input, END after all of it — the natural place for headers and totals:
awk -F, 'NR > 1 { total += $3; n++ }
END { printf "%d rows, average %.2f\n", n, total / n }' sales.csv
Arrays in awk are associative, indexed by strings, and this is where it leaves the other tools behind. Counting occurrences of anything is a two-line program:
awk '{ count[$1]++ } END { for (ip in count) print count[ip], ip }' access.log | sort -rn | head
A few more constructs cover most real use:
# Sum bytes served per status code
awk '{ bytes[$9] += $10 } END { for (s in bytes) printf "%s %.1f MB\n", s, bytes[s]/1048576 }' access.log
# Requests slower than one second, with their path
awk '$NF > 1.0 { printf "%6.2fs %s\n", $NF, $7 }' timing.log
# Field separator that is a regular expression, and output separator
awk -F'[[:space:]]+' -v OFS='\t' '{ print $2, $5 }' report.txt
# Multi-line records: a blank line separates each block
awk -v RS='' -v FS='\n' '/state: failed/ { print $1 }' units.txt
# Pass a shell variable in properly, rather than splicing it into the program text
threshold=500
awk -v t="$threshold" '$3 > t { print }' metrics.tsv
-v is the correct way to get shell values into an awk program: splicing them into a single-quoted string invites quoting bugs and, if the value comes from outside, code injection. Note also that awk has real string functions — length(), substr(), split(), gsub(), match(), sprintf() — and gawk adds strftime() and systime() for date handling. An awk program long enough to need a file can be run with awk -f report.awk data.txt.
The supporting cast
| Tool | What it does |
|---|---|
cut | select fields or byte ranges: cut -d: -f1,6 /etc/passwd, cut -c1-8 |
sort | order lines: -n numeric, -r reverse, -k2,2 by a field, -u unique, -h human sizes, -t separator |
uniq | collapse adjacent duplicates: -c count, -d only duplicated, -u only unique |
tr | translate or delete characters: tr 'A-Z' 'a-z', tr -d '\r', tr -s ' ' |
wc | count lines, words and bytes: wc -l |
head / tail | first or last lines; tail -f follows a growing file |
paste | join files side by side into columns |
join | relational join of two sorted files on a common field |
comm | compare two sorted files: lines only in the first, only in the second, in both |
column -t | align whitespace-separated output into readable columns |
nl, rev, fold, expand | number, reverse, wrap and untabify lines |
The one that trips everyone up is uniq: it only compares neighbouring lines, so sort | uniq -c is the idiom and uniq alone on unsorted input silently does almost nothing. The sort | uniq -c | sort -rn sequence — group, count, rank — is the workhorse of ad-hoc log analysis.
join deserves more use than it gets. Given two files sorted on the same key it performs the same operation as a database join, which is handy when correlating an export with a lookup table:
sort -t, -k1,1 users.csv > /tmp/u.csv
sort -t, -k1,1 orders.csv > /tmp/o.csv
join -t, -1 1 -2 1 /tmp/u.csv /tmp/o.csv
If you are reaching for join frequently on large files, that is a signal the data belongs in a database, where the ideas in SQL best practices apply properly.
Pipelines that answer questions
Take a standard combined-format access log, where field 1 is the client address, 7 the path, 9 the status and 10 the response size.
Which clients generated the most requests?
awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head -20
How many requests per status code, as a share of the total?
awk '{ c[$9]++; n++ } END { for (s in c) printf "%-5s %6d %5.1f%%\n", s, c[s], 100*c[s]/n }' \
access.log | sort -k2 -rn
Which URLs return 500, and how often?
awk '$9 == 500 { print $7 }' access.log | sort | uniq -c | sort -rn | head
How does traffic vary by hour? The timestamp field looks like [07/Sep/2026:14:32:01, so cut on the colon:
awk '{ split($4, t, ":"); hour[t[2]]++ }
END { for (h = 0; h < 24; h++) printf "%02d %s\n", h, hour[sprintf("%02d", h)] + 0 }' access.log
Total bytes served per client, largest first, in megabytes:
awk '{ b[$1] += $10 } END { for (ip in b) printf "%10.2f MB %s\n", b[ip]/1048576, ip }' \
access.log | sort -rn | head
Which errors appeared today that were not there yesterday?
grep -o 'ERROR .*' today.log | sort -u > /tmp/today
grep -o 'ERROR .*' yesterday.log | sort -u > /tmp/yesterday
comm -23 /tmp/today /tmp/yesterday
Follow a live log, keeping only slow requests, and redact tokens:
tail -F /var/log/app/access.log \
| awk '$NF > 2.0' \
| sed -E 's/(token=)[A-Za-z0-9._-]+/\1REDACTED/g'
Note tail -F rather than -f: the capital letter reopens the file after log rotation, which is precisely when you least want your monitoring to go quiet. The same distinction matters in the rotation discussion in logs and troubleshooting.
Build these incrementally. Run the first stage, look at ten lines, add the next stage, look again. Debugging a five-stage pipeline you wrote all at once is much harder than growing it one stage at a time, and inserting | head -5 while you work keeps the feedback instant on a large file.
Choosing the right tool
A rough decision rule:
- Only need to know whether something is present, or to see the lines? grep.
- Need to change text on each line, or delete or print a range? sed.
- Need columns, arithmetic, grouping, formatting, or state across lines? awk.
- Need it fast on a huge file? Put the most selective
grepfirst so the later stages see less data, and prefix withLC_ALL=C. - Need structured data — JSON, YAML, XML? Use a parser.
jq,yqandxmlstarletexist precisely because regular expressions cannot reliably parse nested formats, and a pipeline that greps JSON will break the day the field order changes. The same applies to HTML.
Common mistakes
- Unquoted patterns. The shell expands
*and$first; single-quote every pattern. uniqwithoutsort. Only adjacent duplicates are collapsed, so counts come out wrong.- Greedy
.*swallowing the line. Use a negated class such as[^"]*to stop at the first delimiter. grep | awk '{print $1}'. Thegrepis redundant:awk '/pattern/ { print $1 }'does both in one pass.sed -iwithout a backup or a dry run. Print the result first, then edit with-i.bak.- Splicing shell variables into an awk program. Pass them with
-vinstead. - Parsing JSON or XML with regular expressions. Use
jqor a real parser; nesting defeats the regex engine. - Forgetting that
cut -fneeds a single-character delimiter. For runs of whitespace useawk, or normalise first withtr -s ' '.
Key takeaways
grepselects lines,sedtransforms them,awkcomputes over fields — matching the tool to the question keeps pipelines short.- Learn extended regular expressions (
grep -E,sed -E, awk natively) and always single-quote patterns. grep -o,-c,-q,-vand-Cturn a filter into an extractor, a counter, a test and a context viewer.sedaddresses — line numbers, ranges and pattern ranges — let you edit exactly one section of a configuration file;-i.bakafter a dry run keeps it safe.- awk's associative arrays with
BEGIN/ENDblocks answer counting, summing and grouping questions in one pass; pass shell values in with-v. sort | uniq -c | sort -rnis the ranking idiom;commandjoincompare and correlate sorted files.- Build pipelines one stage at a time, put the most selective filter first, and switch to a real parser for JSON, XML and HTML.
Sources and further reading
Frequently asked questions
- When should I use grep, when sed and when awk?
- Use grep to decide which lines you care about, sed to transform lines with substitutions, and awk when the answer depends on splitting a line into fields, doing arithmetic or accumulating state across lines. If you find yourself chaining several sed commands with cut and a counter, the whole thing is usually one short awk program that is easier to read and faster.
- Why does my regular expression need backslashes in one tool but not another?
- Basic regular expressions, the default for grep and sed, treat plus, question mark, braces, parentheses and pipe as literal characters and require a backslash to give them their special meaning. Extended regular expressions, enabled with grep -E and sed -E and used natively by awk, reverse that. Use the extended dialect consistently and the expressions become far easier to read.
- Is sed -i safe to run on a production configuration file?
- It is safe if you test first and keep a backup. Run the command without -i and read the output, then use -i.bak so the original is preserved as file.bak. Note that GNU sed replaces the file by renaming a temporary copy, so ownership and any hard links can change. On a critical file, edit a copy, diff it, and install the result with the correct owner and mode.
- How do I match a tab, or handle a file with Windows line endings?
- GNU tools accept the escape sequences \t and \r inside regular expressions, and awk understands them in strings too. Files transferred from Windows carry a trailing carriage return that makes the last field of every line mismatch. Strip them once with sed -i 's/\r$//' file, or set the field separator so awk ignores them, rather than debugging invisible characters for an hour.