Linux Administration

grep, sed and awk: Text Processing Pipelines That Answer Real Questions

Intermediate9 min readUpdated

Three tools cover almost all command-line text processing: grep selects lines, sed edits them, awk computes over fields. This guide builds the shared regular-expression foundation, gives each tool the depth it deserves, adds the smaller utilities that glue pipelines together, and then works through realistic questions about a web server log from first match to a ranked report.

What you will learn

  • Write basic and extended regular expressions and know which tool accepts which dialect
  • Select, count and extract matching text with grep and its most useful options
  • Edit streams with sed using addresses, substitution flags and in-place editing
  • Compute per-field summaries with awk using patterns, actions, arrays and BEGIN/END blocks
  • Combine cut, sort, uniq, tr and join into pipelines that answer questions about log files

Command-line text processing rewards a small investment enormously. Every log file, CSV export, configuration file and command output on a Linux system is text, and three programs — grep, sed and awk — cover nearly everything you will want to do with it. They have been stable for decades, they are on every machine including the most stripped-down container, and once you can compose them you stop writing throwaway scripts for questions that a one-line pipeline answers.

The division of labour is worth stating up front, because reaching for the wrong tool is what makes these pipelines ugly. grep decides which lines. sed changes what a line says. awk treats a line as fields and can count, sum and remember. This guide builds the regular-expression foundation they share, then takes each in turn, and finishes by answering realistic questions about a web server access log.

The regular expressions underneath

A regular expression describes a set of strings. The syntax is mostly shared, with one significant fork.

ConstructMatches
.any single character
[abc], [a-z], [^0-9]one character from, or not from, a set
[[:digit:]], [[:space:]], [[:alpha:]]POSIX character classes, locale-aware
^, $start and end of the line
*zero or more of the preceding item
\+, \? (BRE) or +, ? (ERE)one or more; zero or one
\{2,5\} (BRE) or {2,5} (ERE)between two and five repetitions
\(…\) (BRE) or (…) (ERE)a capture group, referenced later as \1
`\(BRE) or|` (ERE)
\b, \<, \>word boundaries (GNU extension)

Basic regular expressions (BRE) are the default for grep and sed; extended ones (ERE) are selected with -E and are what awk uses natively. The only difference is which characters need a backslash, and ERE — where (a|b)+ means what it looks like — is the dialect to standardise on. Perl-compatible expressions with lookahead and non-greedy quantifiers are available as grep -P on GNU systems, but they are absent on BSD and macOS, so avoid them in portable scripts.

Two habits prevent most confusion. Always single-quote a pattern so the shell does not expand *, $ or \ before the tool sees it. And remember that these engines are greedy: .*" matches to the last quote on the line, not the first. Where that bites, match a negated class instead — [^"]*" stops at the first one. Building an expression interactively against sample text, in the regex tester or with grep --color, is far faster than reasoning about it in your head.

grep: choosing lines

grep -i 'timeout' app.log                  # case-insensitive
grep -n 'ERROR' app.log                    # prefix each match with its line number
grep -c 'ERROR' app.log                    # count matching lines, not matches
grep -v '^#' /etc/ssh/sshd_config          # invert: everything that is not a comment
grep -w 'id' schema.sql                    # whole word only: not "uuid" or "identity"
grep -F 'a.b.c' hosts.txt                  # fixed string: no regex interpretation
grep -r --include='*.java' 'TODO' src/     # recursive, filtered by name
grep -l 'password' /etc/myapp/*            # list only the file names that match
grep -o '[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' access.log   # print only the matched part

Four options carry more weight than the rest. -E gives you the readable dialect. -o prints only the matching text, which turns grep into an extractor rather than a filter. -C 3 (or -A/-B) prints context lines around each hit, which is how you read a stack trace found by its exception name. And -q prints nothing and simply sets the exit status, which is what a script wants:

if grep -q '^PasswordAuthentication no' /etc/ssh/sshd_config; then
  echo 'password login is disabled'
fi

grep -f patterns.txt data.txt reads many patterns from a file, which is a fast way to filter one list by another. And on a large file, prefixing with LC_ALL=C can be several times quicker, because the matcher then works on bytes rather than multibyte characters — safe whenever your data is ASCII.

sed: editing a stream

sed reads input line by line, applies a script to each line and prints the result. The overwhelmingly common command is substitution, s/pattern/replacement/flags.

sed 's/localhost/db.internal/' app.conf          # first occurrence per line
sed 's/localhost/db.internal/g' app.conf         # every occurrence
sed 's/DEBUG/TRACE/gi' app.log                   # g and i combine
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/' dates.txt   # reorder with groups
sed 's#/var/www/old#/var/www/new#g' nginx.conf   # any delimiter: # avoids escaping slashes
sed -E 's/(password=)[^ ]+/\1REDACTED/' app.log  # mask a secret before sharing a log

In the replacement, & stands for the whole match and \1 … \9 for capture groups. GNU sed adds \U, \L and \E for case conversion.

Every command can be limited to an address — a line number, a range or a pattern:

sed -n '10,20p' large.log                        # -n suppresses output; p prints: lines 10-20
sed '1d' data.csv                                # delete the header row
sed '/^$/d' notes.txt                            # delete blank lines
sed '/^#/d; /^$/d' /etc/nginx/nginx.conf         # comments and blanks: two commands, semicolon
sed -n '/BEGIN CERT/,/END CERT/p' bundle.pem     # everything between two patterns
sed '/\[database\]/,/^\[/ s/^port =.*/port = 5433/' app.ini   # substitute only inside a section
sed '$a\ExtraLine' file.txt                      # append after the last line
sed -n '/exception/I{=;p}' app.log               # print line numbers and lines, case-insensitive

That last group — a range address with a substitution — is the pattern that makes sed genuinely useful for configuration files, because it lets you change a key only within the section it belongs to.

In-place editing is -i, and the safe form supplies a backup suffix:

sed -i.bak -E 's/^#?(PermitRootLogin) .*/\1 no/' /etc/ssh/sshd_config
diff /etc/ssh/sshd_config.bak /etc/ssh/sshd_config

Always run the command without -i first and look at the output. And be aware that sed -i writes a new file and renames it over the old one, so the inode changes: ownership can be reset, hard links are broken, and a process holding the old file open keeps reading the old content. For files a running service reads, that is usually fine; for anything more delicate, edit a copy and install it explicitly.

sed has a wider command set — y for transliteration, N to pull in the next line, hold-space commands h, H, x, G for multi-line work. Those exist and occasionally save the day, but a multi-line sed program is write-only code. Once you need the hold space, you almost certainly want awk.

awk: fields, patterns and arithmetic

awk splits every input line into fields on whitespace (or a separator you choose) and runs a sequence of pattern { action } rules against it. $1 is the first field, $NF the last, $0 the whole line, NR the record number and NF the field count.

$ awk '{ print $1, $9 }' access.log | head -3
203.0.113.7 200
198.51.100.44 404
203.0.113.7 200

$ awk -F: '$3 >= 1000 { print $1, $6 }' /etc/passwd
alice /home/alice
bob /home/bob

Omit the pattern and the action runs on every line; omit the action and the default is print $0, which makes awk '/error/' a working substitute for grep.

BEGIN runs before any input, END after all of it — the natural place for headers and totals:

awk -F, 'NR > 1 { total += $3; n++ }
         END { printf "%d rows, average %.2f\n", n, total / n }' sales.csv

Arrays in awk are associative, indexed by strings, and this is where it leaves the other tools behind. Counting occurrences of anything is a two-line program:

awk '{ count[$1]++ } END { for (ip in count) print count[ip], ip }' access.log | sort -rn | head

A few more constructs cover most real use:

# Sum bytes served per status code
awk '{ bytes[$9] += $10 } END { for (s in bytes) printf "%s %.1f MB\n", s, bytes[s]/1048576 }' access.log

# Requests slower than one second, with their path
awk '$NF > 1.0 { printf "%6.2fs  %s\n", $NF, $7 }' timing.log

# Field separator that is a regular expression, and output separator
awk -F'[[:space:]]+' -v OFS='\t' '{ print $2, $5 }' report.txt

# Multi-line records: a blank line separates each block
awk -v RS='' -v FS='\n' '/state: failed/ { print $1 }' units.txt

# Pass a shell variable in properly, rather than splicing it into the program text
threshold=500
awk -v t="$threshold" '$3 > t { print }' metrics.tsv

-v is the correct way to get shell values into an awk program: splicing them into a single-quoted string invites quoting bugs and, if the value comes from outside, code injection. Note also that awk has real string functions — length(), substr(), split(), gsub(), match(), sprintf() — and gawk adds strftime() and systime() for date handling. An awk program long enough to need a file can be run with awk -f report.awk data.txt.

The supporting cast

ToolWhat it does
cutselect fields or byte ranges: cut -d: -f1,6 /etc/passwd, cut -c1-8
sortorder lines: -n numeric, -r reverse, -k2,2 by a field, -u unique, -h human sizes, -t separator
uniqcollapse adjacent duplicates: -c count, -d only duplicated, -u only unique
trtranslate or delete characters: tr 'A-Z' 'a-z', tr -d '\r', tr -s ' '
wccount lines, words and bytes: wc -l
head / tailfirst or last lines; tail -f follows a growing file
pastejoin files side by side into columns
joinrelational join of two sorted files on a common field
commcompare two sorted files: lines only in the first, only in the second, in both
column -talign whitespace-separated output into readable columns
nl, rev, fold, expandnumber, reverse, wrap and untabify lines

The one that trips everyone up is uniq: it only compares neighbouring lines, so sort | uniq -c is the idiom and uniq alone on unsorted input silently does almost nothing. The sort | uniq -c | sort -rn sequence — group, count, rank — is the workhorse of ad-hoc log analysis.

join deserves more use than it gets. Given two files sorted on the same key it performs the same operation as a database join, which is handy when correlating an export with a lookup table:

sort -t, -k1,1 users.csv > /tmp/u.csv
sort -t, -k1,1 orders.csv > /tmp/o.csv
join -t, -1 1 -2 1 /tmp/u.csv /tmp/o.csv

If you are reaching for join frequently on large files, that is a signal the data belongs in a database, where the ideas in SQL best practices apply properly.

Pipelines that answer questions

Take a standard combined-format access log, where field 1 is the client address, 7 the path, 9 the status and 10 the response size.

Which clients generated the most requests?

awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head -20

How many requests per status code, as a share of the total?

awk '{ c[$9]++; n++ } END { for (s in c) printf "%-5s %6d  %5.1f%%\n", s, c[s], 100*c[s]/n }' \
  access.log | sort -k2 -rn

Which URLs return 500, and how often?

awk '$9 == 500 { print $7 }' access.log | sort | uniq -c | sort -rn | head

How does traffic vary by hour? The timestamp field looks like [07/Sep/2026:14:32:01, so cut on the colon:

awk '{ split($4, t, ":"); hour[t[2]]++ }
     END { for (h = 0; h < 24; h++) printf "%02d %s\n", h, hour[sprintf("%02d", h)] + 0 }' access.log

Total bytes served per client, largest first, in megabytes:

awk '{ b[$1] += $10 } END { for (ip in b) printf "%10.2f MB  %s\n", b[ip]/1048576, ip }' \
  access.log | sort -rn | head

Which errors appeared today that were not there yesterday?

grep -o 'ERROR .*' today.log   | sort -u > /tmp/today
grep -o 'ERROR .*' yesterday.log | sort -u > /tmp/yesterday
comm -23 /tmp/today /tmp/yesterday

Follow a live log, keeping only slow requests, and redact tokens:

tail -F /var/log/app/access.log \
  | awk '$NF > 2.0' \
  | sed -E 's/(token=)[A-Za-z0-9._-]+/\1REDACTED/g'

Note tail -F rather than -f: the capital letter reopens the file after log rotation, which is precisely when you least want your monitoring to go quiet. The same distinction matters in the rotation discussion in logs and troubleshooting.

Build these incrementally. Run the first stage, look at ten lines, add the next stage, look again. Debugging a five-stage pipeline you wrote all at once is much harder than growing it one stage at a time, and inserting | head -5 while you work keeps the feedback instant on a large file.

Choosing the right tool

A rough decision rule:

  • Only need to know whether something is present, or to see the lines? grep.
  • Need to change text on each line, or delete or print a range? sed.
  • Need columns, arithmetic, grouping, formatting, or state across lines? awk.
  • Need it fast on a huge file? Put the most selective grep first so the later stages see less data, and prefix with LC_ALL=C.
  • Need structured data — JSON, YAML, XML? Use a parser. jq, yq and xmlstarlet exist precisely because regular expressions cannot reliably parse nested formats, and a pipeline that greps JSON will break the day the field order changes. The same applies to HTML.

Common mistakes

  • Unquoted patterns. The shell expands * and $ first; single-quote every pattern.
  • uniq without sort. Only adjacent duplicates are collapsed, so counts come out wrong.
  • Greedy .* swallowing the line. Use a negated class such as [^"]* to stop at the first delimiter.
  • grep | awk '{print $1}'. The grep is redundant: awk '/pattern/ { print $1 }' does both in one pass.
  • sed -i without a backup or a dry run. Print the result first, then edit with -i.bak.
  • Splicing shell variables into an awk program. Pass them with -v instead.
  • Parsing JSON or XML with regular expressions. Use jq or a real parser; nesting defeats the regex engine.
  • Forgetting that cut -f needs a single-character delimiter. For runs of whitespace use awk, or normalise first with tr -s ' '.

Key takeaways

  • grep selects lines, sed transforms them, awk computes over fields — matching the tool to the question keeps pipelines short.
  • Learn extended regular expressions (grep -E, sed -E, awk natively) and always single-quote patterns.
  • grep -o, -c, -q, -v and -C turn a filter into an extractor, a counter, a test and a context viewer.
  • sed addresses — line numbers, ranges and pattern ranges — let you edit exactly one section of a configuration file; -i.bak after a dry run keeps it safe.
  • awk's associative arrays with BEGIN/END blocks answer counting, summing and grouping questions in one pass; pass shell values in with -v.
  • sort | uniq -c | sort -rn is the ranking idiom; comm and join compare and correlate sorted files.
  • Build pipelines one stage at a time, put the most selective filter first, and switch to a real parser for JSON, XML and HTML.

Sources and further reading

Frequently asked questions

When should I use grep, when sed and when awk?
Use grep to decide which lines you care about, sed to transform lines with substitutions, and awk when the answer depends on splitting a line into fields, doing arithmetic or accumulating state across lines. If you find yourself chaining several sed commands with cut and a counter, the whole thing is usually one short awk program that is easier to read and faster.
Why does my regular expression need backslashes in one tool but not another?
Basic regular expressions, the default for grep and sed, treat plus, question mark, braces, parentheses and pipe as literal characters and require a backslash to give them their special meaning. Extended regular expressions, enabled with grep -E and sed -E and used natively by awk, reverse that. Use the extended dialect consistently and the expressions become far easier to read.
Is sed -i safe to run on a production configuration file?
It is safe if you test first and keep a backup. Run the command without -i and read the output, then use -i.bak so the original is preserved as file.bak. Note that GNU sed replaces the file by renaming a temporary copy, so ownership and any hard links can change. On a critical file, edit a copy, diff it, and install the result with the correct owner and mode.
How do I match a tab, or handle a file with Windows line endings?
GNU tools accept the escape sequences \t and \r inside regular expressions, and awk understands them in strings too. Files transferred from Windows carry a trailing carriage return that makes the last field of every line mismatch. Strip them once with sed -i 's/\r$//' file, or set the field separator so awk ignores them, rather than debugging invisible characters for an hour.