Linux Text Processing

Why Text Processing Matters in DevOps

Linux systems expose a huge amount of information as plain text: logs, configuration files, command output, and status reports. That means DevOps engineers do not just read text; they filter it, search it, transform it, count it, and pipe it into other tools.

The commands in this lesson turn the shell into a lightweight data-processing environment.

Searching with grep

grep searches for lines that match a pattern.

grep error app.log
grep -i failed app.log
grep -n timeout app.log
grep -r nginx /etc

Useful flags:

• -i ignores case
• -n shows line numbers
• -r searches recursively

Basic Regular Expressions

grep supports regular expressions, which let you search by pattern instead of fixed text.

grep '^ERROR' app.log
grep '200$' access.log
grep '[0-9][0-9][0-9]' access.log

Examples:

• ^ERROR means lines starting with ERROR
• 200$ means lines ending with 200
• [0-9] matches a digit

Tip: Start simple with fixed text, then add regex only when needed.

Selecting Fields with cut

cut extracts parts of each line.

cut -d: -f1 /etc/passwd
cut -d' ' -f1 access.log

Here:

• -d sets the delimiter
• -f chooses the field number

For /etc/passwd, fields are separated by :. The first field is the username.

Sorting and Deduplicating

sort orders lines. uniq removes adjacent duplicate lines.

sort names.txt
sort access.log | uniq
sort access.log | uniq -c

uniq -c also counts repeated entries.

This is helpful when summarizing logs or finding repeated values.

Counting with wc

wc stands for word count, but it can count more than words.

wc app.log
wc -l app.log
wc -w notes.txt
wc -c payload.json

Useful flags:

• -l counts lines
• -w counts words
• -c counts bytes

Example use case:

grep ERROR app.log | wc -l

This counts how many error lines appear in the log.

Pattern-Based Processing with awk

awk is a powerful text-processing language. For beginners, think of it as a tool for working with columns and applying logic.

awk '{print $1}' access.log
awk '/ERROR/ {print $0}' app.log
awk -F: '{print $1, $7}' /etc/passwd

Key ideas:

• $1, $2, $3 refer to fields
• -F: sets the field separator to :
• /pattern/ { action } runs the action only on matching lines

A realistic example:

awk '$9 == 500 {print $1, $7}' access.log

In a typical web log, this might print client IPs and paths for HTTP 500 responses.

Editing Streams with sed

sed is a stream editor. It is often used for quick substitutions.

sed 's/http:/https:/g' urls.txt
sed 's/DEBUG/INFO/g' app.log

The s/old/new/g pattern means substitute all matches on each line.

In-Place Editing

You can modify a file directly with -i.

sed -i 's/listen 80/listen 8080/' nginx.conf

Be careful with in-place editing, especially on production configs.

Note: It is often smart to copy a config file before running sed -i on it.

Pipes and Redirection

A pipe sends the output of one command into another command.

cat access.log | grep 500 | wc -l

This works, but you can usually skip cat and write:

grep 500 access.log | wc -l

Redirection sends output to files.

grep ERROR app.log > errors.txt
grep WARN app.log >> errors.txt
command 2> error-output.txt

• > overwrites a file
• >> appends to a file
• 2> redirects standard error

A Real Log Analysis Workflow

Suppose you want to find the most common failing endpoint from a web log.

awk '$9 >= 500 {print $7}' access.log | sort | uniq -c | sort -nr

This pipeline does four things:

1. selects lines with server errors
2. prints the request path
3. counts duplicates
4. sorts from highest count to lowest

That is a strong example of why text processing matters. You are turning raw logs into actionable insight with one command chain.

When to Use Which Tool

• Use grep to find matching lines
• Use cut for simple field extraction
• Use awk for field logic and patterns
• Use sed for substitutions and quick edits
• Use sort, uniq, and wc for summarizing output

You do not need to master every advanced flag today. Learn the core use cases first, then build from there.

Test Your Understanding

Exercise 1: Anchoring a grep search

Question: If you want only log lines that begin with ERROR, which grep pattern from the lesson should you use?

Exercise 2: Splitting fields with cut

Question: What does cut -d: -f1 /etc/passwd return?

Exercise 3: Counting repeated values

Scenario: You want to know which HTTP path appears most often among 500-level responses.

Question: What is the main job of sort | uniq -c in the tutorial pipeline after awk prints the request path?

---

Key takeaways

• grep finds lines, sed edits them, awk works with fields. Choose by what you need, not habit.
• grep -r "text" . for a codebase and grep -c when you only need the count.
• awk '{print $3}' beats cut on whitespace, because it treats runs of spaces as one separator.
• sed -i edits in place: and BSD sed on macOS requires an argument, so sed -i '' there and sed -i on Linux.
• sort | uniq -c | sort -rn is the log-analysis idiom worth memorising.
• uniq only collapses adjacent duplicates, so it is useless without sorting first.
• 2>&1 merges stderr into stdout; without it, piping silently drops your error messages.

---

Frequently Asked Questions

When should I use grep, sed or awk?

grep selects lines matching a pattern and does nothing else. Reach for it when the question is "which lines". sed is a stream editor for substitution and deletion, so "change this to that" is its job, usually sed 's/old/new/g'. awk splits each line into fields and can do arithmetic and conditionals, so it wins whenever you care about columns, awk '$3 > 100 {print $1}'. If you find yourself building something with three pipes and a conditional, that is usually one awk program.

How do I replace text in a file from the command line?

sed -i 's/old/new/g' file.txt, -i edits in place, s substitutes, and g replaces every occurrence on each line rather than just the first. There is one portability trap: GNU sed on Linux accepts bare -i, while BSD sed on macOS requires a backup suffix, so the same command needs sed -i '' 's/old/new/g' there. Scripts that must run on both should use perl -pi -e 's/old/new/g', which behaves identically everywhere.

What is the difference between cut and awk for extracting columns?

cut -d' ' -f3 treats every single delimiter as a boundary, so two consecutive spaces produce an empty field, which makes it unreliable on command output aligned with variable whitespace. awk '{print $3}' treats any run of whitespace as one separator by default, which is almost always what you want for ps, df or log output. cut is the better tool for genuinely delimited data such as CSV or /etc/passwd, where the delimiter is exactly one character.

How do I count unique occurrences in a log file?

sort file | uniq -c | sort -rn | head. The first sort is mandatory, because uniq only collapses adjacent duplicates and silently does nothing otherwise, that is the most common mistake with it. uniq -c prefixes each line with its count, and sort -rn orders by that count descending. Combined with awk to pull out a field, it answers most log questions: awk '{print $1}' access.log | sort | uniq -c | sort -rn | head gives your top client IPs.

How do I search recursively through a directory?

grep -rn "pattern" ., r recurses, n adds line numbers so the output is actionable. Add -i for case-insensitive and --include="*.py" to limit by file type. grep -rl lists only the matching filenames, which is what you want when feeding results into another command. For large repositories, ripgrep (rg) is considerably faster and respects .gitignore by default, which usually removes more noise than any set of --exclude flags.

Why does my error message disappear when I pipe a command?

Because a pipe carries stdout only, and errors go to stderr. command | grep error therefore filters the normal output and lets the actual error print straight to your terminal, unfiltered. Redirect stderr into stdout first with command 2>&1 | grep error. The order matters: 2>&1 >file and >file 2>&1 do different things, because redirections are processed left to right. The second is the one that sends both to the file.