Linux Text Processing

Why Text Processing Matters in DevOps

Linux systems expose a huge amount of information as plain text: logs, configuration files, command output, and status reports. That means DevOps engineers do not just read text; they filter it, search it, transform it, count it, and pipe it into other tools.

The commands in this lesson turn the shell into a lightweight data-processing environment.

Searching with grep

grep searches for lines that match a pattern.

grep error app.log
grep -i failed app.log
grep -n timeout app.log
grep -r nginx /etc

Useful flags:

• -i ignores case
• -n shows line numbers
• -r searches recursively

Basic Regular Expressions

grep supports regular expressions, which let you search by pattern instead of fixed text.

grep '^ERROR' app.log
grep '200$' access.log
grep '[0-9][0-9][0-9]' access.log

Examples:

• ^ERROR means lines starting with ERROR
• 200$ means lines ending with 200
• [0-9] matches a digit

Tip: Start simple with fixed text, then add regex only when needed.

Selecting Fields with cut

cut extracts parts of each line.

cut -d: -f1 /etc/passwd
cut -d' ' -f1 access.log

Here:

• -d sets the delimiter
• -f chooses the field number

For /etc/passwd, fields are separated by :. The first field is the username.

Sorting and Deduplicating

sort orders lines. uniq removes adjacent duplicate lines.

sort names.txt
sort access.log | uniq
sort access.log | uniq -c

uniq -c also counts repeated entries.

This is helpful when summarizing logs or finding repeated values.

Counting with wc

wc stands for word count, but it can count more than words.

wc app.log
wc -l app.log
wc -w notes.txt
wc -c payload.json

Useful flags:

• -l counts lines
• -w counts words
• -c counts bytes

Example use case:

grep ERROR app.log | wc -l

This counts how many error lines appear in the log.

Pattern-Based Processing with awk

awk is a powerful text-processing language. For beginners, think of it as a tool for working with columns and applying logic.

awk '{print $1}' access.log
awk '/ERROR/ {print $0}' app.log
awk -F: '{print $1, $7}' /etc/passwd

Key ideas:

• $1, $2, $3 refer to fields
• -F: sets the field separator to :
• /pattern/ { action } runs the action only on matching lines

A realistic example:

awk '$9 == 500 {print $1, $7}' access.log

In a typical web log, this might print client IPs and paths for HTTP 500 responses.

Editing Streams with sed

sed is a stream editor. It is often used for quick substitutions.

sed 's/http:/https:/g' urls.txt
sed 's/DEBUG/INFO/g' app.log

The s/old/new/g pattern means substitute all matches on each line.

In-Place Editing

You can modify a file directly with -i.

sed -i 's/listen 80/listen 8080/' nginx.conf

Be careful with in-place editing, especially on production configs.

Note: It is often smart to copy a config file before running sed -i on it.

Pipes and Redirection

A pipe sends the output of one command into another command.

cat access.log | grep 500 | wc -l

This works, but you can usually skip cat and write:

grep 500 access.log | wc -l

Redirection sends output to files.

grep ERROR app.log > errors.txt
grep WARN app.log >> errors.txt
command 2> error-output.txt

• > overwrites a file
• >> appends to a file
• 2> redirects standard error

A Real Log Analysis Workflow

Suppose you want to find the most common failing endpoint from a web log.

awk '$9 >= 500 {print $7}' access.log | sort | uniq -c | sort -nr

This pipeline does four things:

1. selects lines with server errors
2. prints the request path
3. counts duplicates
4. sorts from highest count to lowest

That is a strong example of why text processing matters. You are turning raw logs into actionable insight with one command chain.

When to Use Which Tool

• Use grep to find matching lines
• Use cut for simple field extraction
• Use awk for field logic and patterns
• Use sed for substitutions and quick edits
• Use sort, uniq, and wc for summarizing output

You do not need to master every advanced flag today. Learn the core use cases first, then build from there.

Test Your Understanding

Let's see how well you understood the concepts! These exercises will help reinforce what you just learned.

Exercise 1: Anchoring a grep search

Question: If you want only log lines that begin with ERROR, which grep pattern from the lesson should you use?

Exercise 2: Splitting fields with cut

Question: What does cut -d: -f1 /etc/passwd return?

Exercise 3: Counting repeated values

Scenario: You want to know which HTTP path appears most often among 500-level responses.

Question: What is the main job of sort | uniq -c in the tutorial pipeline after awk prints the request path?