Back to the shell¶

Exercise back-to-the-shell-annex.pdf · PDF

A new unit, and a few days since your last command. You open a terminal and the cursor blinks at you. How did you make a folder again? Was it mkdir -p or mkdir -r? Don't panic: it is not forgotten. It is in the pantry, not on the countertop. Ten minutes of typing and it comes back.

This unit is about administering an operating system: creating users, giving permissions, watching processes. Almost all of it happens in a terminal. So before we start, we sharpen the tool. And we go one step further than in Unit 4: today you stop just moving files around and start asking the system questions. "How many people work in dev at Tramuntana Apps?" One line of shell answers it.

What you will learn

Rebuild the basic commands for moving around and handling files
Name the three streams of a command (stdin, stdout, stderr) and send each one to a file
Build a pipeline of small commands to answer a question about a file
Create a variable and predict what double quotes, single quotes and $( ) do
Select files with globbing patterns, and say who expands them
Why > does not save "everything you see on the screen"

Warm-up: rebuild your shell¶

Everything today happens in deb-lab, in a terminal, as your normal user. No sudo. Try each mission without looking at your notes first.

Mission	One command that works
Where am I?	`pwd`
Create `tramuntana/dev/api` in your home, in one go	`mkdir -p ~/tramuntana/dev/api`
Go into `api`, then back home the shortest way	`cd tramuntana/dev/api`, then `cd`
Create an empty file	`touch ~/tramuntana/notes.txt`
Copy it	`cp tramuntana/notes.txt tramuntana/dev/notes-dev.txt`
Rename it	`mv tramuntana/notes.txt tramuntana/todo.txt`
List in long format, hidden files too	`ls -la tramuntana`
Find every `.conf` file under `/etc`	`find /etc -name "*.conf"`
Show the line of `/etc/passwd` with your user	`grep student /etc/passwd`
Delete the empty folder `api`	`rmdir tramuntana/dev/api`

The usual holes:

mkdir without -p fails, because dev does not exist yet. -p creates every missing parent.
cd alone goes home. cd .. goes up one level. cd - goes back to where you were before.
The find mission prints annoying Permission denied lines mixed with the results. Keep that annoyance in mind: you are about to make it disappear.

Keep the folder

Don't delete ~/tramuntana. The staff file you create in it today comes back in the next sessions.

Three streams¶

Run find /etc -name "*.conf" again. Results and errors arrive mixed on the same screen. Is Permission denied a result? No. So why is it in the same place?

It only looks like the same place. Every command has three pipes connected to it:

A stream is a channel through which a command receives or sends text. Every command has three, numbered 0, 1 and 2.

Number	Name	What travels through it	Default
0	stdin (standard input)	What the command reads	The keyboard
1	stdout (standard output)	The results	The screen
2	stderr (standard error)	The complaints and error messages	The screen

flowchart LR
    KB["<b>Keyboard</b>"] -->|"0 · stdin"| CMD["<b>Command</b><br>e.g. find, ls, sort"]
    CMD -->|"1 · stdout<br>results"| SCR["<b>Screen</b>"]
    CMD -->|"2 · stderr<br>complaints"| SCR

Both outputs land on the screen, so they look like one. But they are two separate pipes. And the shell can re-plumb each one before the command starts.

Redirection¶

Redirection means telling the shell where each stream of a command should go (or come from) instead of the default.

You write	The shell does
`cmd > file`	stdout goes to the file. The file is emptied first.
`cmd >> file`	stdout is added at the end of the file.
`cmd 2> file`	stderr goes to the file.
`cmd 2> /dev/null`	stderr goes to the bin. `/dev/null` swallows everything you send it.
`cmd > file 2>&1`	Both streams go into the same file ("2 goes where 1 goes").
`cmd < file`	stdin comes from the file instead of the keyboard.

Try it. First, create the staff file of Tramuntana Apps. Type cat > staff.csv, press Enter, type the 13 lines from the Exercise sheet, press Enter, then Ctrl + D on the empty line:

user,dept,role,year
marta,dev,backend,2019
joan,design,ui,2020
aina,admin,office,2018
pau,dev,intern,2026
biel,dev,mobile,2021
neus,dev,frontend,2022
toni,dev,devops,2020
laia,design,ux,2023
xisca,design,brand,2021
tomeu,admin,finance,2019
carme,dev,qa,2024
miquel,dev,mobile,2025

Check it: wc -l staff.csv must print 13 staff.csv. Now predict each line before you press Enter:

ls staff.csv nope.txt
ls staff.csv nope.txt > out.txt
ls staff.csv nope.txt > out.txt 2> err.txt

Line 1 prints the file name and an error.
Line 2 still prints ls: cannot access 'nope.txt': No such file or directory on the screen. That was stderr, and you only moved pipe 1. out.txt contains just staff.csv.
Line 3 prints nothing. Each pipe has its own file.

And the promise: find /etc -name "*.conf" 2> /dev/null gives the clean list, with no Permission denied lines.

Careful: > saves only stdout, and it never asks

> does not save "what I see on the screen". It saves stdout only; errors travel on pipe 2. And > empties the file before the command runs, without asking. sort staff.csv > staff.csv leaves you with an empty staff.csv. Use >> to add, and a different file name to be safe.

Two small traps:

2> err.txt is not 2 > err.txt. With the space, 2 becomes an argument of the command.
wc -l staff.csv prints 13 staff.csv, but wc -l < staff.csv prints just 13. With <, the command reads a stream and never learns the file's name.

Pipes: the garden hose¶

A bit of history¶

In 1964, at Bell Labs, Doug McIlroy wrote an internal memo: programs should connect "like garden hose — screw in another segment when it becomes necessary to massage data in another way". Nobody built it for nine years. In 1973, Ken Thompson added the pipe to Unix (Version 3) in one night. The next morning the team rewrote its tools so that every one read stdin and wrote stdout. The character has not changed since: |.

A pipe (|) connects the stdout of one command to the stdin of the next. A chain of commands joined by pipes is a pipeline.

flowchart LR
    F["<b>staff.csv</b>"] --> A["<b>cut</b><br>keep column 2"]
    A -->|"| pipe"| B["<b>sort</b><br>put equal lines together"]
    B -->|"| pipe"| C["<b>uniq -c</b><br>count each group"]
    C --> S["<b>Screen</b>"]

That is the garden-hose idea, and we keep it all page: each tool is a segment of hose that does one small thing. Screw them together and the water (the text) comes out the other end changed. No file in between.

The toolbox¶

Tool	What it does
`grep text`	Keeps the lines that contain `text`
`cut -d, -f2`	Keeps column 2, with columns separated by commas
`sort`	Orders lines (`-n` as numbers, `-r` reversed, `-t, -k4` by column 4 with `,` as separator)
`uniq -c`	Joins neighbour lines that are equal and counts them
`wc -l`	Counts lines
`head -3` / `tail -3`	First / last three lines (`tail -n +2` means "from line 2 to the end")
`tee file`	Saves a copy to a file and passes the text on

Build it one segment at a time¶

"How many people per department?" Run each line and watch the data change shape:

cut -d, -f2 staff.csv
cut -d, -f2 staff.csv | sort
cut -d, -f2 staff.csv | sort | uniq -c

      2 admin
      1 dept
      3 design
      7 dev

Who is dept? The header line. Fix it by skipping line 1, and sort the counts, biggest first:

tail -n +2 staff.csv | cut -d, -f2 | sort | uniq -c | sort -nr

      7 dev
      3 design
      2 admin

Never write a pipeline in one go

Write one command, look. Add a pipe, look. When it breaks, you know which segment of hose leaks.

Careful: uniq only joins neighbours

uniq does not remove "all repeated lines". It only joins equal lines that touch. Without sort first, dev appears several times. That is why sort | uniq -c almost always go together.

More questions you can answer the same way:

grep -c ",dev," staff.csv                                # how many in dev: 7
tail -n +2 staff.csv | sort -t, -k4 -n | head -3         # the three who joined first
tail -n +2 staff.csv | sort -t, -k4 -n | tail -1 | cut -d, -f1   # the last one: pau

Why ",dev," with commas and not just dev? Because grep dev would also match a role like devops or devrel. Searching with the separators is the safe version. And sort -k4 without -t, sorts by the fourth word, not the fourth column: you must give the separator.

/etc/passwd is a table too, like staff.csv, with : instead of ,. Column 7 is each account's shell:

cut -d: -f7 /etc/passwd | sort | uniq -c | sort -nr

Most accounts use /usr/sbin/nologin: most accounts on a Linux system are not people. Hold that thought: users and groups start very soon.

Variables and quotes¶

A variable is a label on a value. You create it with name=value and read it with $name.

name=marta
echo "Hello $name"     # Hello marta
echo 'Hello $name'     # Hello $name
today=$(date +%F)      # $( ) runs a command and keeps its output
mkdir backup-$today    # a folder like backup-2027-01-07

You write	What the shell does
`"..."` double quotes	Still replaces `$name` with its value
`'...'` single quotes	Touches nothing: prints `$name` literally
`$( command )`	Command substitution: runs the command and puts its output there

You can combine them:

n=$(tail -n +2 staff.csv | wc -l)
echo "Tramuntana has $n people"     # Tramuntana has 12 people

Some variables already exist: $USER (your user name), $HOME (your home folder), $PATH (the folders where the shell looks for commands).

Careful: in the shell a space is never decoration

name = marta is not the same as name=marta. With spaces, Bash looks for a command called name and answers name: command not found. A space separates words. No spaces around =.

Globbing¶

A glob is a pattern for file names: * means any characters, ? exactly one character, [0-9] one character from the set.

touch report-{01..12}.txt          # creates report-01.txt ... report-12.txt
ls report-0?.txt                   # 01 to 09
ls report-1*.txt                   # 10, 11, 12
ls report-[0-9][02468].txt         # the six even ones

The key idea: the shell replaces the pattern with the matching names before the command starts. ls never sees the star; it receives a list of files. In rm *.txt, it is the shell, not rm, that decides which files die.

What if nothing matches? echo *.jpg prints *.jpg literally: Bash leaves the pattern as it is. That is why ls *.jpg complains about a file called *.jpg.

Braces are not globs

report-{01..12}.txt is brace expansion, not globbing. Braces generate text whether the files exist or not. Globs match files that already exist.

Help without the internet¶

You don't need a browser to find out how a command works:

Command	What it gives you
`man ls`	The manual page: search with `/word`, next match with `n`, quit with `q`
`ls --help`	A short summary of the options
`type cd`	What a name is: `cd is a shell builtin`, `ls is /usr/bin/ls`
`help cd`	Help for the commands built into Bash, which have no man page
`man -k word`	Searches the titles of all manuals

A builtin is a command that lives inside Bash itself, not as a file in /usr/bin. cd and history are builtins. So "no man page" does not mean "the command does not exist": try help.

And the history saves you typing: Up arrow for the previous command, Ctrl + R to search an old one by typing a piece of it, history | tail for the last ten, !! to repeat the last command.

Key takeaways¶

Every command has three streams: stdin (0), stdout (1) and stderr (2). Results and errors only look like one stream on the screen.
Redirection re-plumbs them: > empties and writes, >> appends, 2> moves errors, 2> /dev/null throws them away, < reads from a file.
A pipe | screws one command's stdout into the next one's stdin. Build pipelines one segment at a time; sort before uniq -c.
A variable is a label on a value: no spaces around =. Double quotes replace $name, single quotes don't, $( ) captures a command's output.
A glob (*, ?, [ ]) is expanded by the shell before the command starts.
man, --help, type and help answer most questions without the internet.

Quick check¶

1. find /etc -name '*.conf' > list.txt still prints Permission denied on the screen. Why?

Errors travel on stderr, which > does not move
The find command ignores every redirection
The file list.txt is already full of data
Redirecting into a file needs sudo to work

> moves only stdout (stream 1). Error messages travel on stderr (stream 2), which still goes to the screen. Add 2> /dev/null to throw them away, or 2> err.txt to keep them.

2. notes.txt has 5 lines. After running echo hi > notes.txt, how many lines does it have?

1 line
6 lines
5 lines
0 lines

A single > empties the file first and then writes stdout into it, so only hi is left. To add a sixth line you need >>, which appends.

3. cut -d, -f2 staff.csv | uniq -c shows dev several times. What is missing?

A sort before uniq -c
A wc -l at the very end
The -r option in cut
A grep dev at the start

uniq only joins equal lines that are neighbours. If the dev lines are scattered, each group is counted separately. sort puts equal lines together first.

4. What does echo 'I am $USER' print?

I am $USER
I am student
I am
An error message

Inside single quotes the shell replaces nothing, so $USER is printed literally. With double quotes, "I am $USER" would print your user name.

5. In rm *.txt, who turns *.txt into a list of files?

The shell, before rm starts
The rm command itself
The kernel of the system
The file system on the disk

Globbing is done by the shell. It replaces the pattern with the matching names, and rm only receives the finished list. That is why you should check a glob with ls or echo before using it with rm.

6. You type name = marta and Bash answers name: command not found. Why?

The spaces made Bash read name as a command
The variable name is reserved by the system
Variable values must always be in quotes
Variables can only be created with sudo

In the shell, a space separates words, and the first word is the command. A variable assignment must be written with no spaces around =: name=marta.

7. You want to know which three Tramuntana Apps employees joined most recently (column 4 is the year). Pick the pipeline that works.

tail -n +2 staff.csv | sort -t, -k4 -n | tail -3
tail -n +2 staff.csv | sort -k4 -n | head -3
tail -n +2 staff.csv | uniq -c | sort -n | tail -3
tail -n +2 staff.csv | cut -d, -f4 | head -3

tail -n +2 skips the header, sort -t, -k4 -n sorts numerically by column 4 using the comma as separator, and tail -3 keeps the three largest years. Without -t,, sort uses words instead of columns; cut -f4 alone would lose the names.

Your shell is back in shape. Next: the people behind the prompt. Who is student, and how does the system tell marta from joan? See Users and groups on Linux.