Network troubleshooting¶

Practical 3.9 network-troubleshooting-annex.pdf · PDF

"The Internet is down." You have said it. Your family has said it. But what does it actually mean? Is the cable loose? Did the PC lose its address? Is the router dead? Is it only one website? Each of those has a different fix, and guessing wastes hours.

Professionals do not guess. They follow the same short method every time, from the bottom up, and the network tells them where it hurts. By the end of this page you will have that method, and you will have used it on seven broken networks.

What you will learn

Apply a fixed bottom-up method, the six-rung ladder, instead of trying things at random
Pick the command that tests each rung: ip -br link, ip -br addr, ip route, ping, traceroute, dig, nc, ss
Read the error messages that tell rungs apart: Network is unreachable, 100% packet loss, no servers could be reached, Connection refused, timed out
Locate a fault on a host, on a router or on a server, and write a ticket
Why "ping works" does not mean "the network is fine"

The day Facebook disappeared¶

On 4 October 2021, Facebook, Instagram and WhatsApp vanished from the Internet for about six hours. What users saw was a name problem: facebook.com could not be resolved. Phones everywhere said, in effect, "I don't know where that is".

But DNS was only the symptom. During maintenance, a command had disconnected Facebook's data centres from each other. Its routers then stopped announcing to the rest of the Internet the routes that lead to its DNS servers. The fault was at the routing level; the error message appeared at the name level.

That is the lesson of this page: a network fails at the bottom and complains at the top. So you never start where the complaint is. You start at the bottom and climb.

The ladder¶

Troubleshooting is finding the cause of a fault with a method, proving it with evidence, and fixing it.

Our method is a ladder with six rungs. Each rung is one question, with one command that answers it. Every rung stands on the one below: you cannot reach a server's port if you have no address, and you cannot have an address if the cable is unplugged.

Rung	Question	Command	If it fails, look at
1 · Link	Is my interface up?	`ip -br link`	Cable, port, Wi-Fi, interface disabled
2 · Address	Do I have an address in the right network?	`ip -br addr`	DHCP server, static address, mask
3 · Gateway	Do I know the way out, and does the router answer?	`ip route` · `ping` the gateway	Default route, VLAN, the router's LAN side
4 · Path	Does an address beyond the router answer?	`ping` · `traceroute -n`	Routing, NAT, firewall on the router
5 · Names	Do names become addresses?	`dig name` · `dig @server name`	DNS server address, the DNS server itself
6 · Service	Is the port open at the other end?	`nc -zv host port` · `curl`	The service (`ss -tuln` on the server), a firewall rule for that port

flowchart TB
    R1["<b>1 · Link</b><br>ip -br link"] -->|"UP"| R2["<b>2 · Address</b><br>ip -br addr"]
    R2 -->|"right network"| R3["<b>3 · Gateway</b><br>ip route · ping gateway"]
    R3 -->|"answers"| R4["<b>4 · Path</b><br>ping · traceroute -n"]
    R4 -->|"answers"| R5["<b>5 · Names</b><br>dig"]
    R5 -->|"resolves"| R6["<b>6 · Service</b><br>nc -zv · curl"]
    R1 & R2 & R3 & R4 & R5 & R6 -.->|"first failure"| STOP(["<b>Stop here:</b><br>this rung is the fault"])

Two rules go with the ladder:

Climb in order and stop at the first rung that fails. Everything above a broken rung fails too, so the errors above it tell you nothing new.
Change one thing, then test again. If the test still fails, undo the change. "I'll restart everything" may hide the fault, and it teaches you nothing.

Careful: ping works, so the network is fine?

ping only tests rungs 1 to 4. A web page also needs rung 5 (the name must become an address) and rung 6 (the web server must listen on its port). Ping working is good news, not the end of the diagnosis.

A healthy network first¶

You cannot recognise a sick network until you know what a healthy one looks like. In the lab (the office topology from the IPv6 session: pc1 behind the router r1, the server srv beyond it) climb all six rungs on a network that works.

cd ~/lab/office && containerlab deploy -t office.clab.yml
docker exec -it clab-office-pc1 bash

eth0 is containerlab's management port: ignore every line that mentions eth0 or 172.20.20. The server is 203.0.113.10, and its name is intranet.tramuntana.internal.

# ip -br link show eth1
eth1@if24        UP             aa:c1:ab:92:d2:7f <BROADCAST,MULTICAST,UP,LOWER_UP>
# ip route
default via 10.1.10.1 dev eth1
10.1.10.0/24 dev eth1 proto kernel scope link src 10.1.10.10
# traceroute -n 203.0.113.10
 1  10.1.10.1  0.028 ms  0.008 ms  0.006 ms
 2  203.0.113.10  0.014 ms  0.025 ms  0.004 ms
# dig +short intranet.tramuntana.internal
203.0.113.10
# nc -zv 203.0.113.10 80
Connection to 203.0.113.10 80 port [tcp/http] succeeded!

Rung by rung: the link is UP; there is a default route via 10.1.10.1; the server is two hops away (the router, then the server); the name resolves; and port 80 answers succeeded. cat /etc/resolv.conf tells you which DNS server the PC uses. Try other ports with nc -zv too: an open port says succeeded, a closed one says Connection refused.

Reading the error messages¶

The error message is your best clue. Learn to read these six.

Message	Where you see it	What it really means	Rung
Network is unreachable	`ping`	My own routing table has no way to send this. The packet never left the PC	1, 2 or 3
100% packet loss	`ping`	The packet left, and no answer came back	4 (or the gateway, on rung 3)
`* * *` after hop 1	`traceroute -n`	Packets reach the router and die after it	4
no servers could be reached	`dig`	The DNS server you are using does not answer	5
Connection refused	`nc`, `curl`	The server answered at once: "nobody listens on that port"	6, on the server
timed out	`nc`, `curl`	No answer at all: something on the way threw the packet away	6, a firewall

Careful: Linux removes routes with the link

If an interface goes down, Linux deletes the routes that used it, including the default route. So after ip link set eth1 up the ping may still fail. Good: go back down the ladder. Now rung 3 fails, and ip route add default via 10.1.10.1 brings the way out back.

Two different silences¶

These two lines are real output from the lab:

nc: connect to 203.0.113.10 port 80 (tcp) failed: Connection refused
nc: connect to 203.0.113.10 port 80 (tcp) timed out: Operation now in progress

Refused: the server said no. The path is fine; look at the service on the server with ss -tuln. If no line ends in :80, the web server is not running.
Timed out: nobody said anything. A firewall somewhere dropped the packet without a word. If port 22 of the same server succeeds, the path is proven good and only port 80 is being dropped.

Careful: ping fails, so the host is off?

Many hosts and firewalls drop ping on purpose. If what you need is a web page, test the port you need with nc -zv host 80. And * * * in traceroute only means that one hop did not answer the probe: what matters is whether the last hop answers.

When the ladder is not enough: watch the packets¶

Sometimes you know the packet dies inside the router, but not why. Then you watch it. With tcpdump you capture on each side of r1:

docker exec -it clab-office-r1 tcpdump -ni eth1 tcp port 80     # packets arrive from pc1
docker exec -it clab-office-r1 tcpdump -ni eth3 tcp port 80     # nothing leaves towards srv

Packets in on one side, nothing out on the other: the router is dropping them. On a Linux router, nft list ruleset shows its firewall rules. This is what Wireshark and tcpdump are for in real life: to see where a packet dies.

The ticket¶

When you find a fault, you write it down. A ticket is the record of a problem: the four things another technician needs to trust your diagnosis.

Field	What you write	Example
Symptom	What the user sees	"The intranet does not load"
Evidence	The command and the line that proves the first failing rung	`nc -zv 203.0.113.10 80` → Connection refused
Cause	What is really broken	The web server on `srv` is stopped: no `:80` in `ss -tuln`
Fix	What you changed, and the test that proves it works	Started the web server; `curl -I http://intranet.tramuntana.internal` answers `200 OK`

"The Internet is down" is a symptom, not a diagnosis. "Rung 5 fails: dig says no servers could be reached, dig @203.0.113.10 answers, so /etc/resolv.conf points at the wrong server" is a diagnosis.

Seven tickets and a real router¶

Now practise on broken networks. The Exercise sheet (Practical 3.9) has the ladder and a ticket table.

In the lab. From a second terminal on the host (not inside a node), open one ticket at a time with the lab's fault script, diagnose it from pc1, fill in the ticket line, repair it, test, then open the next:

cd ~/lab/office && bash fault.sh 1       # then 2 ... 7;  bash fault.sh fix  repairs all

You are the administrator of the whole office: you may also enter r1 and srv with docker exec -it clab-office-r1 bash. Don't read fault.sh: your mark comes from the evidence, not from the name of the fault. A ticket is closed when curl -I http://intranet.tramuntana.internal answers 200 OK.

On the real MikroTik. Each group plants one secret fault on its own router and then diagnoses the neighbours' network with the ladder, from the PC on ether4. On their router you may only look. These RouterOS commands show the router's side of each rung:

/interface print
/ip address print
/ip dhcp-server print
/ip dhcp-server network print
/ip firewall nat print
/ip firewall filter print
/ping 1.1.1.1 count=3
/tool traceroute 1.1.1.1

In print output, an X in front of a line means that item is disabled.

Hand in practical-3.9-surname.md on Google Classroom: the ticket table typed in Markdown, with one evidence line copied from the terminal for each ticket.

Key takeaways¶

A network fails at the bottom and complains at the top: the Facebook outage of 4 October 2021 looked like DNS but was routing.
The ladder: 1 Link, 2 Address, 3 Gateway, 4 Path, 5 Names, 6 Service. Climb in order, stop at the first rung that fails, change one thing and test again.
ping proves rungs 1–4 only; names need dig and ports need nc -zv or curl.
Network is unreachable is local (the packet never left); 100% packet loss means it left and nobody answered.
Connection refused points at the service on the server; timed out points at a firewall on the way.
A ticket records symptom, evidence, cause and fix.

Quick check¶

1. ping 10.1.10.1 (your gateway) works, but ping 203.0.113.10 says Network is unreachable. What is missing?

A default route
A network cable
An IP address
A DNS server

The gateway answers, so rungs 1 and 2 are fine and the gateway is in your own network. Network is unreachable means your own routing table has no way out: the default route is missing. Fix: ip route add default via 10.1.10.1.

2. ping 1.1.1.1 works, but ping example.com says Name or service not known. Which rung fails?

5 · Names
4 · Path
3 · Gateway
6 · Service

Reaching 1.1.1.1 proves rungs 1–4. Only the name fails, so the problem is DNS. dig @another-server example.com tells you whether the fault is the DNS server you are using.

3. nc -zv 203.0.113.10 80 answers Connection refused at once. Where do you look first?

At the service on the server
At your own network cable
At your DNS server settings
At the router's NAT rule

Refused is an answer from the server itself: the path works, but nobody is listening on that port. On the server, ss -tuln shows whether anything listens on :80.

4. Port 22 of a server succeeds, but port 80 of the same server times out. What is the most likely cause?

A firewall dropping port 80
A broken cable on your PC
A wrong default gateway
A wrong IP address on your PC

Port 22 working proves the cable, the address, the gateway and the path. Timed out means nobody answered: something on the way silently dropped the port 80 packets, which is what a firewall rule does.

5. Why must you climb the ladder in order and stop at the first rung that fails?

Every rung above a broken one fails too, so its errors add no new information
Because the commands only work when they are typed in that order
Because the top rungs need administrator rights that you get later
Because each command repairs the rung below it automatically

Each rung stands on the one below: no address means no gateway, no path, no names, no service. The first rung that fails is where the fault is; the errors above it are just consequences.

6. After ip link set eth1 up, the interface is UP again but ping to the server still fails. What do you do?

Go back down the ladder: Linux removed the default route with the link
Restart the PC, the router and the server to clear everything
Replace the cable, because UP is not reliable
Assume the server is off, because ping fails

When an interface goes down, Linux deletes its routes, including the default one. Climbing again shows rung 3 failing now; ip route add default via ... fixes it. One change, one test.

7. A classmate cannot open any web page, but ping 1.1.1.1 works. Select the two next steps that follow the method.

Run dig on a name to test rung 5
Test the web port with nc -zv or curl to check rung 6
Restart the router, because that fixes most problems
Check the cable first, because ping could be lying

Ping to 1.1.1.1 already proves rungs 1 to 4, so the next rungs are names (dig) and service (nc -zv, curl). Restarting hides the fault, and the cable is already proven by the working ping.

Next: some of those ports carry passwords in plain text that anyone could read. See how SSH and TLS lock them — SSH and TLS.