Lab 04-05 · Recursive routing and failover (optional)¶

Your phone says it is connected to the WiFi: full bars. You open Instagram and... nothing loads. The phone was not lying: the link to the WiFi router is perfect. The problem is further away: the router's own Internet connection is down. The bars only measure the first metre of the journey.

A small office with two Internet providers has the same problem. The classic failover, "if the provider's router stops answering, use the other provider", only checks the first metre: the cable to the provider. If the provider breaks somewhere behind its router, the office keeps sending everything into a dead end. In this lab you build the cure taught on slide p184: test a real address on the Internet, through each provider, and switch only when that test fails. That trick is called recursive routing.

The analogy: WiFi bars vs "does Instagram load?"¶

Failover, distance and check-gateway¶

Failover is switching automatically to a backup path when the main one fails.

You already have all the pieces from the previous labs:

• Distance decides the order: the main route gets 1, the backup 2. While both are active, distance 1 wins (they are not equal, so this is not ECMP).
• check-gateway=ping makes a route inactive when its gateway misses two pings in a row (about 20 s). Then the distance-2 route is the best one left and becomes active.

The router you receive already has this naive failover:

0.0.0.0/0  gateway=10.1.1.1  distance=1  check-gateway=ping    ← ISP1's router
0.0.0.0/0  gateway=10.2.2.1  distance=2  check-gateway=ping    ← ISP2's router

It pings ISP1's router, 10.1.1.1. If the cable is cut, the ping fails and the backup takes over. But if ISP1's router is alive and its link to the Internet is dead, 10.1.1.1 keeps answering, the route stays active, and the office is offline. That is the failure "in an upper node" of slide p184.

Recursive routing¶

A recursive route is a route whose gateway is not in a directly connected network. The router finds out how to reach that gateway by looking it up in its routing table again: it recurses.

The trick of slide p184 in three lines:

1. A probe route: 8.8.8.8/32 (one single address) through ISP1's router 10.1.1.1. Now 8.8.8.8 can only be reached through ISP1. Same for 1.1.1.1/32 through ISP2.
2. The main default route uses 8.8.8.8 as its gateway, with check-gateway=ping and distance 1. The router resolves "how do I reach 8.8.8.8?" through the probe route: via 10.1.1.1. So the traffic really goes to ISP1, but the health check pings 8.8.8.8, across ISP1's whole network.
3. The backup default route does the same with 1.1.1.1 (ISP2), distance 2.

If anything between you and 8.8.8.8 breaks, inside ISP1 or beyond, the ping fails, the main route goes inactive, and the backup takes over.

flowchart LR
    R1["<b>r1</b><br>default → 8.8.8.8 (d=1)<br>default → 1.1.1.1 (d=2)"]
    ISP1["<b>isp1</b><br>10.1.1.1"]
    ISP2["<b>isp2</b><br>10.2.2.1"]
    NET["<b>internet</b><br>8.8.8.8 · 1.1.1.1<br>198.51.100.10"]
    R1 -->|"8.8.8.8/32 via 10.1.1.1<br>ping every 10 s"| ISP1
    R1 -->|"1.1.1.1/32 via 10.2.2.1"| ISP2
    ISP1 -->|"this link breaks"| NET
    ISP2 --> NET

Scope and target-scope: permission to recurse¶

By default RouterOS does not resolve a gateway through just any route; otherwise a mistake could create loops. Two numbers control it:

scope is a number on every route: how "close" it is. Connected routes have scope 10, static routes scope 30 by default.

target-scope is a number on a route that says which routes may be used to resolve its gateway: only routes whose scope is within the target-scope. Static routes have target-scope 10 by default.

So by default a static route can only find its gateway through connected routes (scope 10). To let the default route find 8.8.8.8 through the probe route, you give the probe route scope=10 and the default route a bigger target-scope. Slide p185 states the RouterOS v7 rule: target-scope must be bigger than scope, so target-scope=11. We tested it on RouterOS 7.23: with target-scope=10 the route stays inactive (its immediate-gw stays empty); with 11 it becomes active, with immediate-gw=10.1.1.1%ether2.

The scenario¶

There is only one trainer's router in the classroom, so this lab builds its own small world: one office router and two simulated Internet providers. 8.8.8.8 and 1.1.1.1 are famous public DNS servers (Google and Cloudflare); here they are just two addresses on the internet node, and nothing leaves your computer.

flowchart LR
    PC1["<b>pc1</b><br>office laptop<br>192.168.88.x"]
    subgraph R1["r1 · office router"]
        E3["ether3 · bridge1<br>192.168.88.1/24"]
        E2["ether2 · WAN 1<br>10.1.1.2/24"]
        E4["ether4 · WAN 2<br>10.2.2.2/24"]
    end
    ISP1["<b>isp1</b> (main)<br>10.1.1.1 · 172.16.1.1"]
    ISP2["<b>isp2</b> (backup)<br>10.2.2.1 · 172.16.2.1"]
    NET["<b>internet</b><br>8.8.8.8 · 1.1.1.1<br>web 198.51.100.10"]
    PC1 --- E3
    E2 --- ISP1
    E4 --- ISP2
    ISP1 ---|"172.16.1.0/30"| NET
    ISP2 ---|"172.16.2.0/30"| NET

Start the lab¶

Download the lab files and unzip them: 04-05-recursive-failover.zip. Open a terminal inside the 04-05-recursive-failover folder.

sudo containerlab deploy -t topology.clab.yml

Give RouterOS about two minutes to boot. Then:

ssh admin@clab-mtcna-04-05-r1            # password: admin
docker exec -it clab-mtcna-04-05-pc1 bash
docker exec -it clab-mtcna-04-05-isp1 bash   # you will "break" ISP1 from here

At any moment you can run CLAB_EXEC="docker exec" ./check.sh. The check breaks ISP1's Internet link for about half a minute and repairs it at the end.

The laptops ask for their address by themselves when the lab starts; if ip -br address show eth1 shows no IPv4 address yet (the router was still booting), run dhclient -v eth1.

Step 1 · Test the naive failover (it works for a cut cable)¶

Look at the routes you received. On r1 (WinBox: IP → Routes):

/ip route print

Two default routes: via 10.1.1.1 (distance 1, active) and via 10.2.2.1 (distance 2). On pc1:

traceroute -n 198.51.100.10

Hop 2 is 10.1.1.1: ISP1's router. Now pull the cable to ISP1. On r1:

/interface disable ether2
/ip route print where dst-address=0.0.0.0/0

The route via 10.1.1.1 becomes I immediately (its interface is down) and the backup is active. Repeat the traceroute: hop 2 is now 10.2.2.1. Plug the cable back:

/interface enable ether2

Done when you have seen the traffic move to ISP2 and back.

Step 2 · Break ISP1 upstream (the naive failover fails)¶

Now the nasty failure: ISP1's router is fine, but it loses its Internet link. On isp1:

ip link set eth2 down

Wait 30 seconds, then on r1 and pc1:

/ip route print where dst-address=0.0.0.0/0

curl -m 5 http://198.51.100.10
traceroute -n 198.51.100.10

The route via 10.1.1.1 is still active: ISP1's router answers every ping. But curl fails, and the traceroute stops at 10.1.1.1 (with !N: ISP1 has no way to the Internet). The office is offline, and the backup sits unused. Repair ISP1 before going on (Linux removes a route when its link goes down, so put it back too):

ip link set eth2 up
ip route replace default via 172.16.1.2

Done when you have seen the office go offline while the main route stayed active.

Step 3 · Add the probe routes¶

Pin 8.8.8.8 to ISP1 and 1.1.1.1 to ISP2, with scope=10. WinBox: IP → Routes → +, Dst. Address 8.8.8.8/32, Gateway 10.1.1.1, Scope 10 (and the same for 1.1.1.1/32 via 10.2.2.1).

/ip route add dst-address=8.8.8.8/32 gateway=10.1.1.1 scope=10
/ip route add dst-address=1.1.1.1/32 gateway=10.2.2.1 scope=10
/ping 8.8.8.8 count=3
/ping 1.1.1.1 count=3

Both pings answer. Each address can now only be reached through its own provider.

Done when both probe routes are active and both pings work.

Step 4 · Add the recursive default routes¶

WinBox: IP → Routes → +, Dst. Address 0.0.0.0/0, Gateway 8.8.8.8, Check Gateway ping, Distance 1, Target Scope 11. Then the backup with 1.1.1.1 and Distance 2.

First see the v7 rule in action. Add the main route without target-scope:

/ip route add dst-address=0.0.0.0/0 gateway=8.8.8.8 check-gateway=ping distance=1
/ip route print detail where gateway=8.8.8.8

It shows I (inactive), target-scope=10 and immediate-gw="": RouterOS is not allowed to resolve 8.8.8.8 through a route of scope 10. Fix it, and add the backup:

/ip route set [find where dst-address=0.0.0.0/0 and gateway=8.8.8.8] target-scope=11
/ip route add dst-address=0.0.0.0/0 gateway=1.1.1.1 target-scope=11 check-gateway=ping distance=2
/routing route print where dst-address=0.0.0.0/0

Expected: the route via 8.8.8.8 is active, with immediate gateway 10.1.1.1%ether2: it found its way through the probe route. The one via 1.1.1.1 waits with distance 2.

Look closely: right now two default routes have distance 1 (the naive one via 10.1.1.1 and the new one via 8.8.8.8), so RouterOS shows both with +: that is ECMP, not what you want. Finally remove the naive routes:

/ip route remove [find where comment~"naive"]
/ip route print

Expected: two default routes (via 8.8.8.8, active; via 1.1.1.1, standby) and the two probe routes.

Done when the default route via 8.8.8.8 is active with immediate gateway 10.1.1.1.

Step 5 · Break ISP1 upstream again (now it works)¶

Repeat Step 2. On isp1: ip link set eth2 down. On r1, watch for about 30 seconds:

/ip route print where dst-address=0.0.0.0/0

After two missed pings to 8.8.8.8 the main route becomes I and the route via 1.1.1.1 becomes A. On pc1:

curl http://198.51.100.10
traceroute -n 198.51.100.10

curl works again, and hop 2 is now 10.2.2.1: ISP2. Repair ISP1:

ip link set eth2 up
ip route replace default via 172.16.1.2

Within about 10 seconds the ping to 8.8.8.8 succeeds, the main route is active again, and the traceroute goes back through 10.1.1.1. Nobody had to touch the office router.

Done when the traffic moved to ISP2 and came back by itself. Run CLAB_EXEC="docker exec" ./check.sh: every line must say PASS.

If something goes wrong¶

Clean up¶

sudo containerlab destroy -t topology.clab.yml --cleanup

Key takeaways¶

• Failover with distance (1 main, 2 backup) and check-gateway only sees what it pings: pinging the provider's router misses upstream failures.
• A recursive route has a gateway that is not directly connected; RouterOS finds it through another route.
• scope and target-scope control which routes may resolve a gateway; in RouterOS v7 the target-scope must be bigger than the probe route's scope (scope=10, target-scope=11).
• Pinning a public address (8.8.8.8) to each provider with a /32 route turns check-gateway into an end-to-end test of that provider.
• In v7 you add routes in /ip route and inspect every detail in /routing route.

Quick check¶

That closes the routing module. Next module: wireless, where the cable you have relied on so far is replaced by radio.