Lab 04-04 · ECMP: two equal paths at once (optional)¶

At the supermarket there are two checkouts open, both with short queues. The staff do not send everybody to checkout 1 until it is full: customers spread over both. And if checkout 2 suddenly closes, nobody stays stuck in front of it; everybody walks over to checkout 1.

At the end of Lab 04-03 your router reached neighbour A's network through B (the detour). In Lab 04-02 it reached it directly. Both paths work, both are one route with distance 1. Why choose? In this lab you give your router both routes at the same time. It will spread the traffic over the two paths, and when one of them dies it will notice and use only the other: ECMP with check-gateway.

The analogy: two open checkouts¶

What is ECMP?¶

ECMP (Equal Cost Multi-Path) means the router keeps several routes to the same destination with the same distance and uses all of them, spreading the traffic.

Remember Lab 04-01: when two routes are equally specific, the lower distance wins. If the distances are also equal, RouterOS does not throw one away: both become active and are marked with the flag + (ECMP). All the reachable next hops are copied to the FIB (Forwarding Information Base, slide p179), the fast table the router actually uses to forward packets, built from the best routes of the routing table.

Which packet takes which path?¶

Sending packet 1 by path A, packet 2 by path B and so on would mix up the order of a conversation. So the router chooses a path per flow: it computes a hash (a quick fingerprint number) from fields of the packet, and the same fields always give the same path. The slides (p179) list the fields used by RouterOS v6: source and destination address, incoming interface, routing mark and ToS. In RouterOS v7 the fields are set in IP → Settings → IPv4 Multipath Hash Policy; the default, l3, uses the source and destination addresses.

Consequence: pc1 talking to pc2 always uses the same path. To see both paths in use you need several different address pairs, as in a real office with many laptops. You will fake that in Step 3.

check-gateway: noticing a closed till¶

A static route does not know whether its gateway is alive. With check-gateway=ping (slide p165) the router pings the gateway every 10 seconds; after two timeouts in a row the route becomes inactive (flag I). In an ECMP group, only that route is excluded and the others keep working. When the gateway answers again, the route comes back by itself.

The scenario¶

The same network as Lab 04-03, which is already solved: your router reaches A's LAN through B.

flowchart LR
    PC1["<b>pc1</b><br>your laptop<br>+ 8 test addresses"]
    R1["<b>r1 · you</b><br>10.0.0.11"]
    R3["<b>r3 · B</b><br>10.0.0.13<br>route to A"]
    R2["<b>r2 · A</b><br>10.0.0.12"]
    PC2["<b>pc2</b><br>A's laptop<br>192.168.89.254"]
    PC1 --> R1
    R1 -->|"path 1 · via B (from Lab 04-03)"| R3
    R3 --> R2
    R1 -->|"path 2 · direct (new)"| R2
    R2 --> PC2

Start the lab¶

Download the lab files and unzip them: 04-04-ecmp.zip. Open a terminal inside the 04-04-ecmp folder.

sudo containerlab deploy -t topology.clab.yml

Four RouterOS routers have to boot: give it about three minutes. Then:

ssh admin@clab-mtcna-04-04-r1            # password: admin   (also r2, r3)
docker exec -it clab-mtcna-04-04-pc1 bash   # also pc2

At any moment you can run CLAB_EXEC="docker exec" ./check.sh. Note that the check switches r3's ether2 off for about half a minute to test the failover, and switches it back on at the end.

The laptops ask for their address by themselves when the lab starts; if ip -br address show eth1 shows no IPv4 address yet (the router was still booting), run dhclient -v eth1.

Step 1 · Add the second path¶

Your router already has path 1 (via B). Add path 2, straight to A, with the same destination and the same distance (1, the default for static routes).

First try the v6 way from the slides, to see it fail:

/ip route add dst-address=192.168.89.0/24 gateway=10.0.0.12,10.0.0.13

RouterOS 7 answers invalid or unexpected argument. Now the v7 way. WinBox: IP → Routes → +, Dst. Address 192.168.89.0/24, Gateway 10.0.0.12, Check Gateway ping.

/ip route add dst-address=192.168.89.0/24 gateway=10.0.0.12 check-gateway=ping
/ip route print where dst-address=192.168.89.0/24

Expected:

Flags: A - ACTIVE; s - STATIC; + - ECMP
#     DST-ADDRESS      GATEWAY    ROUTING-TABLE  DISTANCE
0 As+ 192.168.89.0/24  10.0.0.13  main                  1
1 As+ 192.168.89.0/24  10.0.0.12  main                  1

Both routes are active and both carry the + flag. That is ECMP.

Done when both routes show As+.

Step 2 · Let the old route check its gateway too¶

Path 1 was created in Lab 04-03 without check-gateway. Without it, if B dies, your router would keep sending half the flows into a black hole. WinBox: double-click the route via 10.0.0.13, Check Gateway ping.

/ip route set [find where dst-address=192.168.89.0/24 and gateway=10.0.0.13] check-gateway=ping
/ip route print detail where dst-address=192.168.89.0/24

Expected: both routes show check-gateway=ping.

Done when both ECMP routes have check-gateway=ping.

Step 3 · See the load sharing¶

pc1 talking to pc2 always hashes to the same path. To imitate an office full of laptops, give pc1 eight extra addresses and send a traceroute from each one. On pc1:

for i in 1 2 3 4 5 6 7 8; do ip addr add 192.168.88.20$i/24 dev eth1; done
for i in 1 2 3 4 5 6 7 8; do
  echo -n "from 192.168.88.20$i: "; traceroute -n -q 1 -m 2 -s 192.168.88.20$i 192.168.89.254 | tail -1
done

Expected: some sources show 10.0.0.12 as hop 2 (direct to A), others 10.0.0.13 (via B). Run the second loop again: every source keeps its path. The split is decided by the hash, so your numbers will differ from your classmates'.

Done when you see both 10.0.0.12 and 10.0.0.13 in the results.

Step 4 · Close one checkout: failover¶

Now switch neighbour B's router off the class network. On r3:

/interface disable ether2

On r1, watch the routes for about half a minute (WinBox: IP → Routes, it refreshes by itself):

/ip route print where dst-address=192.168.89.0/24

After two failed pings (about 20 seconds), the route via B changes:

Flags: I - INACTIVE, A - ACTIVE; s - STATIC
0 Is 192.168.89.0/24  10.0.0.13  main  1
1 As 192.168.89.0/24  10.0.0.12  main  1

The + is gone: there is only one active path left. Run the traceroute loop of Step 3 again on pc1: every source now goes through 10.0.0.12, and pings still work.

Bring B back:

/interface enable ether2

Within about 10 seconds (the next successful ping) the route via B is active again and the + flags return.

Done when you have seen the route via B go I and come back A. Then remove the test addresses (for i in 1 2 3 4 5 6 7 8; do ip addr del 192.168.88.20$i/24 dev eth1; done) and run CLAB_EXEC="docker exec" ./check.sh: every line must say PASS.

If something goes wrong¶

Clean up¶

sudo containerlab destroy -t topology.clab.yml --cleanup

Key takeaways¶

• ECMP uses several routes to the same destination with the same distance at the same time; RouterOS marks them with +.
• In RouterOS v7 you create ECMP with one route per gateway; the v6 gateway=A,B syntax is rejected.
• The router picks a path per flow with a hash; in v7 the default (l3) uses source and destination address, so one conversation stays on one path.
• The usable next hops are copied into the FIB, the table the router really forwards with.
• check-gateway=ping removes a dead path after two missed pings (about 20 s) and restores it when the gateway answers again.

Quick check¶

Next: ECMP and check-gateway only notice a dead gateway. What if the gateway is fine but the provider behind it is broken? The answer is recursive routing.