Debian routes external network IP ping through the lo interface
One IP address in subnet 10.0.25.0/24 cannot be pinged from Debian Zabbix host V11 (192.168.112.61). Other IP addresses in the same subnet are reachable from V11. All hosts in 10.0.25.0/24 are reachable through the same router (192.168.112.1) from other hosts on 192.168.112.0/24.
Resolution after the first incident
Reboot the affected host.
The problem returned about a day later. Analyze the situation while the issue is occurring:
# System
root@v11:~# uname -a
Linux v11 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
#
root@v11:~# cat /etc/debian_version
7.6
# Routes
root@v11:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.112.1 0.0.0.0 UG 0 0 0 eth0
192.168.112.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
There is no problem with addresses 10.0.25.2, .8, .9, or others.
root@v11:~# traceroute 10.0.25.2
traceroute to 10.0.25.2 (10.0.25.2), 30 hops max, 60 byte packets
1 gateway.firma.local (192.168.112.1) 0.284 ms 0.221 ms 0.265 ms
2 192.168.112.90 (192.168.112.90) 1.447 ms 1.452 ms 1.489 ms
3 10.0.25.2 (10.0.25.2) 2.144 ms 1.605 ms 1.931 ms
root@v11:~# traceroute 10.0.25.3
traceroute to 10.0.25.3 (10.0.25.3), 30 hops max, 60 byte packets
1 v11.firma.local (192.168.112.61) 2999.628 ms !H 2999.625 ms !H 2999.618 ms !H
PING 10.0.25.3 (10.0.25.3) from 192.168.112.61 eth0: 56(84) bytes of data.
From 192.168.112.61 icmp_seq=1 Destination Host Unreachable
From 192.168.112.61 icmp_seq=2 Destination Host Unreachable
From 192.168.112.61 icmp_seq=3 Destination Host Unreachable
Inspect the lo interface with tcpdump while attempting to ping the affected address 10.0.25.3:
#
root@v11:~# ping 10.0.25.3
PING 10.0.25.3 (10.0.25.3) 56(84) bytes of data.
From 192.168.112.61 icmp_seq=1 Destination Host Unreachable
From 192.168.112.61 icmp_seq=2 Destination Host Unreachable
From 192.168.112.61 icmp_seq=3 Destination Host Unreachable
From 192.168.112.61 icmp_seq=4 Destination Host Unreachable
From 192.168.112.61 icmp_seq=5 Destination Host Unreachable
From 192.168.112.61 icmp_seq=6 Destination Host Unreachable
From 192.168.112.61 icmp_seq=7 Destination Host Unreachable
From 192.168.112.61 icmp_seq=8 Destination Host Unreachable
From 192.168.112.61 icmp_seq=9 Destination Host Unreachable
^C
--- 10.0.25.3 ping statistics ---
11 packets transmitted, 0 received, +9 errors, 100% packet loss, time 9999ms
#
# TCPDUMP at the same time
root@v11:~# tcpdump -i lo 'icmp and src 192.168.112.61'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
15:11:39.495921 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:39.495940 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:39.495945 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:42.495892 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:42.495898 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:42.495902 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:45.495919 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:45.495936 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:45.495941 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:48.495915 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
15:11:48.495920 IP v11.firma.local > v11.firma.local: ICMP host 10.0.25.3 unreachable, length 92
^C
11 packets captured
22 packets received by filter
0 packets dropped by kernel
Resolution after the second incident
This helps:
Updated the system after the third recurrence
Monitor the system. A few hours later, the problem recurred.
Working solution
Root cause
The router (192.168.112.1) has a static route to target subnet 10.0.25.0/24 through the L2TP tunnel client IP 192.168.112.90. The router also has the IP Send Redirects option enabled (=1). When the tunnel to the target subnet went down and the affected host tried to reach 10.0.25.3, the router sent it a redirect to 192.168.112.90. The affected client followed this instruction because net.ipv4.conf.default.accept_redirects=1 is enabled by default.
Options
- Disable Send Redirects on the router.
- Disable Accept Redirect on the host.