1 What happened?
Docker published port is NOT accessible from the host itself, but can be accessed from outside the host.
On the Docker host:
$ docker run -d -p 80:80 httpd
$ # ip -br a
lo UNKNOWN 127.0.0.1/8
ens160 UP 182.150.183.109/25
ens224 UP 182.150.151.109/24
docker0 UP 172.17.0.1/16
$ curl http://182.150.183.109
curl: (7) Failed connect to 182.150.151.109:80; Connection refused
On another host:
# curl http://182.150.183.109
<html><body><h1>It works!</h1></body></html>
2 Root cause
This Docker host has customized routing tables for each IP address configured.
As a result, the packets generated from the host follow that customized routing table, NOT the "main" table.
Docker updates the "main" table, but does NOT update the customized routing table. So the customized table doesn't know docker bridge at all, and routes the output packets to default gateway...
# ip rule
0: from all lookup local
32764: from 182.150.151.109 lookup management
32765: from 182.150.183.109 lookup apps
32766: from all lookup main
32767: from all lookup default
# ip route list table apps
default via 182.150.183.1 dev ens160
182.150.183.0/25 dev ens160 scope link
3 Solution
Manully adding docker bridge into customized routing tables can solve the problem. e.g.
# ip route add 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 table apps
Verify by
# ip route list table apps
default via 142.150.183.1 dev ens160
182.150.183.0/25 dev ens160 scope link
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
# curl http://182.150.183.109
<html><body><h1>It works!</h1></body></html>
This solution only works for fixed Docker bridge docker0. Docker compose dynamically creates/deletes bridges, which makes manual updating routing tables impractical!
If one interface is used only for "system management", and all other business traffic go through the other interface, then a solution would be letting the business interface use the "main" routing table. Container apps are usually for business apps anyway.
No comments:
Post a Comment