Winse Blog

走走停停都是风景, 熙熙攘攘都向最好, 忙忙碌碌都为明朝, 何畏之.

Docker多主机网络配置 - Pipework

前面使用nat+route的方式手动连通两台机器上的docker容器。pipework是通过脚本的方式（手动）设置网络以及修改路由来进行配置的。

参考：

2、将物理网卡桥接到虚拟网桥，使得容器与宿主机配置在同一网段下

在各个宿主机上都建立一个新虚拟网桥设备br0，将各自物理网卡eth0桥接br0上，eth0的IP地址赋给br0；同时修改Docker daemon的DOCKER_OPTS，设置-b=br0（替代docker0），并限制Container IP地址的分配范围为同物理段地址（–fixed-cidr）。重启各个主机的Docker Daemon后，处于与宿主机在同一网段的Docker容器就可以实现跨主机访问了。这个方案同样存在局限和扩展性差的问题：比如需将物理网段的地址划分 成小块，分布到各个主机上，防止IP冲突；子网划分依赖物理交换机设置；Docker容器的主机地址空间大小依赖物理网络划分等。

原理就是建立一条连接link，一端 在主机 一端 在容器 ；然后手动配置容器ip和路由；最后把主机Ethernet和新建的Bridge桥接连接到物理网络。

容器的ip地址和主机的ip地址在一个网段内，所以在同一交换机下的所有主机、里面的容器都互通。

查看原网络的信息：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

[root@kube-worker1 ~]# nmcli d show ens33 
GENERAL.DEVICE:                         ens33
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         00:0C:29:2E:67:4D
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     ens33
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/0
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         192.168.191.141/24
IP4.GATEWAY:                            192.168.191.2
IP4.ROUTE[1]:                           dst = 172.17.3.0/24, nh = 192.168.191.140, mt = 100
IP4.DNS[1]:                             192.168.191.2
IP4.DOMAIN[1]:                          localdomain
IP6.ADDRESS[1]:                         fe80::3995:4490:e2e7:1d0f/64
IP6.GATEWAY:

安装pipework

1
2

git clone https://github.com/jpetazzo/pipework
cp ~/pipework/pipework /usr/local/bin/

运行docker

1
2
3
4
5
6
7
8
9
10
11
12
13

#设置ip转发
echo 1 > /proc/sys/net/ipv4/ip_forward

vi /etc/sysctl.conf
net.ipv4.ip_forward = 1  

NAME=test1

#如不需要安装软件，可以加 --net none
docker run -itd --name $NAME centos /bin/bash

#docker ps -a -f name=$NAME | grep $NAME && docker start $NAME 
#docker exec test1 yum install -y iproute net-tools

配置网络

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

function docker_container_ip () {
  local name=$1
  local ip=$2
  local gateway=${3:-$GATEWAY}
  
  pipework br0 $name $ip@$gateway
  #docker exec $name ifconfig 
  #docker exec $name route -n 
}

function docker_hosted_bridge_network_reset () {
  local ip=$1
  local gateway=$2
  local iface=$3
  
  if nmcli d show $iface | grep -i ethernet ; then
    #把地址给网桥，然后把ethernet和bridge连起来：(SSH连接操作的话，需要一条命令搞定！修改br0地址后route会变)
    ip addr add $ip dev br0 ; \
    ip addr del $ip dev $iface ; \
    brctl addif br0 $iface ; \
    #ip route del default ; \
    ip route add default via $gateway 
  fi 
  
  brctl show br0
}

GATEWAY=$( route -n | grep '^0.0.0.0' | awk '{print $2}' )
IFACE=$( route -n | grep '^0.0.0.0' | awk '{print $8}' )
HOSTED_IPADDR=$( ip addr show $IFACE | grep "inet " | awk '{print $2}' )

设置容器的IP：

1
2
3
4
5

NAME=test1
IP=192.168.191.210/24

docker_container_ip $NAME $IP $GATEWAY
docker_hosted_bridge_network_reset $HOSTED_IPADDR $GATEWAY $IFACE

上面的方式配置方式重启就失效的，可以通过写配置文件的方式来永久生效。如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

[root@kube-worker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 
TYPE=Ethernet
DEVICE=ens33
BRIDGE=br0
[root@kube-worker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br0 
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.191.141
NETMASK=255.255.255.0
GATEWAY=192.168.191.2
DNS1=192.168.191.2

USERCTL=no

测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

[root@kube-worker1 ~]# screen

  GATEWAY=$( route -n | grep '^0.0.0.0' | awk '{print $2}' )
  IFACE=$( route -n | grep '^0.0.0.0' | awk '{print $8}' )
  HOSTED_IPADDR=$( ip addr show $IFACE | grep "inet " | awk '{print $2}' )

  docker run -itd --name test21 centos /bin/bash
  docker run -itd --name test22 centos /bin/bash

  docker_container_ip test21 192.168.191.231/24 $GATEWAY
  docker_container_ip test22 192.168.191.232/24 $GATEWAY

  docker exec test21 ping  192.168.191.140
  docker exec test21 ping  192.168.191.141

[root@kube-master ~]# screen #会话"不断"

  docker_container_ip test11 192.168.191.221/24 $GATEWAY 
  docker_container_ip test12 192.168.191.222/24 $GATEWAY

  docker_hosted_bridge_network_reset $HOSTED_IPADDR $GATEWAY $IFACE

  docker exec test11 ping 192.168.191.233

注意：容器重启后，这些配置的网卡/路由都没有了，要重新配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

[root@kube-worker1 ~]# docker stop test21
test21
[root@kube-worker1 ~]# docker start test21
test21
[root@kube-worker1 ~]# pipework route test21 show 
default via 172.18.0.1 dev eth0 
172.18.0.0/16 dev eth0  proto kernel  scope link  src 172.18.0.2 

[root@kube-worker1 ~]# docker_container_ip test21 192.168.191.231/24 $GATEWAY
[root@kube-worker1 ~]# pipework route test21 show                            
default via 192.168.191.2 dev eth1 
172.18.0.0/16 dev eth0  proto kernel  scope link  src 172.18.0.2 
192.168.191.0/24 dev eth1  proto kernel  scope link  src 192.168.191.231 

[root@kube-worker1 ~]# docker exec test21 ping 192.168.191.140

了解原理后，操作起来还是比较容易的。就是每次重启都要重新配置比较烦。可以写成脚本，启动docker容器的时刻就执行下网络配置。

pipework还可以用来配置vlan，暂时没这个需求，并且基本的操作都类似就没有实际操作了。

话说， pipework还可以用来创建多网卡的容器。用docker network connect其实更简单。

后记

除了通过pipework来实现共享物理网络外，docker network也可以实现这个功能：

1
2
3
4
5
6
7
8

#中间会导致网络断掉，一条命令搞定才行
docker network create --gateway=192.168.191.141 --subnet 192.168.191.0/24 --aux-address "DefaultGatewayIPv4=192.168.191.2" -o com.docker.network.bridge.name=br-home-net homenet ; \
ip addr del 192.168.191.141/24 dev ens33 ; \
brctl addif br-home-net ens33 

#主机不上外网可以不加
ip route add default via 192.168.191.2 ; 
echo "nameserver 114.114.114.114" >>/etc/resolv.conf ;
  • 测试 docker-net + bridge：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

[root@kube-worker1 ~]# ip a
...
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br-home-net state UP qlen 1000
    link/ether 00:0c:29:2e:67:4d brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:44:ef:32:28 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
11: br-home-net: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:84:97:c2:25 brd ff:ff:ff:ff:ff:ff
    inet 192.168.191.141/24 scope global br-home-net
       valid_lft forever preferred_lft forever
    inet6 fe80::42:84ff:fe97:c225/64 scope link 
       valid_lft forever preferred_lft forever
       
[root@kube-worker1 ~]# docker run -tid --name c200 --net homenet --ip 192.168.191.200 busybox /bin/sh 
2579c2ddd18d23322eb1e145ad630205933dbc527b8981169ec6b125da8d8f1e

[root@kube-worker1 ~]# docker exec -ti c200 sh 
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:c0:a8:bf:c8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.191.200/24 scope global eth0
       valid_lft forever preferred_lft forever
/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.191.2   0.0.0.0         UG    0      0        0 eth0
192.168.191.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
/ # ping baidu.com 
PING baidu.com (111.13.101.208): 56 data bytes
64 bytes from 111.13.101.208: seq=0 ttl=128 time=48.225 ms
^C
--- baidu.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 48.225/48.225/48.225 ms
/ # ping 192.169.191.140
PING 192.169.191.140 (192.169.191.140): 56 data bytes
^C
--- 192.169.191.140 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
/ # ping 192.168.191.140
PING 192.168.191.140 (192.168.191.140): 56 data bytes
64 bytes from 192.168.191.140: seq=0 ttl=64 time=2.572 ms
64 bytes from 192.168.191.140: seq=1 ttl=64 time=1.076 ms
^C
--- 192.168.191.140 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.076/1.824/2.572 ms
/ # ping 192.168.191.141
PING 192.168.191.141 (192.168.191.141): 56 data bytes
64 bytes from 192.168.191.141: seq=0 ttl=64 time=0.474 ms
64 bytes from 192.168.191.141: seq=1 ttl=64 time=0.138 ms
^C
--- 192.168.191.141 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.138/0.306/0.474 ms
/ # ping 192.168.191.1
PING 192.168.191.1 (192.168.191.1): 56 data bytes
64 bytes from 192.168.191.1: seq=0 ttl=128 time=1.068 ms
64 bytes from 192.168.191.1: seq=1 ttl=128 time=0.603 ms
^C
--- 192.168.191.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.603/0.835/1.068 ms

–END

Comments