Winse Blog

走走停停都是风景, 熙熙攘攘都向最好, 忙忙碌碌都为明朝, 何畏之.

Docker多主机网络配置 - Macvlan

参考

Note: In Macvlan you are not able to ping or communicate with the default namespace IP address. For example, if you create a container and try to ping the Docker host’s eth0 it will not work. That traffic is explicitly filtered by the kernel modules themselves to offer additional provider isolation and security.

主机

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@kube-master140 ~]# ip addr show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:40:2d:15 brd ff:ff:ff:ff:ff:ff
    inet 192.168.191.140/24 brd 192.168.191.255 scope global dynamic ens33
       valid_lft 1765sec preferred_lft 1765sec
    inet6 fe80::1186:2fe5:9ee5:8790/64 scope link 
       valid_lft forever preferred_lft forever

[root@kube-worker141 ~]# ip addr show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:2e:67:4d brd ff:ff:ff:ff:ff:ff
    inet 192.168.191.141/24 brd 192.168.191.255 scope global dynamic ens33
       valid_lft 1779sec preferred_lft 1779sec
    inet6 fe80::dd23:1df6:b37:efae/64 scope link 
       valid_lft forever preferred_lft forever

创建网络

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@kube-worker141 ~]# docker network create \
-d macvlan \
--subnet=192.168.191.0/24 \
--gateway=192.168.191.2 \
-o parent=ens33 pub_net
4370998ed03024bc0057a894f1280d5b0fcdba526fd9e8da612a3abb0dbc884b

[root@kube-worker141 ~]# docker network list 
NETWORK ID          NAME                DRIVER              SCOPE
eee9236a36ba        bridge              bridge              local               
ddc7f59215c1        host                host                local               
d8dc7fbc40a6        none                null                local               
4370998ed030        pub_net             macvlan             local               

[root@kube-worker141 ~]# docker network inspect pub_net
...

使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
docker rm -f $( docker ps -a | grep -v IMAGE | awk '{print $1}' ) 

[root@kube-worker141 ~]# docker run --net=pub_net --ip=192.168.191.200 --name c200 -tid busybox /bin/sh
2e0a2ede40e80a2f1739330bb3a6c45b91ea08d78d26d165ad13945bedbea40f

[root@kube-worker141 ~]# docker ps 
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
2e0a2ede40e8        busybox             "/bin/sh"           13 seconds ago      Up 11 seconds                           c200
[root@kube-worker141 ~]# docker exec c200 ifconfig 
eth0      Link encap:Ethernet  HWaddr 02:42:C0:A8:BF:C8  
          inet addr:192.168.191.200  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:c0ff:fea8:bfc8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

[root@kube-worker141 ~]# docker exec c200 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.191.2   0.0.0.0         UG    0      0        0 eth0
192.168.191.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
[root@kube-worker141 ~]# docker exec c200 ping baidu.com 
PING baidu.com (111.13.101.208): 56 data bytes
64 bytes from 111.13.101.208: seq=0 ttl=128 time=45.029 ms
64 bytes from 111.13.101.208: seq=1 ttl=128 time=44.616 ms

#201
[root@kube-worker141 ~]# docker run --net=pub_net --ip=192.168.191.201 -tid busybox /bin/sh 
c8cfd3443f2b7b3973a06470cb95442eadface8d89c8cb1749ad73ebbd7e9e39

##本地容器互通: 
#: HOST141-200 ping HOST141-201
[root@kube-worker141 ~]# docker exec c200 ping -W 10 192.168.191.201
PING 192.168.191.201 (192.168.191.201): 56 data bytes
64 bytes from 192.168.191.201: seq=0 ttl=64 time=0.523 ms

#210 
[root@kube-master ~]# docker run --net=pub_net --ip=192.168.191.210 -tid busybox /bin/sh 
7929c136c3dbc646b68b3b7302e8525a25fe2f583db2246fea0da85a448b7b78

##B访问A主机的容器: 
#: HOST140 ping HOST141-201 
[root@kube-master140 ~]# ping 192.168.191.201 
PING 192.168.191.201 (192.168.191.201) 56(84) bytes of data.
64 bytes from 192.168.191.201: icmp_seq=1 ttl=64 time=1.44 ms

##A主机容器访问B主机容器: 
#: HOST141-200 ping HOST140-210
[root@kube-worker141 ~]# docker exec c200 ping -W 10 192.168.191.210
PING 192.168.191.210 (192.168.191.210): 56 data bytes
64 bytes from 192.168.191.210: seq=0 ttl=64 time=2.049 ms
64 bytes from 192.168.191.210: seq=1 ttl=64 time=0.993 ms

#主机与所在容器互相不能访问 (--!): 
#: HOST141 ping HOST141-200
[root@kube-worker141 ~]# ping 192.168.191.200
PING 192.168.191.200 (192.168.191.200) 56(84) bytes of data.
From 192.168.191.141 icmp_seq=1 Destination Host Unreachable
From 192.168.191.141 icmp_seq=2 Destination Host Unreachable
#: HOST141-200 ping HOST141
[root@kube-worker1 ~]# docker exec c200 ping 192.168.191.141

针对主机与本机容器不能互通的问题,可以增加一张默认的网卡:Multiple Docker Networks

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#先通过默认网络创建
[root@kube-worker1 ~]# docker run --name c200 -tid busybox /bin/sh                                   
47b7c1813b95cbec471b1a6de6a870e5537cfa70d54120873a5edb4e444b373b
#然后连接pub_net!
[root@kube-worker1 ~]# docker network connect --ip=192.168.191.200 pub_net c200        
[root@kube-worker1 ~]# docker exec c200 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:2/64 scope link 
       valid_lft forever preferred_lft forever
16: eth1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:c0:a8:bf:c8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.191.200/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c0ff:fea8:bfc8/64 scope link 
       valid_lft forever preferred_lft forever
       

方式1:

与主机的通信,通过 172.18.0.0/24 的网络。其他的通过 192.168.191.0/24 。还是感觉有点鸡肋!!

方式2:

增加route:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#route add -host $container_ip gw $lan_router_ip $if_device_nic2

[root@kube-worker1 ~]# route add -net 192.168.191.200 gw 172.18.0.1 netmask 255.255.255.255 dev docker0
[root@kube-worker1 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.191.2   0.0.0.0         UG    100    0        0 ens33
172.17.3.0      192.168.191.140 255.255.255.0   UG    100    0        0 ens33
172.17.4.0      0.0.0.0         255.255.255.0   U     425    0        0 kbr0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.191.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
192.168.191.200 172.18.0.1      255.255.255.255 UGH   0      0        0 docker0
[root@kube-worker1 ~]# ping 192.168.191.200
PING 192.168.191.200 (192.168.191.200) 56(84) bytes of data.
64 bytes from 192.168.191.200: icmp_seq=1 ttl=64 time=0.239 ms
64 bytes from 192.168.191.200: icmp_seq=2 ttl=64 time=0.106 ms
^C
--- 192.168.191.200 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.106/0.172/0.239/0.067 ms

通过操作与pipework比较,互有优劣:

  • pipework会创建网卡,然后所有的ip都是互通的,但是绑定、还得把主机的ip配置到br0上。
  • 而docker-network的方式与主机互通需要做额外的配置。

–END

Docker多主机网络配置 - Pipework

前面使用nat+route的方式手动连通两台机器上的docker容器。pipework是通过脚本的方式(手动)设置网络以及修改路由来进行配置的。

参考:

2、将物理网卡桥接到虚拟网桥,使得容器与宿主机配置在同一网段下

在各个宿主机上都建立一个新虚拟网桥设备br0,将各自物理网卡eth0桥接br0上,eth0的IP地址赋给br0;同时修改Docker daemon的DOCKER_OPTS,设置-b=br0(替代docker0),并限制Container IP地址的分配范围为同物理段地址(–fixed-cidr)。重启各个主机的Docker Daemon后,处于与宿主机在同一网段的Docker容器就可以实现跨主机访问了。这个方案同样存在局限和扩展性差的问题:比如需将物理网段的地址划分 成小块,分布到各个主机上,防止IP冲突;子网划分依赖物理交换机设置;Docker容器的主机地址空间大小依赖物理网络划分等。

原理就是建立一条连接link,一端 在主机 一端 在容器 ;然后手动配置容器ip和路由;最后把主机Ethernet和新建的Bridge桥接连接到物理网络。

容器的ip地址和主机的ip地址在一个网段内,所以在同一交换机下的所有主机、里面的容器都互通。

查看原网络的信息:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@kube-worker1 ~]# nmcli d show ens33 
GENERAL.DEVICE:                         ens33
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         00:0C:29:2E:67:4D
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     ens33
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/0
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         192.168.191.141/24
IP4.GATEWAY:                            192.168.191.2
IP4.ROUTE[1]:                           dst = 172.17.3.0/24, nh = 192.168.191.140, mt = 100
IP4.DNS[1]:                             192.168.191.2
IP4.DOMAIN[1]:                          localdomain
IP6.ADDRESS[1]:                         fe80::3995:4490:e2e7:1d0f/64
IP6.GATEWAY:                            

安装pipework

1
2
git clone https://github.com/jpetazzo/pipework
cp ~/pipework/pipework /usr/local/bin/

运行docker

1
2
3
4
5
6
7
8
9
10
11
12
13
#设置ip转发
echo 1 > /proc/sys/net/ipv4/ip_forward

vi /etc/sysctl.conf
net.ipv4.ip_forward = 1  

NAME=test1

#如不需要安装软件,可以加 --net none
docker run -itd --name $NAME centos /bin/bash

#docker ps -a -f name=$NAME | grep $NAME && docker start $NAME 
#docker exec test1 yum install -y iproute net-tools 

配置网络

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
function docker_container_ip () {
  local name=$1
  local ip=$2
  local gateway=${3:-$GATEWAY}
  
  pipework br0 $name $ip@$gateway
  #docker exec $name ifconfig 
  #docker exec $name route -n 
}

function docker_hosted_bridge_network_reset () {
  local ip=$1
  local gateway=$2
  local iface=$3
  
  if nmcli d show $iface | grep -i ethernet ; then
    #把地址给网桥,然后把ethernet和bridge连起来:(SSH连接操作的话,需要一条命令搞定!修改br0地址后route会变)
    ip addr add $ip dev br0 ; \
    ip addr del $ip dev $iface ; \
    brctl addif br0 $iface ; \
    #ip route del default ; \
    ip route add default via $gateway 
  fi 
  
  brctl show br0
}

GATEWAY=$( route -n | grep '^0.0.0.0' | awk '{print $2}' )
IFACE=$( route -n | grep '^0.0.0.0' | awk '{print $8}' )
HOSTED_IPADDR=$( ip addr show $IFACE | grep "inet " | awk '{print $2}' )

设置容器的IP:

1
2
3
4
5
NAME=test1
IP=192.168.191.210/24

docker_container_ip $NAME $IP $GATEWAY
docker_hosted_bridge_network_reset $HOSTED_IPADDR $GATEWAY $IFACE

上面的方式配置方式重启就失效的,可以通过写配置文件的方式来永久生效。如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@kube-worker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 
TYPE=Ethernet
DEVICE=ens33
BRIDGE=br0
[root@kube-worker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br0 
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.191.141
NETMASK=255.255.255.0
GATEWAY=192.168.191.2
DNS1=192.168.191.2

USERCTL=no

测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[root@kube-worker1 ~]# screen

  GATEWAY=$( route -n | grep '^0.0.0.0' | awk '{print $2}' )
  IFACE=$( route -n | grep '^0.0.0.0' | awk '{print $8}' )
  HOSTED_IPADDR=$( ip addr show $IFACE | grep "inet " | awk '{print $2}' )

  docker run -itd --name test21 centos /bin/bash
  docker run -itd --name test22 centos /bin/bash

  docker_container_ip test21 192.168.191.231/24 $GATEWAY
  docker_container_ip test22 192.168.191.232/24 $GATEWAY

  docker exec test21 ping  192.168.191.140
  docker exec test21 ping  192.168.191.141

[root@kube-master ~]# screen #会话"不断"

  docker_container_ip test11 192.168.191.221/24 $GATEWAY 
  docker_container_ip test12 192.168.191.222/24 $GATEWAY

  docker_hosted_bridge_network_reset $HOSTED_IPADDR $GATEWAY $IFACE

  docker exec test11 ping 192.168.191.233

注意:容器重启后,这些配置的网卡/路由都没有了,要重新配置:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@kube-worker1 ~]# docker stop test21
test21
[root@kube-worker1 ~]# docker start test21
test21
[root@kube-worker1 ~]# pipework route test21 show 
default via 172.18.0.1 dev eth0 
172.18.0.0/16 dev eth0  proto kernel  scope link  src 172.18.0.2 

[root@kube-worker1 ~]# docker_container_ip test21 192.168.191.231/24 $GATEWAY
[root@kube-worker1 ~]# pipework route test21 show                            
default via 192.168.191.2 dev eth1 
172.18.0.0/16 dev eth0  proto kernel  scope link  src 172.18.0.2 
192.168.191.0/24 dev eth1  proto kernel  scope link  src 192.168.191.231 

[root@kube-worker1 ~]# docker exec test21 ping 192.168.191.140

了解原理后,操作起来还是比较容易的。就是每次重启都要重新配置比较烦。可以写成脚本,启动docker容器的时刻就执行下网络配置。

pipework还可以用来配置vlan,暂时没这个需求,并且基本的操作都类似就没有实际操作了。

话说, pipework还可以用来创建多网卡的容器。用docker network connect其实更简单。

后记

除了通过pipework来实现共享物理网络外,docker network也可以实现这个功能:

1
2
3
4
5
6
7
8
#中间会导致网络断掉,一条命令搞定才行
docker network create --gateway=192.168.191.141 --subnet 192.168.191.0/24 --aux-address "DefaultGatewayIPv4=192.168.191.2" -o com.docker.network.bridge.name=br-home-net homenet ; \
ip addr del 192.168.191.141/24 dev ens33 ; \
brctl addif br-home-net ens33 

#主机不上外网可以不加
ip route add default via 192.168.191.2 ; 
echo "nameserver 114.114.114.114" >>/etc/resolv.conf ; 
  • 测试 docker-net + bridge:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
[root@kube-worker1 ~]# ip a
...
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br-home-net state UP qlen 1000
    link/ether 00:0c:29:2e:67:4d brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:44:ef:32:28 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
11: br-home-net: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:84:97:c2:25 brd ff:ff:ff:ff:ff:ff
    inet 192.168.191.141/24 scope global br-home-net
       valid_lft forever preferred_lft forever
    inet6 fe80::42:84ff:fe97:c225/64 scope link 
       valid_lft forever preferred_lft forever
       
[root@kube-worker1 ~]# docker run -tid --name c200 --net homenet --ip 192.168.191.200 busybox /bin/sh 
2579c2ddd18d23322eb1e145ad630205933dbc527b8981169ec6b125da8d8f1e

[root@kube-worker1 ~]# docker exec -ti c200 sh 
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:c0:a8:bf:c8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.191.200/24 scope global eth0
       valid_lft forever preferred_lft forever
/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.191.2   0.0.0.0         UG    0      0        0 eth0
192.168.191.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
/ # ping baidu.com 
PING baidu.com (111.13.101.208): 56 data bytes
64 bytes from 111.13.101.208: seq=0 ttl=128 time=48.225 ms
^C
--- baidu.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 48.225/48.225/48.225 ms
/ # ping 192.169.191.140
PING 192.169.191.140 (192.169.191.140): 56 data bytes
^C
--- 192.169.191.140 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
/ # ping 192.168.191.140
PING 192.168.191.140 (192.168.191.140): 56 data bytes
64 bytes from 192.168.191.140: seq=0 ttl=64 time=2.572 ms
64 bytes from 192.168.191.140: seq=1 ttl=64 time=1.076 ms
^C
--- 192.168.191.140 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.076/1.824/2.572 ms
/ # ping 192.168.191.141
PING 192.168.191.141 (192.168.191.141): 56 data bytes
64 bytes from 192.168.191.141: seq=0 ttl=64 time=0.474 ms
64 bytes from 192.168.191.141: seq=1 ttl=64 time=0.138 ms
^C
--- 192.168.191.141 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.138/0.306/0.474 ms
/ # ping 192.168.191.1
PING 192.168.191.1 (192.168.191.1): 56 data bytes
64 bytes from 192.168.191.1: seq=0 ttl=128 time=1.068 ms
64 bytes from 192.168.191.1: seq=1 ttl=128 time=0.603 ms
^C
--- 192.168.191.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.603/0.835/1.068 ms

–END

STAF Start Guide

STAF(Software Testing Automation Framework) 是用来构建自动化测试的框架,用来部署测试环境。包括跨平台自动化部署的各种组件:config、process、handle、fs、log、monitor、queue、var、security等各种服务,还可以轻松的扩展。机器通过STAFProc启动的端口来进行通讯(set、query等)。

STAF的作用实际上就是提供了机器之间的通信通道并提供基于这个通道的基础服务。

安装

直接用 InstallAnywhere(IA) 的方式安装。文档介绍了整个安装过程中做的所有事情,非常非常的详细。下面针对Linux的安装配置步骤:

  • 安装配置项(选项),三种方式选一种: 交互方式 文件方式 默认值
  • 环境变量:使用InstallAnywhere的方式会把环境变量配置好,文档中介绍了使用的环境变量。
  • 启动:shell脚本、开机启动(centos7的服务配置可以参考文档的[Fedora 15 and later]部分)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
#安装(默认值)
./STAF3426-setup-linux-amd64-NoJVM.bin -i silent -DACCEPT_LICENSE=1

#启动
su -
/usr/local/staf/bin/STAFProc
或者
./startSTAFProc.sh 

#测试
[root@cdb1 staf]# staf local ping ping
Response
--------
PONG
[root@cdb1 staf]# staf local process start command ifconfig
Response
--------
5
#staf远程控制 产生的输出是在运行的机器上的!
[root@cdb1 staf]# less nohup.out 

Machine          : cdb1
Machine nickname : cdb1
Startup time     : 20171006-08:50:55

STAFProc version 3.4.26 initialized
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.193.101  netmask 255.255.192.0  broadcast 192.168.255.255
        inet6 fe80::932a:7b98:cc20:f791  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::9ba3:ce9e:cc9c:477d  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::7d8c:b318:d441:70b2  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:92:26:93  txqueuelen 1000  (Ethernet)
        RX packets 11020400  bytes 3439058707 (3.2 GiB)
        RX errors 0  dropped 717  overruns 0  frame 0
        TX packets 9067587  bytes 3110097659 (2.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 70483  bytes 10164575 (9.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 70483  bytes 10164575 (9.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


#多机测试
[root@cdb2 staf]# vi bin/STAF.cfg
trust machine 192.168.193.101 level 5

[root@cdb1 staf]# staf cdb2 ping ping 
[root@cdb1 staf]# staf cdb2 process start command ifconfig 
[root@cdb2 staf]# less nohup.out 


#安全退出
STAF local SHUTDOWN SHUTDOWN
C:\STAF\bin\STAF.exe local SHUTDOWN SHUTDOWN

Getting Started

http://staf.sourceforge.net/current/STAFGS.pdf

STAFGS 对所有的知识点都做了简单的介绍和归纳。更详细的需要看 User Guide 文档。

STAF is an Open Source automation framework designed around the idea of reusable components. It is intended to make it easier to create automated testcases and workloads. STAF can help you increase the efficiency, productivity, and quality of your testing by improving your level of automation and reuse in your individual testcases as well as your overall test environment.

STAF operates in a peer-to-peer environment; in other words, there is no client-server hierarchy among machines running STAF.

Basic STAF Concepts

STAF Instances Since multiple instances of STAF can be run at the same time on the same system, a STAF Instance name is used to specify a name for each STAF instance. You specify the instance name to be used by setting the environment variable STAF_INSTANCE_NAME. The default instance name is “STAF”.

A basic description of each level follows

1
2
3
4
5
6
Level 0 - No access
Level 1 - Restricted access. Only PING and helps available.
Level 2 - Limited access. Only query/view facilities available.
Level 3 - Standard access. Non-destructive updates allowed, e.g., logging.
Level 4 - Advanced access. Update abilities, e.g., copying files, deleting log files.
Level 5 - All access, e.g., SHUTDOWN, Process invocation, Trust definition manipulation

STAF requests submitted from the command line are generally used to query information from STAF services.

STAF Commands

You can also start STAFProc by simply typing STAFProc at a command prompt window. ( /usr/local/staf/bin is in your PATH )

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
STAF  <Endpoint>/[<Interface>://]<System Identifier>[@<Port>]  <Service> <Request>

[root@cdb1 staf]# staf local ping ping 
Response
--------
PONG
#没运行
[root@cdb1 staf]# staf cdb2 ping ping 
Error submitting request, RC: 16
Additional info
---------------
STAFConnectionProviderConnect: Error performing test read on connected endpoint: recv() RC=111: 22, Endpoint: ssl://cdb2
[root@cdb1 staf]# staf cdb2 ping ping 
Response
--------
PONG

[root@cdb1 staf]# staf local service help 
Response
--------
*** SERVICE Service Help ***

LIST    [ SERVICES | SERVICELOADERS | AUTHENTICATORS |
          REQUESTS <[PENDING] [COMPLETE] [LONG]> | [SUMMARY] ]
QUERY   SERVICE <Service Name> | SERVICELOADER <ServiceLoader Name> |
        AUTHENTICATOR <Authenticator Name> | REQUEST <Request Number>
ADD     SERVICE <Service Name> LIBRARY <Library Name>
        [EXECUTE <Executable>] [OPTION <Name[=Value]>]...
        [PARMS <Parameters>]
REMOVE  SERVICE <Service Name>
FREE    REQUEST <Request Number> [FORCE]
HELP

[root@cdb1 staf]# staf local shutdown help 
Response
--------
*** SHUTDOWN Service Help ***

SHUTDOWN

NOTIFY REGISTER   [MACHINE <Machine>] [HANDLE <Handle> | NAME <Name>]
                  [PRIORITY <Priority>]
NOTIFY UNREGISTER [MACHINE <Machine>] [HANDLE <Handle> | NAME <Name>]
                  [PRIORITY <Priority>]
NOTIFY LIST

HELP

>>>> The information returned by Help show us the options we can place after "STAF local shutdown ....." 
in command requests

[root@cdb1 staf]# staf local service list 
Response
--------
Name      Library    Executable
--------- ---------- ----------
CONFIG    <Internal> <None>    
DELAY     <Internal> <None>    
DIAG      <Internal> <None>    
ECHO      <Internal> <None>    
FS        <Internal> <None>    
HANDLE    <Internal> <None>    
HELP      <Internal> <None>    
LIFECYCLE <Internal> <None>    
MISC      <Internal> <None>    
PING      <Internal> <None>    
PROCESS   <Internal> <None>    
QUEUE     <Internal> <None>    
SEM       <Internal> <None>    
SERVICE   <Internal> <None>    
SHUTDOWN  <Internal> <None>    
TRACE     <Internal> <None>    
TRUST     <Internal> <None>    
VAR       <Internal> <None>    

[root@cdb1 staf]# staf local var list 
Response
--------
STAF/Config/BootDrive             : /
STAF/Config/CodePage              : UTF-8
STAF/Config/ConfigFile            : /usr/local/staf/bin/STAF.cfg
STAF/Config/DefaultAuthenticator  : none
STAF/Config/DefaultInterface      : ssl
STAF/Config/InstanceName          : STAF
STAF/Config/Machine               : cdb1
STAF/Config/MachineNickname       : cdb1
STAF/Config/Mem/Physical/Bytes    : 3974971392
STAF/Config/Mem/Physical/KB       : 3881808
STAF/Config/Mem/Physical/MB       : 3790
STAF/Config/OS/MajorVersion       : 3.10.0-693.el7.x86_64
STAF/Config/OS/MinorVersion       : #1 SMP Tue Aug 22 21:09:27 UTC 2017
STAF/Config/OS/Name               : Linux
STAF/Config/OS/Revision           : x86_64
STAF/Config/Processor/NumAvail    : 4
STAF/Config/Sep/Command           : ;
STAF/Config/Sep/File              : /
STAF/Config/Sep/Line              : 

STAF/Config/Sep/Path              : :
STAF/Config/STAFRoot              : /usr/local/staf
STAF/Config/StartupTime           : 20171006-10:33:42
STAF/DataDir                      : /usr/local/staf/data/STAF
STAF/Env/_                        : /usr/bin/nohup
STAF/Env/CLASSPATH                : /usr/local/staf/lib/JSTAF.jar:/usr/local/staf/samples/demo/STAFDemo.jar:/usr/local/staf/samples/demo/STAFDemo.jar:/usr/local/staf/lib/JSTAF.jar:
STAF/Env/HISTCONTROL              : ignoredups
STAF/Env/HISTSIZE                 : 1000
STAF/Env/HOME                     : /root
STAF/Env/HOSTNAME                 : cdb1
STAF/Env/JAVA_HOME                : /usr/local/jdk
STAF/Env/LANG                     : en_US.UTF-8
STAF/Env/LD_LIBRARY_PATH          : /usr/local/staf/lib:/usr/local/staf/lib:
STAF/Env/LESSOPEN                 : ||/usr/bin/lesspipe.sh %s
STAF/Env/LOGNAME                  : root
STAF/Env/LS_COLORS                : rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
STAF/Env/MAIL                     : /var/spool/mail/root
STAF/Env/PATH                     : /usr/local/staf/bin:/usr/local/staf/bin:/usr/local/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
STAF/Env/PWD                      : /usr/local/staf
STAF/Env/SELINUX_LEVEL_REQUESTED  : 
STAF/Env/SELINUX_ROLE_REQUESTED   : 
STAF/Env/SELINUX_USE_CURRENT_RANGE: 
STAF/Env/SHELL                    : /bin/bash
STAF/Env/SHLVL                    : 2
STAF/Env/SSH_AUTH_SOCK            : /tmp/ssh-yEmD907zdB/agent.28259
STAF/Env/SSH_CLIENT               : 192.168.193.10 34774 22
STAF/Env/SSH_CONNECTION           : 192.168.193.10 34774 192.168.193.101 22
STAF/Env/SSH_TTY                  : /dev/pts/2
STAF/Env/STAF_INSTANCE_NAME       : STAF
STAF/Env/STAFCONVDIR              : /usr/local/staf/codepage
STAF/Env/TERM                     : vt100
STAF/Env/USER                     : root
STAF/Env/XDG_DATA_DIRS            : /root/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/
STAF/Env/XDG_RUNTIME_DIR          : /run/user/0
STAF/Env/XDG_SESSION_ID           : 2080
STAF/Version                      : 3.4.26

>>>> STAF predefines many useful variables, including information about the machine's Operating System 
and File/Line/Path separators. 

[root@cdb1 staf]# staf local var resolve system string {STAF/Config/Sep/File}
Response
--------
/

[root@cdb1 staf]# staf local handle list handles
Response
--------
Handle Handle Name                     State      Last Used Date-Time
------ ------------------------------- ---------- -------------------
1      STAF_Process                    InProcess  20171006-10:33:42  
2      STAF/Service/STAFServiceLoader1 InProcess  20171006-10:33:42  
11     STAF/Client                     Registered 20171006-10:41:39  

>>>> handle 1 is assigned to STAFProc. Each of the STAF/Client requests represent each 
of the three "STAF local handle list handles" commands you submitted. Note that each request is assigned a new handle
number, and that the previous handles have been deleted 

Configuring STAF

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
[root@cdb1 staf]# staf local log log  machine logname log1 level info message test-message
Response
--------

[root@cdb1 staf]# staf local log list machines
Response
--------
cdb1

[root@cdb1 staf]# staf local log query machine cdb1 logname log1 
Response
--------
Date-Time         Level Message     
----------------- ----- ------------
20171006-10:54:33 Info  test-message


#配置
[root@cdb1 staf]# echo "MACHINENICKNAME cdb1.dev" >>bin/STAF.cfg 
[root@cdb1 staf]# staf local shutdown shutdown 
[root@cdb1 staf]# ./startSTAFProc.sh 

[root@cdb1 staf]# staf local log log  machine logname log3 level info message test-message
Response
--------

[root@cdb1 staf]# staf local log list machines
Response
--------
cdb1
cdb1.dev
[root@cdb1 staf]# staf local log query machine cdb1.dev logname log3
Response
--------
Date-Time         Level Message     
----------------- ----- ------------
20171006-10:57:59 Info  test-message

>>>> This primarily effects the data stored by
services such as the Log and Monitor services, which store data based on the machine from which it came by using the
STAF/Config/MachineNickname system variable as part of the directory path when creating logs and monitor data. By
allowing the STAF/Config/MachineNickname system variable to be overridden, it allows you to better manage your
data.
Note that the machine nickname is not used to communicate with other systems and does not have any effect on trust. 


#配置
echo "SET DATADIR /data" >>bin/STAF.cfg
...
SET SYSTEM VAR Test/TestABC=websphere
SET SYSTEM VAR Test/TestXYZ=150

restart STAFProc and from a command prompt, try the STAF local var list command

TRUST LEVEL 2 DEFAULT
TRUST LEVEL 5 MACHINE 192.168.193.*
TRUST LEVEL 4 MACHINE tcp://9.3.41.*
TRUST LEVEL 5 MACHINE tcP://9.41.53.147

[root@cdb1 staf]# staf local trust list 
Response
--------
Type    Entry             Trust Level
------- ----------------- -----------
Default <None>            2          
Machine *://192.168.193.* 5          
Machine local://local     5          
Machine tcp://9.3.41.*    4          
Machine tcP://9.41.53.147 5          

Using the Help Service

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@cdb1 staf]# staf local help help 
Response
--------
*** HELP Service Help ***

REGISTER   SERVICE <Name> ERROR <Number> INFO <String> DESCRIPTION <String>

UNREGISTER SERVICE <Name> ERROR <Number>

[SERVICE <Name>] ERROR <Number>

LIST SERVICES | [SERVICE <Name>] ERRORS

HELP

#错误码详情
[root@cdb1 staf]# staf local error list 
Error submitting request, RC: 2
Additional info
---------------
error
[root@cdb1 staf]# staf local help error 2
Response
--------
Description: Unknown service
Details    : You have tried to submit a request to a service that is unknown to STAFProc.  Verify that you have correctly registered the service.

Registering STAF Services

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
http://staf.sourceforge.net/getservices.php 下载EventV315.tar
[root@cdb1 staf]# tar xf EventV315.tar 

[root@cdb1 staf]# vi bin/STAF.cfg 
...
service Event library JSTAF execute /usr/local/staf/event/STAFEvent.jar

[root@cdb1 staf]# staf local shutdown shutdown 
[root@cdb1 staf]# ./startSTAFProc.sh 
[root@cdb1 staf]# staf local service list 
Response
--------
Name      Library    Executable                         
--------- ---------- -----------------------------------
...
EVENT     JSTAF      /usr/local/staf/event/STAFEvent.jar
...

#帮助文档: http://staf.sourceforge.net/current/event.htm

STAF Demo( 前提得安装上面的EVENT,并且STAFProc也得在图形界面Terminal启动!!

  1. each machine must give the other machine a TRUST level of 5
  2. samples\demo\STAFDemo.jar在CLASSPATH里面.
  3. java STAFDemoController是图形界面程序!!
  4. 弹出的窗口【An Arbitrary Process:Handle X】的标题(界面没看到的话,可能是被遮住了)。

However, the STAFProcess window should be displayed on your remote machine。

具体的代码解析查阅【8.2. STAF Demo Code - Leveraging STAF】这个章节。

User’s Guide

STAX Getting Started

http://staf.sourceforge.net/current/staxgs.pdf

Verify that the CLASSPATH environment variable contains the JSTAF.jar file. JSTAF.jar contains the STAF Java APIs to communicate with STAF from Java programs and is required to register STAF services written in Java.

1
2
3
4
5
6
[root@cdb1 staf]# tar xf ~/STAXV3517.tar 
[root@cdb1 staf]# vi bin/STAF.cfg 
...
SERVICE STAX LIBRARY JSTAF EXECUTE {STAF/Config/STAFRoot}/stax/STAX.jar OPTION J2=-Xmx2048m
SERVICE EVENT LIBRARY JSTAF EXECUTE {STAF/Config/STAFRoot}/stax/STAFEvent.jar
SET MAXQUEUESIZE 10000

If you do not want to include the JVM bin directory in your PATH, then you can use the “OPTION JVM=xxx” to specify which Java executable to use for the services.

1
2
3
4
5
6
7
8
9
[root@cdb1 staf]# staf local stax version 
Response
--------
3.5.17
[root@cdb1 staf]# staf local stax version jython
Response
--------
2.5.2-staf-v1
[root@cdb1 staf]# 

Errors that occur when running the STAX service will be stored in its JVM log. This log is data/STAF/lang/java/jvm/STAFJVM1/JVMLog.1 in your root STAF directory

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
source /etc/profile
cd stax
java -jar STAXMon.jar

>>>> You use script elements within your STAX jobs to define Python variables and execute Python code. However, also note that in most cases, all of
the element content and element attributes in your STAX jobs will also be evaluated as Python code. 

<script>testName = 'CoolTest1'</script>
<testcase name="testName">
<testcase name="'%s Part A' % testName">
<testcase name="'%s Part A on machine %s' % (testName, machineName)">

#DoesNothing.xml 
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE stax SYSTEM "stax.dtd">

<stax>

  <defaultcall function="main"/>

  <function name="main">
    <nop/>
  </function>

</stax>


#generate dtd for xml editor
set STAF_QUIET_MODE=1 (or if on Unix: export STAF_QUIET_MODE=1)
STAF local STAX GET DTD > stax.dtd
set STAF_QUIET_MODE= (or if on Unix: unset STAF_QUIET_MODE)

#RunNotepadProcess.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE stax SYSTEM "stax.dtd">

<stax>

  <defaultcall function="main"/>

  <function name="main">

    <process>
      <location>'local'</location>
      <command>'notepad'</command>
    </process>

  </function>

</stax>

结束任务,打开的程序也会被kill掉!!!

–END

连接树莓派 - Respberrypi

启动树莓派后,总得连上去操作才能告诉它做事情。最常用的就是通过SSH远程控制,前提是能连上网络、知道树莓派获取到的地址。下来把了解到的,以及自实践的连接方式做下小结。

通过路由器

网线、无线连接后,通过 显示器 路由管理界面 获取树莓派的地址(在管理web界面有明确的respberrypi的字样)。

通过USB

  1. USB转COM

USB连接电脑,连树莓派的GPIO对应的针。然后通过COM口协议与树莓派通信

  1. USB共享网络

使用手机的USB共享网络。

USB连树莓派,Micro口连手机。手机上打开USB网络共享,树莓派中会建立一个usb0的网卡。这样就能通过这个网卡进行上网了(网上也有说同时打开wifi热点,没啥用啊,usb和wlan是两个不同的网段)。

手机上安装一个ssh的工具(juicessh等),先连上本地的shell,然后执行 cat /proc/net/arp 或者 busybox arp -a 查看与 rndis0 同一个网段的ip(一般就是连接到树莓派的地址了)。

在手机上安装一个IP扫描软件应该也行,但 通过ARP是最简单最高效的方式了。

注意:busybox感觉像一个工具集,包含了很多linux的命令,并且有些命令参数比系统提供的更全,如 tar -j 。可以用 busybox --help 查看帮助。

网线互联

一跟网线直接连电脑和树莓派。

互传数据应该有用。当前感觉,这种方式没啥优势,有点鸡肋。上网比较麻烦:手动设置IP、域名解析、还要在电脑上面搞网卡绑定。

下面自动获取的方法(没试,应该是可以的吧):

网线直接把树莓派与电脑连接起来,电脑需要连wifi(如果不连接无法使用网络共享让树莓派获取到ip地址),在 设置->网络->wlan->网络与共享中心->wlan->属性->共享->允许​其它用户通过它来连接->确定。

稍等片刻树莓派应该就获取到一个ip地址了,此时打开命令提示符(我用的Xshell),运行arp -a命令,应该就可以看到​一个局域网段,如192.168.xx.1,通常以192.168开头,最后一位是1的那个接口,下面多出来的一条动态记录就是树莓派的。

参考:

键盘输入、手机显示

前提:安装screen。

  • 连接USB键盘,启动树莓派。
  • 键盘盲打登录(输入:root回车centos回车screen -S pi)。
  • 然后手机上ssh连接,进入pi的会话(screen -x pi)

这样就能用键盘敲、手机看了!觉得挺好玩的。

重新折腾树莓派

  • 安装centos7: 密码root/centos

http://www.21ic.com/evm/trick/201605/675705.htm

  • 扩大容量:

http://blog.csdn.net/qq_20480611/article/details/48657827

fdisk删掉分区然后重新加,重启后执行resize2fs

  • 安装wiringpi
1
2
3
git clone git://git.drogon.net/wiringPi
cd wiringPi
./build
  • epel

https://hobo.house/2016/03/03/installing-centos-on-the-raspberry-pi-2/

1
2
3
4
5
6
7
8
cat > /etc/yum.repos.d/epel-unsigned.repo << EOF
[epel]
name=Epel rebuild for armhfp
baseurl=https://armv7.dev.centos.org/repodir/epel-pass-1/
enabled=1
gpgcheck=0

EOF

–END

两台主机的docker通过route互联互通

前面一直用k8s的flannel来建立主机间docker容器的互联,但是当仅有两台机器用来做测试的时刻,安装一个flannel也是挺纠结的:麻烦、还有未知的问题,起一个服务在那里总会有那么些担忧。

其实可以直接通过建立路由来实现两台机器间容器的互联互通:Docker多台宿主机间的容器互联-centos7(直接路由)

两台主机(centos7/docker-1.12.6):

  • 192.168.191.140 kube-master
  • 192.168.191.141 kube-worker1

安装/配置docker

这里不多讲了,参考 Kubeadm部署kubernetes 进行docker的安装。

建立新网卡,修改docker配置使用新网卡

  • 安装/更新依赖
1
yum install net-tools bridge-utils -y
  • 关防火墙、关selinux
1
2
3
4
5
6
7
setenforce 0

vi /etc/selinux/config
SELINUX=disabled

systemctl stop firewalld
systemctl disable firewalld
  • 设置ip转发
1
2
3
4
echo 1 > /proc/sys/net/ipv4/ip_forward

vi /etc/sysctl.conf
net.ipv4.ip_forward = 1  
  • 删docker0,建kbr0

先停docker!先停docker!先停docker!(好像docker会缓冲bridge的ip)

1
2
3
4
service docker stop
brctl addbr kbr0
ip link set dev docker0 down
ip link del dev docker0

下面的配置,两台机不同,如下:

192.168.191.140 kube-master 192.168.191.141 kube-worker1
vi /etc/sysconfig/network-scripts/ifcfg-kbr0
DEVICE=kbr0 DEVICE=kbr0
ONBOOT=yes ONBOOT=yes
BOOTPROTO=static BOOTPROTO=static
IPADDR=172.17.3.1 IPADDR=172.17.4.1
NETMASK=255.255.255.0 NETMASK=255.255.255.0
GATEWAY=172.17.3.0 GATEWAY=172.17.4.0
USERCTL=no USERCTL=no
TYPE=Bridge TYPE=Bridge
IPV6INIT=no IPV6INIT=no
———————————————– ——————————————————–
vi /etc/sysconfig/network-scripts/route-ens33 (ip对应的网卡名称)
172.17.4.0/24 via 192.168.191.141 dev ens33 172.17.3.0/24 via 192.168.191.140 dev ens33
———————————————– ——————————————————–

参考: Configuring Static Routes

  • 修改docker配置
1
2
3
4
vi /usr/lib/systemd/system/docker.service     
ExecStart=/usr/bin/dockerd --bridge=kbr0 

systemctl daemon-reload 
  • 重新启动

先起网卡!先起网卡!先起网卡!

1
2
3
service network restart

systemctl start docker

最终效果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
| 192.168.191.140 kube-master                                                   | 192.168.191.141 kube-worker1                            
|:------------------------------------------------------------------------------|:-------------------------------------------------------
| [root@kube-master ~]# ifconfig                                                | [root@kube-worker1 ~]# ifconfig
| ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500                   | ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
|         inet 192.168.191.140  netmask 255.255.255.0  broadcast 192.168.191.255|         inet 192.168.191.141  netmask 255.255.255.0  broadcast 192.168.191.255
|         inet6 fe80::1186:2fe5:9ee5:8790  prefixlen 64  scopeid 0x20<link>     |         inet6 fe80::3995:4490:e2e7:1d0f  prefixlen 64  scopeid 0x20<link>
|         ether 00:0c:29:40:2d:15  txqueuelen 1000  (Ethernet)                  |         ether 00:0c:29:2e:67:4d  txqueuelen 1000  (Ethernet)
|         RX packets 18010  bytes 10754845 (10.2 MiB)                           |         RX packets 19871  bytes 12247126 (11.6 MiB)
|         RX errors 0  dropped 0  overruns 0  frame 0                           |         RX errors 0  dropped 0  overruns 0  frame 0
|         TX packets 4797  bytes 475332 (464.1 KiB)                             |         TX packets 5647  bytes 561624 (548.4 KiB)
|         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0            |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
|                                                                               | 
| kbr1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500                            | kbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
|         inet 172.17.3.1  netmask 255.255.255.0  broadcast 172.17.3.255        |         inet 172.17.4.1  netmask 255.255.255.0  broadcast 172.17.4.255
|         ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)                  |         ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)
|         RX packets 179  bytes 13932 (13.6 KiB)                                |         RX packets 139  bytes 10492 (10.2 KiB)
|         RX errors 0  dropped 0  overruns 0  frame 0                           |         RX errors 0  dropped 0  overruns 0  frame 0
|         TX packets 43  bytes 3894 (3.8 KiB)                                   |         TX packets 36  bytes 3004 (2.9 KiB)
|         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0            |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
|                                                                               | 
| lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536                                  | lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
|         inet 127.0.0.1  netmask 255.0.0.0                                     |         inet 127.0.0.1  netmask 255.0.0.0
|         inet6 ::1  prefixlen 128  scopeid 0x10<host>                          |         inet6 ::1  prefixlen 128  scopeid 0x10<host>
|         loop  txqueuelen 1  (Local Loopback)                                  |         loop  txqueuelen 1  (Local Loopback)
|         RX packets 140  bytes 11644 (11.3 KiB)                                |         RX packets 215  bytes 18260 (17.8 KiB)
|         RX errors 0  dropped 0  overruns 0  frame 0                           |         RX errors 0  dropped 0  overruns 0  frame 0
|         TX packets 140  bytes 11644 (11.3 KiB)                                |         TX packets 215  bytes 18260 (17.8 KiB)
|         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0            |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
|                                                                               | 
| [root@kube-master ~]# route -n                                                | [root@kube-worker1 ~]# route -n 
| Kernel IP routing table                                                       | Kernel IP routing table
| Destination     Gateway         Genmask         Flags Metric Ref    Use Iface | Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
| 0.0.0.0         192.168.191.2   0.0.0.0         UG    100    0        0 ens33 | 0.0.0.0         192.168.191.2   0.0.0.0         UG    100    0        0 ens33
| 172.17.3.0      0.0.0.0         255.255.255.0   U     427    0        0 kbr1  | 172.17.3.0      192.168.191.140 255.255.255.0   UG    100    0        0 ens33
| 172.17.4.0      192.168.191.141 255.255.255.0   UG    100    0        0 ens33 | 172.17.4.0      0.0.0.0         255.255.255.0   U     425    0        0 kbr0
| 192.168.191.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33 | 192.168.191.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
| [root@kube-master ~]#                                                         | [root@kube-worker1 ~]# 
| [root@kube-master ~]# docker run -ti --rm busybox sh                          | [root@kube-worker1 ~]# docker run -ti --rm busybox sh                  
| / # ifconfig                                                                  | / # ifconfig
| eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:03:02                       | eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:04:02  
|           inet addr:172.17.3.2  Bcast:0.0.0.0  Mask:255.255.255.0             |           inet addr:172.17.4.2  Bcast:0.0.0.0  Mask:255.255.255.0
|           inet6 addr: fe80::42:acff:fe11:302/64 Scope:Link                    |           inet6 addr: fe80::42:acff:fe11:402/64 Scope:Link
|           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1                  |           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
|           RX packets:23 errors:0 dropped:0 overruns:0 frame:0                 |           RX packets:16 errors:0 dropped:0 overruns:0 frame:0
|           TX packets:15 errors:0 dropped:0 overruns:0 carrier:0               |           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
|           collisions:0 txqueuelen:0                                           |           collisions:0 txqueuelen:0 
|           RX bytes:1870 (1.8 KiB)  TX bytes:1222 (1.1 KiB)                    |           RX bytes:1296 (1.2 KiB)  TX bytes:648 (648.0 B)
|                                                                               | 
| lo        Link encap:Local Loopback                                           | lo        Link encap:Local Loopback  
|           inet addr:127.0.0.1  Mask:255.0.0.0                                 |           inet addr:127.0.0.1  Mask:255.0.0.0
|           inet6 addr: ::1/128 Scope:Host                                      |           inet6 addr: ::1/128 Scope:Host
|           UP LOOPBACK RUNNING  MTU:65536  Metric:1                            |           UP LOOPBACK RUNNING  MTU:65536  Metric:1
|           RX packets:0 errors:0 dropped:0 overruns:0 frame:0                  |           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
|           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0                |           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
|           collisions:0 txqueuelen:1                                           |           collisions:0 txqueuelen:1 
|           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                              |           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
| / # ping 172.17.4.2                                                           | 
| PING 172.17.4.2 (172.17.4.2): 56 data bytes                                   | / # ping 172.17.3.2
| 64 bytes from 172.17.4.2: seq=0 ttl=62 time=2.598 ms                          | PING 172.17.3.2 (172.17.3.2): 56 data bytes
| 64 bytes from 172.17.4.2: seq=1 ttl=62 time=1.569 ms                          | 64 bytes from 172.17.3.2: seq=0 ttl=62 time=1.421 ms
| 64 bytes from 172.17.4.2: seq=2 ttl=62 time=1.194 ms                          | 64 bytes from 172.17.3.2: seq=1 ttl=62 time=1.446 ms
| ^C                                                                            | ^C
| --- 172.17.4.2 ping statistics ---                                            | --- 172.17.3.2 ping statistics ---
| 3 packets transmitted, 3 packets received, 0% packet loss                     | 2 packets transmitted, 2 packets received, 0% packet loss
| round-trip min/avg/max = 1.194/1.787/2.598 ms                                 | round-trip min/avg/max = 1.421/1.433/1.446 ms
| 
|-------------------------------------------------------------------------------|--------------------------------------------------------

效果还不错,什么都没有安装route两台机器的docker就互联互通了。二三台机器使用这种方式最省事的,并且理论上效率也是最高的。

其他参考

–END