浅谈DockerOverlay网络

794 阅读6分钟

这是我参与更文挑战的第10天,活动详情查看: 更文挑战

DockerOverlay网络

Overlay是一种覆盖在主机上的虚拟网络,我理解的overlay他需要借助插件或工具完成对数据的保存和转发。我们这里使用etcd来完成overlay的跨主机容器通信

1. 安装etcd(这里我只启动两个节点的etcd,也可单机模式)

我这里选择用安装包方式,你也可以选择yum

wget https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz

scp etcd-v3.5.0-linux-amd64.tar.gz  root@172.16.3.142:/root

参考官网

以下操作两个主机均需要进行

tar -zxvf etcd-v3.5.0-linux-amd64.tar.gz
cd etcd-v3.5.0-linux-amd64
cp etcdctl /usr/local/bin/

我这里遇到一个坑,一定要加上--enable-v2否则会报错,无法创建overlay网络

nohup ./etcd --name docker-node1 --initial-advertise-peer-urls http://172.16.3.141:2380 --listen-peer-urls http://172.16.3.141:2380 --listen-client-urls http://172.16.3.141:2379,http://127.0.0.1:2379 --advertise-client-urls http://172.16.3.141:2379 --initial-cluster-token etcd-cluster --initial-cluster docker-node1=http://172.16.3.141:2380,docker-node2=http://172.16.3.142:2380 --enable-v2 --initial-cluster-state new 2>&1 &

nohup ./etcd --name docker-node2 --initial-advertise-peer-urls http://172.16.3.142:2380 --listen-peer-urls http://172.16.3.142:2380 --listen-client-urls http://172.16.3.142:2379,http://127.0.0.1:2379 --advertise-client-urls http://172.16.3.142:2379 --initial-cluster-token etcd-cluster --initial-cluster docker-node1=http://172.16.3.141:2380,docker-node2=http://172.16.3.142:2380 --enable-v2 --initial-cluster-state new 2>&1 &


etcdctl member list 查看一下节点信息

分别添加配置,指定外部存储使用etcd

vi /etc/docker/daemon.json

{
   "cluster-store":"etcd://172.16.3.141:2379",
   "cluster-advertise":"172.16.3.141:2375"
}

一定要重启下

systemctl daemon-reload
service  docker restart

2. 创建overlay network

# 也可指定subnet
docker  network  create  -d overlay  etcdnet

可以看到另一台也出现了etcdnet

docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
810ca8a6e002   bridge    bridge    local
19234a37ec11   etcdnet   overlay   global

我们来看一下etcd中存储的数据格式,可以看到节点信息和子网的配置都有了

[root@localhost etcd-v3.5.0-linux-amd64]# etcdctl ls
/docker
[root@localhost etcd-v3.5.0-linux-amd64]# etcdctl ls /docker
/docker/nodes
/docker/network
[root@localhost etcd-v3.5.0-linux-amd64]# etcdctl ls /docker/nodes
/docker/nodes/172.16.3.141:2375
/docker/nodes/172.16.3.142:2375
[root@localhost etcd-v3.5.0-linux-amd64]# etcdctl ls /docker/network/v1.0
/docker/network/v1.0/endpoint
/docker/network/v1.0/network
/docker/network/v1.0/endpoint_count
/docker/network/v1.0/ipam
/docker/network/v1.0/idm
/docker/network/v1.0/overlay
[root@localhost etcd-v3.5.0-linux-amd64]# etcdctl get /docker/network/v1.0/overlay/network/19234a37ec11b48a6d1491d99264825ee6452fc3007d0841429ec5ca85c94d64
{"mtu":0,"secure":false,"subnets":[{"SubnetIP":"10.0.9.0/24","GwIP":"10.0.9.1/24","Vni":0}]}

3. 使用overlay network创建容器

在141上启动busybox0 在142上启动busybox1

docker run -it -d --name busybox0 --network etcdnet busybox
docker run -it -d --name busybox1 --network etcdnet busybox

查看下两个节点的ip分配情况

docker network inspect etcdnet
"Containers": {
            "67845aca4909ec68518ee09a5de72c12472bdd5ab044c3df180df85012b381e8": {
                "Name": "busybox0",
                "EndpointID": "e90dff8605bbe7c7f7db508a6b576d738ddb5faf1e7564191c068134c8c79312",
                "MacAddress": "02:42:0a:00:09:02",
                "IPv4Address": "10.0.9.2/24",
                "IPv6Address": ""
            },
            "ep-fc9be25e3f8d99df596d59e283fd0237834bb6ace2b72f3182181f132fd51bbd": {
                "Name": "busybox1",
                "EndpointID": "fc9be25e3f8d99df596d59e283fd0237834bb6ace2b72f3182181f132fd51bbd",
                "MacAddress": "02:42:0a:00:09:03",
                "IPv4Address": "10.0.9.3/24",
                "IPv6Address": ""
            }
        },

我们进入容器看一下网卡信息,发现有两个网卡,这里eth0是overlay的eth1是bridge,从上篇文章的docker0可知默认的bridgeIp不应该是172.17.0.1网段上分配的吗?172.18.0.2又是哪里来的?

[root@localhost etcd-v3.5.0-linux-amd64]# docker exec -it busybox0 sh -c 'ifconfig'
eth0      Link encap:Ethernet  HWaddr 02:42:0A:00:09:02
          inet addr:10.0.9.2  Bcast:10.0.9.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 02:42:AC:12:00:02
          inet addr:172.18.0.2  Bcast:172.18.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:836 (836.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

我们在运行该容器的主机上查看一下network,发现突然多了一个网络配置叫docker_gwbridge是一个bridge类型的

[root@localhost etcd-v3.5.0-linux-amd64]# docker network ls
NETWORK ID     NAME              DRIVER    SCOPE
cd915be3ae23   bridge            bridge    local
84d1aff11b90   docker_gwbridge   bridge    local
19234a37ec11   etcdnet           overlay   global

看一下docker_gwbridge对应的详细信息,和容器内的eth1是可以对应上的

docker network inspect 84d1aff11b90
"Containers": {
            "67845aca4909ec68518ee09a5de72c12472bdd5ab044c3df180df85012b381e8": {
                "Name": "gateway_a6711fa937d4",
                "EndpointID": "4ec1a016078d849ae46012c54a712caa7cc64312ebbd78facaecb15e387bda21",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },

4. overlay network测试

在141机器上运行的busbox0去ping142机器上运行的busbox1,可以看到无论是ping容器名还是ip都可以

[root@docker01 etcd-v3.5.0-linux-amd64]# docker exec -it busybox0 sh -c 'ping busybox1'
PING busybox1 (10.0.9.3): 56 data bytes
64 bytes from 10.0.9.3: seq=0 ttl=64 time=0.499 ms
64 bytes from 10.0.9.3: seq=1 ttl=64 time=0.902 ms
64 bytes from 10.0.9.3: seq=2 ttl=64 time=1.757 ms

[root@docker01 etcd-v3.5.0-linux-amd64]# docker exec -it busybox0 sh -c 'ping 10.0.9.3'
PING 10.0.9.3 (10.0.9.3): 56 data bytes
64 bytes from 10.0.9.3: seq=0 ttl=64 time=0.684 ms
64 bytes from 10.0.9.3: seq=1 ttl=64 time=1.390 ms

反过来试试也是没问题的

[root@docker02 etcd-v3.5.0-linux-amd64]# docker exec -it busybox1 sh -c 'ping busybox0'
PING busybox0 (10.0.9.2): 56 data bytes
64 bytes from 10.0.9.2: seq=0 ttl=64 time=0.563 ms
64 bytes from 10.0.9.2: seq=1 ttl=64 time=0.959 ms

[root@docker02 etcd-v3.5.0-linux-amd64]# docker exec -it busybox1 sh -c 'ping 10.0.9.2'
PING 10.0.9.2 (10.0.9.2): 56 data bytes
64 bytes from 10.0.9.2: seq=0 ttl=64 time=0.638 ms

至于为什么能解析容器名为ip呢,这也很容易想到,这些信息etcd中都有,创建的时候添加一条规则即可

[root@docker02 etcd-v3.5.0-linux-amd64]# docker exec -it busybox1 sh -c 'cat /etc/hosts'
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
10.0.9.3	8139c23b0e57

5. overly总结

最后测试阶段发现容器互相ping是ping不通的,我很奇怪,去查了一些资料似乎没什么结果,最后升级了一下内核版本从3.10升级到了5.12.13-1.el7.elrepo.x86_64就可以了

6. 没想到吧总结之后还有内容,哎我就是玩

从上一篇docker网络浅析中我们知道网桥模式的玩法,我们这里尝试不使用任何插件来实现跨主机容器互联

首先我们先改一下两台机器的docker0的ip地址,默认是172.17.0.1,不改的话就导致都路由到一个网段了

vi /etc/docker/daemon.json
"bip":"172.17.3.100/24"
vi /etc/docker/daemon.json
"bip":"172.17.4.100/24"

两天机器分别都改一下然后刷新重启

//主机141
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.3.100  netmask 255.255.255.0  broadcast 172.25.0.255
        inet6 fe80::42:5aff:fe07:e6d6  prefixlen 64  scopeid 0x20<link>
//主机142        
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.4.100  netmask 255.255.255.0  broadcast 172.25.0.255
        inet6 fe80::42:5aff:fe07:e6d6  prefixlen 64  scopeid 0x20<link>

然后我们添加两条路由规则

//主机141
route add -net 172.17.4.0 netmask 255.255.255.0 gw 172.16.3.142
//主机142
route add -net 172.17.3.0 netmask 255.255.255.0 gw 172.16.3.141

//防止丢包两边分别加上
iptables -I DOCKER  --dst 172.17.0.0/16 -j ACCEPT

我们尝试从141机器的172.17.3.0网段的容器,请求142机器的172.17.4.0网段的容器,看看能否成功

//141
docker run -it -d --name t1 busybox   -> ip = 172.17.3.1
//142
docker run -it -d --name t2 busybox   -> ip = 172.17.4.1
[root@docker01 etcd-v3.5.0-linux-amd64]# docker exec -it t1 sh -c 'ping 172.17.4.1'
PING 172.17.4.1 (172.17.4.1): 56 data bytes
64 bytes from 172.17.4.1: seq=0 ttl=62 time=2.386 ms
64 bytes from 172.17.4.1: seq=1 ttl=62 time=2.061 ms

[root@docker02 etcd-v3.5.0-linux-amd64]# docker exec -it t2 sh -c 'ping 172.17.3.1'
PING 172.17.3.1 (172.17.3.1): 56 data bytes
64 bytes from 172.17.3.1: seq=0 ttl=62 time=1.144 ms
64 bytes from 172.17.3.1: seq=1 ttl=62 time=1.606 ms

7. 掘金这个主题针不戳啊

---
theme: mk-cute
---