k8s-从部署到实战

什么是kubernetes

Google开源的容器集群管理系统

为容器化（Docker）的应用提供部署运行、资源调度、服务发现和动态伸缩、高可用等

Kubernetes的核心概念

master： APIServer、scheduler、controller manager等

Node：Pod的服务节点（亦叫agent或minion）/Pod运行的宿主机

Pod：是Kurbernetes进行创建、调度和管理的最小单位，它提供了比容器更高层次的抽象。

Replication Controller：管理Pod的副本/弹性伸缩、动态扩容和滚动升级的核心

Service：Pod的逻辑集合和访问该集合的策略，是真实服务的抽象/使用kube-proxy进行网络的控制

Label：实质是一系列的K/V键值对 Replication Controller和Service运行的基础

Kubernetes架构和组件

部署

环境信息

操作系统

[root@master ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

网络

master：192.168.108.132/24 node1：192.168.108.133/24 node2：192.168.108.134/24

前期工作准备

以node1节点为例，每一台主机类似配置

修改主机名

hostnamectl set-hostname node1

关闭防火墙服务

[root@node1 ~]# systemctl stop firewalld
[root@node1 ~]# systemctl disable firewalld

关闭selinux

vim /etc/selinux/config
SELINUX=disabled

master安装部署

安装

# 使用yum安装etcd和kubernetes-master
[root@master ~]# yum install etcd kubernetes-master  -y

修改配置文件

/etc/etcd/etcd.conf

[root@master ~]# cat /etc/etcd/etcd.conf |grep -v "^#"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_NAME="default"
ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"

/etc/kubernetes/apiserver

[root@master ~]# cat /etc/kubernetes/apiserver |grep -Ev "^#|^$"
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
KUBE_API_PORT="--port=8089"
KUBELET_PORT="--kubelet-port=10250"
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"
KUBE_API_ARGS=""

启动etcd、kube-apiserver、kube-controller-manager、kube-scheduler等服务，并设置开机启动

for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do systemctl start $SERVICES;systemctl enable $SERVICES;systemctl status $SERVICES; done

在etcd中定义flannel网络

[root@master ~]# etcdctl mk /atomic.io/network/config '{"Network":"172.40.0.0/16"}'
{"Network":"172.40.0.0/16"}

node节点安装部署

安装

yum  install flannel kubernetes-node  -y

安装时发现如下报错：

错误：docker-ce-cli conflicts with 2:docker-1.13.1-203.git0be3e21.el7.centos.x86_64
错误：docker-ce conflicts with 2:docker-1.13.1-203.git0be3e21.el7.centos.x86_64
 您可以尝试添加 --skip-broken 选项来解决该问题
 您可以尝试执行：rpm -Va --nofiles --nodigest

原因本节点已经安装了docker，卸载后重装即可。

[root@node1 ~]# yum list installed | grep docker
containerd.io.x86_64            1.4.3-3.1.el7                  @docker-ce-stable
docker-ce-cli.x86_64            1:20.10.1-3.el7                @docker-ce-stabl
[root@node1 ~]# yum -y remove containerd.io.x86_64
[root@node1 ~]# yum -y remove docker-ce-cli.x86_64

为flannel网络指定etcd服务，修改/etc/sysconfig/flanneld文件

[root@node1 ~]# cat /etc/sysconfig/flanneld |grep -Ev "^#|^$"
FLANNEL_ETCD_ENDPOINTS="http://192.168.108.132:2379"
FLANNEL_ETCD_PREFIX="/atomic.io/network"

/etc/kubernetes/config文件

[root@node1 ~]# cat  /etc/kubernetes/config |grep -Ev "^#|^$"
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=0"
KUBE_ALLOW_PRIV="--allow-privileged=false"
KUBE_MASTER="--master=http://192.168.108.132:8089"

修改node节点配置文件/etc/kubernetes/kubelet

[root@node1 ~]# vim /etc/kubernetes/kubelet
[root@node1 ~]# cat /etc/kubernetes/kubelet  |grep -Ev "^#|^$"
KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_HOSTNAME="--hostname-override=192.168.108.133"
KUBELET_API_SERVER="--api-servers=http://192.168.108.132:8089"
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

配置完node1，2后，在所有Node节点上启动flanneld，docker，kube-proxy,kubelet服务，并设置开机启动

# 一定要注意etcd—->flannel—–>docker的先后启动顺序
[root@node1 ~]# for SERVICES in flanneld  docker kube-proxy kubelet;do systemctl start $SERVICES;systemctl enable $SERVICES;systemctl status $SERVICES; done

验证集群状态

验证集群是否安装成功

[root@master ~]# kubectl   get nodes
the server doesn't have a resource type "nodes"

说明失败，查看服务的状态：

systemctl  status kube-controller-manager.service -l

发现有如下报错：

12月 22 14:26:45 master kube-controller-manager[102174]: E1222 14:26:45.219794  102174 leaderelection.go:228] error retrieving resource lock kube-system/kube-controller-manager: the server does not allow access to the requested resource (get endpoints kube-controller-manager)

这是因为配置文件导致，把8089端口，改为8080后，重启服务,在验证：

[root@master ~]# kubectl   get node
NAME              STATUS    AGE
192.168.108.133   Ready     2h
192.168.108.134   Ready     2h

查看网络情况

[root@master ~]# etcdctl  ls  -r  | grep  subnets
/atomic.io/network/subnets
/atomic.io/network/subnets/172.40.99.0-24
/atomic.io/network/subnets/172.40.66.0-24
[root@master ~]# etcdctl   get /atomic.io/network/subnets/172.40.99.0-24
{"PublicIP":"192.168.108.134"}
[root@master ~]# etcdctl   get /atomic.io/network/subnets/172.40.66.0-24
{"PublicIP":"192.168.108.133"}

pod

创建pod

附 YAML 简介

基本语法
    大小写敏感
	使用缩进表示层级关系
	缩进不允许使用tab，只允许空格
	缩进的空格数不重要，只要相同层级的元素左对齐即可
	'#'表示注释

使用yml文件创建

[root@master ~]# vim ng-pod.yml
[root@master ~]# cat ng-pod.yml
apiVersion: v1
kind: Pod
metadata:
   name: swh-ng
spec:
  containers:
  - name: swh-ng   #pod名称
    image: nginx    #镜像名
    ports:
    - containerPort: 80  #容器监听端口号
      hostPort: 80       #node监听端口号，对外暴露并提供服务
[root@master ~]# kubectl create -f ng-pod.yml
Error from server (ServerTimeout): error when creating "ng-pod.yml": No API token found for service account "default", retry after the token is automatically created and added to the service account

发现有报错，解决方法：

# 生成秘钥
[root@master ~]# openssl genrsa -out /etc/kubernetes/service.key 2048
Generating RSA private key, 2048 bit long modulus
........................................................................+++
....+++
e is 65537 (0x10001)
[root@master ~]# vim /etc/kubernetes/apiserver
# Add your own!
KUBE_API_ARGS="--service_account_key_file=/etc/kubernetes/service.key"
[root@master ~]# vim /etc/kubernetes/controller-manager
# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--service_account_key_file=/etc/kubernetes/service.key"
systemctl restart kube-apiserver

经处理，错误依旧。

[root@master ~]# kubectl create -f ng-pod.yml
Error from server (ServerTimeout): error when creating "ng-pod.yml": No API token found for service account "default", retry after the token is automatically created and added to the service account

方法2：

vim /etc/kubernetes/apiserver	
# default admission control policies
#KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"
[root@master ~]# systemctl restart kube-apiserver

验证：

[root@master ~]# kubectl create -f ng-pod.yml
pod "swh-ng" created

问题解决了。

[root@master ~]# kubectl get pod
NAME      READY     STATUS              RESTARTS   AGE
swh-ng    0/1       ContainerCreating   0          1m
[root@master ~]# kubectl describe pod  swh-ng
Name:           swh-ng
Namespace:      default
Node:           192.168.108.134/192.168.108.134
Start Time:     Tue, 22 Dec 2020 18:00:55 +0800
Labels:         <none>
Status:         Pending
IP:
Controllers:    <none>
Containers:
  swh-ng:
    Container ID:
    Image:                      nginx
    Image ID:
    Port:                       80/TCP
    State:                      Waiting
      Reason:                   ContainerCreating
    Ready:                      False
    Restart Count:              0
    Volume Mounts:              <none>
    Environment Variables:      <none>
Conditions:
  Type          Status
  Initialized   True
  Ready         False
  PodScheduled  True
No volumes.
QoS Class:      BestEffort
Tolerations:    <none>
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason          Message
  ---------     --------        -----   ----                            -------------   --------        ------          -------
  2m            2m              1       {default-scheduler }                            Normal          Scheduled       Successfully assigned swh-ng to 192.168.108.134
  2m            59s             4       {kubelet 192.168.108.134}                       Warning         FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for registry.access.redhat.com/rhel7/pod-infrastructure:latest, this may be because there are no credentials on this request.  details: (open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory)"

  2m    4s      8       {kubelet 192.168.108.134}               Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "POD" with ImagePullBackOff: "Back-off pulling image \"registry.access.redhat.com/rhel7/pod-infrastructure:latest\""

基础镜像不存在，软链接指向文件不存在导致无法拉去基础镜像。

解决方法1)：

yum remove *rhsm* -y
yum install *rhsm* -y

验证：

[root@master ~]# kubectl delete  pod swh-ng
pod "swh-ng" deleted
[root@master ~]# kubectl create -f ng-pod.yml
pod "swh-ng" created
[root@master ~]# kubectl get pod
NAME      READY     STATUS              RESTARTS   AGE
swh-ng    0/1       ContainerCreating   0          34s

还是没有解决。

解决方法2)：

[root@master ~]# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm
[root@master ~]# rpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm | cpio -iv --to-stdout ./etc/rhsm/ca/redhat-uep.pem | tee /etc/rhsm/ca/redhat-uep.pem
[root@master ~]# yum install docker -y
[root@master ~]# systemctl start docker
[root@master ~]# systemctl enable docker
[root@master ~]# docker pull registry.access.redhat.com/rhel7/pod-infrastructure:latest

验证：

[root@master ~]# kubectl delete  pod swh-ng
pod "swh-ng" deleted
[root@master ~]# kubectl create -f ng-pod.yml
pod "swh-ng" created
[root@master ~]# kubectl get pod
NAME      READY     STATUS              RESTARTS   AGE
swh-ng    0/1       ContainerCreating   0          8s
[root@master ~]# kubectl get pod  swh-ng  -o wide
NAME      READY     STATUS              RESTARTS   AGE       IP        NODE
swh-ng    0/1       ContainerCreating   0          1m        <none>    192.168.108.134

还是没有解决。

方法3）：

yum install -y *rhsm*

wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm

rpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm | cpio -iv --to-stdout ./etc/rhsm/ca/redhat-uep.pem | tee /etc/rhsm/ca/redhat-uep.pem

验证：

[root@master ~]# kubectl get  pod swh-ng
NAME      READY     STATUS    RESTARTS   AGE
swh-ng    1/1       Running   0          14h
[root@master ~]# kubectl get pod  swh-ng  -o wide
NAME      READY     STATUS    RESTARTS   AGE       IP            NODE
swh-ng    1/1       Running   0          14h       172.40.99.2   192.168.108.134

成功了！

验证pod

[root@master ~]# curl 192.168.108.134
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Replication Controller

用来管理Pod的副本，保证集群中存在指定数量的Pod副本

确保pod数量
确保pod健康
弹性伸缩
滚动升级 也就是蓝绿发布

创建rc

[root@master ~]# vim ng-rc.yml
[root@master ~]# cat ng-rc.yml
apiVersion: v1
# 定义这是一个RC
kind: ReplicationController
metadata:
  name: ng-rc
spec:
#定义将创建2个pod
  replicas: 2
  selector:
    app: ng-rc
  template:
    metadata:
      labels:
        app: ng-rc
    spec:
      containers:
      - name: ng-rc
#使用nginx镜像
        image: nginx
#容器监听端口号
        ports:
        - containerPort: 80
[root@master ~]# kubectl  create  -f  ng-rc.yml
replicationcontroller "ng-rc" created

查看rc

[root@master ~]# kubectl  get pod -o wide
NAME          READY     STATUS    RESTARTS   AGE       IP            NODE
ng-rc-kxsz5   1/1       Running   0          1m        172.40.66.2   192.168.108.133
ng-rc-v8t89   1/1       Running   0          1m        172.40.99.3   192.168.108.134
swh-ng        1/1       Running   0          15h       172.40.99.2   192.168.108.134
[root@master ~]# kubectl   get rc ng-rc  -o wide
NAME      DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)   SELECTOR
ng-rc     2         2         2         1m        ng-rc          nginx      app=ng-rc

弹性伸缩和扩容

[root@master ~]# kubectl   scale  rc  ng-rc  --replicas=4
replicationcontroller "ng-rc" scaled
[root@master ~]# kubectl   get rc ng-rc  -o wide
NAME      DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)   SELECTOR
ng-rc     4         4         3         4m        ng-rc          nginx      app=ng-rc
[root@master ~]# kubectl  get pod -o wide
NAME          READY     STATUS    RESTARTS   AGE       IP            NODE
ng-rc-cp941   1/1       Running   0          22s       172.40.99.4   192.168.108.134
ng-rc-h1gqd   1/1       Running   0          22s       172.40.66.3   192.168.108.133
ng-rc-kxsz5   1/1       Running   0          4m        172.40.66.2   192.168.108.133
ng-rc-v8t89   1/1       Running   0          4m        172.40.99.3   192.168.108.134
swh-ng        1/1       Running   0          15h       172.40.99.2   192.168.108.134
[root@master ~]# kubectl   scale  rc  ng-rc  --replicas=1
replicationcontroller "ng-rc" scaled
[root@master ~]# kubectl  get pod -o wide
NAME          READY     STATUS    RESTARTS   AGE       IP            NODE
ng-rc-v8t89   1/1       Running   0          4m        172.40.99.3   192.168.108.134
swh-ng        1/1       Running   0          15h       172.40.99.2   192.168.108.134
[root@master ~]# kubectl   get rc ng-rc  -o wide
NAME      DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)   SELECTOR
ng-rc     1         1         1         4m        ng-rc          nginx      app=ng-rc

svc

Pod的逻辑集合和访问该集合的策略，是真实服务的抽象

统一的服务访问入口以及服务代理和发现机制

负责网络服务

集群内部访问（clusterIP ）和集群外部访问（NodePort）模式

创建

[root@master ~]# cat ng-rc-svc.yml
apiVersion: v1
# 指定创建一个Service
kind: Service
metadata:
  name: ng-svc
  labels:
    name: ng-svc
spec:
  ports:
# 指的是监听88端口
  - port: 88
# 后端pod监听的端口
    targetPort: 80
    protocol: TCP
  selector:
# 具有app标签，且值为ng-rc 的pod全部纳入本组Service
    app: ng-rc
[root@master ~]# kubectl  create  -f  ng-rc-svc.yml

查看

查看到clusterIP和端口等信息

[root@master ~]# kubectl  get  svc   ng-svc   -o wide
NAME      CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE       SELECTOR
ng-svc    10.254.123.255   <none>        88/TCP    1m        app=ng-rc

查看Service 是否正确接管pod的网络服务

[root@master ~]# kubectl   get endpoints
NAME         ENDPOINTS              AGE
kubernetes   192.168.108.132:6443   22h
ng-svc       172.40.99.3:80         3m

解决其他节点无法访问问题

ReplicationController（RC）和ReplicaSet（RS)

ReplicaSet 和 Replication Controller之间的唯一区别是对选择器的支持。

RC 替代方法：

（1）ReplicaSet
（2）Deployment（推荐）