LXCFS在Docker和Kubernetes下的实践

1 什么是lxcfs？

LXCFS是小型的fuse文件系统，目的是让Linux容器更像一个虚拟机。 LXCFS关注procfs下面提供的文件，比如：

/proc/cpuinfo
/proc/diskstats
/proc/meminfo
/proc/stat
/proc/swaps
/proc/uptime
/proc/slabinfo
/sys/devices/system/cpu/online

2 为什么要用lxcfs？

Linuxs利用Cgroup实现了对容器的资源限制，但在容器内部依然缺省挂载了宿主机上的procfs的/proc目录，其包含如：meminfo, cpuinfo，stat， uptime等资源信息。一些监控工具如free/top或遗留应用还依赖上述文件内容获取资源配置和使用情况。当它们在容器中运行时，就会把宿主机的资源状态读取出来，引起错误和不便。

3 如何应用lxcfs？

3.1 docker实践

• 安装

yum install fuse fuse-lib fuse-devel
git clone git://github.com/lxc/lxcfs
cd lxcfs
./bootstrap.sh
./configure
make
make install

• 编写服务启动脚本

mkdir -p /var/lib/lxcfs
cat > /usr/lib/systemd/system/lxcfs.service <<EOF
[Unit]
Description=lxcfs

[Service]
ExecStart=/usr/bin/lxcfs -f /var/lib/lxcfs 
Restart=on-failure
#ExecReload=/bin/kill -s SIGHUP $MAINPID

[Install]
WantedBy=multi-user.target
EOF

• 启动lxcfs

systemctl daemon-reload
systemctl start lxcfs

• 启动容器并查看

docker run -it -m 256m \
      -v /var/lib/lxcfs/proc/cpuinfo:/proc/cpuinfo:rw \
      -v /var/lib/lxcfs/proc/diskstats:/proc/diskstats:rw \
      -v /var/lib/lxcfs/proc/meminfo:/proc/meminfo:rw \
      -v /var/lib/lxcfs/proc/stat:/proc/stat:rw \
      -v /var/lib/lxcfs/proc/swaps:/proc/swaps:rw \
      -v /var/lib/lxcfs/proc/uptime:/proc/uptime:rw \
      ubuntu:latest /bin/bash
root@e203bbf2c394:/# free
              total        used        free      shared  buff/cache   available
Mem:         262144        1124      261016           0           4      261020
Swap:        524288           0      524288

3.2 kubernetes实践

• 验证

# 查看是否开启
$ kubectl api-versions | grep 'admissionregistration.k8s.io/v1beta1'

admissionregistration.k8s.io/v1beta1

• 下载 lxcfs-admission-webhook 项目

$ git clone https://github.com/denverdino/lxcfs-admission-webhook.git
$ cd lxcfs-admission-webhook

• 部署daemonset-lxcfs

$ kubectl apply -f deployment/lxcfs-daemonset.yaml
$ kubectl  get pod | grep lxcfs
lxcfs-ch95x                                          1/1     Running   0          2d22h
lxcfs-j6pjk                                          1/1     Running   0          2d22h

• 部署lxcfs-admission-webhook

$ # 执行 shell 部署脚本
$ deployment/install.sh

• 测试

$ kubectl label namespace default lxcfs-admission-webhook=enabled
$ kubectl apply -f deployment/web.yaml
$ kubectl get pod | grep web
web-79c99559c7-fmqhl                                 1/1     Running   0          23h
web-79c99559c7-mlzkb                                 1/1     Running   0          23h
$ kubectl exec -it web-79c99559c7-fmqhl -- free -h
             total       used       free     shared    buffers     cached
Mem:          256M       2.6M       253M         0B         0B       272K
-/+ buffers/cache:       2.4M       253M
Swap:           0B         0B         0B

Q&A

在kubernetes部署daemonset时，会报错如下:

kubectl  logs -f lxcfs-rlnl9
/usr/local/bin/lxcfs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory

解决办法

• 方法一 在每个节点上面安装fuse-libs

yum install fuse-libs -y

• 方法二 更改部署脚本，重新打包。已经作为PR提交到github PR 上，目前还没有被merge

• lxcfs-image/Dockerfile

FROM centos:7 as build
RUN yum -y update
RUN yum -y install fuse-devel pam-devel wget install gcc automake autoconf libtool make
ENV LXCFS_VERSION 3.1.2
RUN wget https://linuxcontainers.org/downloads/lxcfs/lxcfs-$LXCFS_VERSION.tar.gz && \
        mkdir /lxcfs && tar xzvf lxcfs-$LXCFS_VERSION.tar.gz -C /lxcfs  --strip-components=1 && \
        cd /lxcfs && ./configure && make

FROM centos:7
STOPSIGNAL SIGINT
COPY --from=build /lxcfs/lxcfs /usr/local/bin/lxcfs
COPY --from=build /lxcfs/.libs/liblxcfs.so /usr/local/lib/lxcfs/liblxcfs.so
COPY --from=build /lxcfs/lxcfs /lxcfs/lxcfs
COPY --from=build /lxcfs/.libs/liblxcfs.so /lxcfs/liblxcfs.so
COPY --from=build /usr/lib64/libfuse.so.2.9.2 /lxcfs/libfuse.so.2.9.2
COPY --from=build /usr/lib64/libulockmgr.so.1.0.1 /lxcfs/libulockmgr.so.1.0.1

COPY start.sh /
CMD ["/start.sh"]

• lxcfs-image/start.sh

#!/bin/bash

# Cleanup
nsenter -m/proc/1/ns/mnt fusermount -u /var/lib/lxcfs 2> /dev/null || true
nsenter -m/proc/1/ns/mnt [ -L /etc/mtab ] || \
        sed -i "/^lxcfs \/var\/lib\/lxcfs fuse.lxcfs/d" /etc/mtab

# Prepare
mkdir -p /usr/local/lib/lxcfs /var/lib/lxcfs

# Update lxcfs
cp -f /lxcfs/lxcfs /usr/local/bin/lxcfs
cp -f /lxcfs/liblxcfs.so /usr/local/lib/lxcfs/liblxcfs.so

cp -f /lxcfs/libfuse.so.2.9.2 /usr/lib64/libfuse.so.2.9.2
cp -f /lxcfs/libulockmgr.so.1.0.1 /usr/lib64/libulockmgr.so.1.0.1

ln -s /usr/lib64/libfuse.so.2.9.2 /usr/lib64/libfuse.so.2
ln -s /usr/lib64/libulockmgr.so.1.0.1 /usr/lib64/libulockmgr.so.1

# Mount
exec nsenter -m/proc/1/ns/mnt /usr/local/bin/lxcfs /var/lib/lxcfs/

• lxcfs-image/Dockerfile

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: lxcfs
  labels:
    app: lxcfs
spec:
  selector:
    matchLabels:
      app: lxcfs
  template:
    metadata:
      labels:
        app: lxcfs
    spec:
      hostPID: true
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: lxcfs
        image: registry.cn-hangzhou.aliyuncs.com/denverdino/lxcfs:3.1.2
        imagePullPolicy: Always
        securityContext:
          privileged: true
        volumeMounts:
        - name: cgroup
          mountPath: /sys/fs/cgroup
        - name: lxcfs
          mountPath: /var/lib/lxcfs
          mountPropagation: Bidirectional
        - name: usr-local
          mountPath: /usr/local
        - name: usr-lib
          mountPath: /usr/lib64
      volumes:
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
      - name: usr-local
        hostPath:
          path: /usr/local
      - name: usr-lib
        hostPath:
          path: /usr/lib64
      - name: lxcfs
        hostPath:
          path: /var/lib/lxcfs
          type: DirectoryOrCreate