K8S Containerd 环境添加NVIDIA Runtime支持

1,200 阅读3分钟

k8s 集群部署参考

运行环境

  • CentOS Linux release 7.9.2009 (Core)

  • k8s 1.23.4

  • containerd github.com/containerd/containerd v1.6.8

前置要求

  • NVIDIA drivers ~= 384.81
  • nvidia-docker >= 2.0 || nvidia-container-toolkit >= 1.7.0 (>= 1.11.0 to use integrated GPUs on Tegra-based systems)
  • Kubernetes version >= 1.10

部署NVIDIA驱动

1. 查看显卡型号

方法1:

lspci | grep -i nvidia | grpe VGA

但是通过输出可能看不出显卡型号,比如:

01:00.0 VGA compatible controller: NVIDIA Corporation Device 2484 (rev a1) 

这时候可以使用下面的网址进行查询:

PCI Devices

进入到上面网站,输入2484就可以看到显卡的具体型号(根据输出的id搜索)

方法2:

update-pciids

lspci | grep -i nvidia |grpe VGA01:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070] (rev a1)                           

2. 验证是否有GCC编译环境

gcc -v

若没有先安装gcc,一般系统会自动安装

3. 验证系统是否安装了正确的内核头文件和开发包

yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r)

部署 NVIDIA Container Toolkit

下述命令用于centos7,其他环境请点击查看官方文档

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
   && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

yum clean expire-cache && yum install nvidia-container-toolkit -y

4. 下载驱动

1.打开 NVIDIA 驱动下载链接 Advanced Driver Search | NVIDIA

根据自己的型号搜索并下载

5. 安装

chmod +x NVIDIA-Linux-x86_64-525.89.02.run && sh NVIDIA-Linux-x86_64-525.89.02.run --ui=none --no-questions

6. 验证

nvidia-smi
正常输出即安装成功

配置containerd

containerd配置文件为 /etc/containerd/config.toml

  1. 如果需要将nvidia-container-runtime设置为默认runtime,请参考如下配置修改

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/bin/nvidia-container-runtime"
    

  2. 如果需要单node运行多runtime,请参考如下配置修改,在原始配置后添加nvidia runtime

  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "nvidia"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/bin/nvidia-container-runtime"

systemctl restart containerd

多runtime配置

如果有多runtime共存部署需求,创建相应的runtime class

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia

此处需要注意,如果多runtime共存,需要指定runtimeclass保证pod使用nvidia runtime

需添加runtimeClassName: nvidia 配置,如果默认runtime就是nvidia,可以移除这行配置

部署nvidia k8s-device-plugin

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      runtimeClassName: nvidia
      #tolerations:
      #- key: nvidia.com/gpu
      #  operator: Exists
      #  effect: NoSchedule
      # Mark this pod as a critical add-on; when enabled, the critical add-on
      # scheduler reserves resources for critical add-on pods so that they can
      # be rescheduled after a failure.
      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
      priorityClassName: "system-node-critical"
      containers:
      - image: nvcr.io/nvidia/k8s-device-plugin:v0.13.0
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

待pod running 验证

kubectl describe node "node名称" |grep "nvidia.com/gpu"

部署测试样例

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  restartPolicy: Never
  runtimeClassName: nvidia
  containers:
    - name: cuda-container
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
      command: ["/bin/sh","-c"]
      args: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1 # requesting 1 GPU
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

查看结果

kubectl logs gpu-pod

Wed Feb 22 08:46:51 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   52C    P8    14W / 220W |      0MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+