k8s 集群部署参考
运行环境
-
CentOS Linux release 7.9.2009 (Core)
-
k8s 1.23.4
-
containerd github.com/containerd/containerd v1.6.8
前置要求
- NVIDIA drivers ~= 384.81
- nvidia-docker >= 2.0 || nvidia-container-toolkit >= 1.7.0 (>= 1.11.0 to use integrated GPUs on Tegra-based systems)
- Kubernetes version >= 1.10
部署NVIDIA驱动
1. 查看显卡型号
方法1：
lspci | grep -i nvidia | grpe VGA
但是通过输出可能看不出显卡型号，比如:
01:00.0 VGA compatible controller: NVIDIA Corporation Device 2484 (rev a1)
这时候可以使用下面的网址进行查询：
进入到上面网站，输入2484就可以看到显卡的具体型号(根据输出的id搜索)
方法2：
update-pciids
lspci | grep -i nvidia |grpe VGA01:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070] (rev a1)
2. 验证是否有GCC编译环境
gcc -v
若没有先安装gcc,一般系统会自动安装
3. 验证系统是否安装了正确的内核头文件和开发包
yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r)
部署 NVIDIA Container Toolkit
下述命令用于centos7，其他环境请点击查看官方文档
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
&& curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
yum clean expire-cache && yum install nvidia-container-toolkit -y
4. 下载驱动
1.打开 NVIDIA 驱动下载链接 Advanced Driver Search | NVIDIA 。
根据自己的型号搜索并下载
5. 安装
chmod +x NVIDIA-Linux-x86_64-525.89.02.run && sh NVIDIA-Linux-x86_64-525.89.02.run --ui=none --no-questions
6. 验证
nvidia-smi
正常输出即安装成功
配置containerd
containerd配置文件为 /etc/containerd/config.toml
-
如果需要将nvidia-container-runtime设置为默认runtime，请参考如下配置修改
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options] BinaryName = "/usr/bin/nvidia-container-runtime"
2. 如果需要单node运行多runtime，请参考如下配置修改，在原始配置后添加nvidia runtime
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "nvidia"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
privileged_without_host_devices = false
runtime_engine = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
BinaryName = "/usr/bin/nvidia-container-runtime"
systemctl restart containerd
多runtime配置
如果有多runtime共存部署需求，创建相应的runtime class
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: nvidia
handler: nvidia
此处需要注意，如果多runtime共存，需要指定runtimeclass保证pod使用nvidia runtime
需添加runtimeClassName: nvidia 配置，如果默认runtime就是nvidia,可以移除这行配置
部署nvidia k8s-device-plugin
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nvidia-device-plugin-daemonset
namespace: kube-system
spec:
selector:
matchLabels:
name: nvidia-device-plugin-ds
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
name: nvidia-device-plugin-ds
spec:
runtimeClassName: nvidia
#tolerations:
#- key: nvidia.com/gpu
# operator: Exists
# effect: NoSchedule
# Mark this pod as a critical add-on; when enabled, the critical add-on
# scheduler reserves resources for critical add-on pods so that they can
# be rescheduled after a failure.
# See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
priorityClassName: "system-node-critical"
containers:
- image: nvcr.io/nvidia/k8s-device-plugin:v0.13.0
name: nvidia-device-plugin-ctr
env:
- name: FAIL_ON_INIT_ERROR
value: "false"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumeMounts:
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
volumes:
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
待pod running 验证
kubectl describe node "node名称" |grep "nvidia.com/gpu"
部署测试样例
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
restartPolicy: Never
runtimeClassName: nvidia
containers:
- name: cuda-container
image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
command: ["/bin/sh","-c"]
args: ["nvidia-smi"]
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
查看结果
kubectl logs gpu-pod
Wed Feb 22 08:46:51 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| 0% 52C P8 14W / 220W | 0MiB / 8192MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
