DPDK介绍
DPDK(Data Plane Development Kit)是一个用于高性能数据包处理的软件集合。
DPDK主要特点和功能如下:
- 以加速网络数据包处理为目标,旨在实现最佳性能。
- 直接从用户空间访问网络设备,无需操作系统缓冲和复制,降低CPU和内存消耗。
- 提供专门针对数据平面的编程接口,比如流式网络 I/O 和内存管理。
- 支持多种计算平台,如x86、ARM和POWER。常用作高速网络和数据中介软件。
- 内含基于环形缓冲的框架和线程模型,用于无锁的包处理。
- 常用模块包括Poll Mode Driver(PMD)和Memory Manager(MEMPOOL)等。
- 主要应用场景包括高性能交换机、路由器、防火墙、TCP私有网关等。
- 一些知名项目使用DPDK,如:Linux VRouter虚拟路由,Apache Cassandra等。
- 出于性能考虑,DPDK应用通常需要绕过OS网络堆栈,实现零拷贝。
总之,DPDK是一个用于高性能数据包I/O和处理的重量级开源软件库,广泛应用于高性能网络应用和设备上。它利用零拷贝和无锁设计大幅提升网络数据处理能力。
创建虚拟机
我们使用Vmware创建一个Ubuntu的虚拟机，并添加一块网卡，其中桥接网卡作为DPDK运行的网卡，NAT网卡作为我们SSH连接使用的网卡
安装使用的Ubuntu版本
gillbert@dpdk:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.7 LTS
Release: 16.04
Codename: xenial
安装必备的工具包
apt install build-essential python
修改虚拟机网卡信息
将 ethernet0.virtualDev 由 e1000 修改 vmxnet3，因为 vmware 的 vmxnet3 支持多队列网卡
修改系统的GRUB启动参数
- 物理机： default_hugepages=1G hugepagesz=1G hugepages=20 isolcpus=0-7
- 虚拟机： default_hugepages=1G hugepagesz=2M hugepages=1024 isolcpus=0-2
find_preseed=/preseed.cfg noprompt net.ifnames=0 biosdevname=0 default_hugepagesz=2M hugepagesz=2M hugepages=1024 isolcpus=0-2
更新grub
更新grub，然后重启系统
sudo update-grub
修改网卡配置文件
此时可以看到系统的网卡名变为了eth开头的网卡，因此我们需要修改网卡配置文件
gillbert@dpdk:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:0c:29:e5:bd:78 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:e5:bd:82 brd ff:ff:ff:ff:ff:ff
inet 192.168.222.132/24 brd 192.168.222.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fee5:bd82/64 scope link
valid_lft forever preferred_lft forever
查看系统是否支持多队列网卡
cat /proc/interrupts | grep eth0
可以看到eth0有4个网卡队列eth0-rxtx-0、eth0-rxtx-1、eth0-rxtx-2、eth0-rxtx-3，对应的中断号分别是56、57、58、59。这里虚拟机有多少个CPU就会有多少个队列。
开始编译DPDK
下载DPDK源码
wget https://fast.dpdk.org/rel/dpdk-19.08.2.tar.xz
解压源码
tar -xf dpdk-19.08.2.tar.xz
设置环境变量
export RTE_SDK=/home/gillbert/dpdk-stable-19.08.2/
export RTE_TARGET=x86_64-native-linux-gcc
执行DPDK提供的脚本，开始编译
./usertools/dpdk-setup.sh
------------------------------------------------------------------------------
RTE_SDK exported as /home/gillbert/dpdk-stable-19.08.2
------------------------------------------------------------------------------
----------------------------------------------------------
Step 1: Select the DPDK environment to build
----------------------------------------------------------
[1] arm64-armada-linuxapp-gcc
[2] arm64-armada-linux-gcc
[3] arm64-armv8a-linuxapp-clang
[4] arm64-armv8a-linuxapp-gcc
[5] arm64-armv8a-linux-clang
[6] arm64-armv8a-linux-gcc
[7] arm64-bluefield-linuxapp-gcc
[8] arm64-bluefield-linux-gcc
[9] arm64-dpaa2-linuxapp-gcc
[10] arm64-dpaa2-linux-gcc
[11] arm64-dpaa-linuxapp-gcc
[12] arm64-dpaa-linux-gcc
[13] arm64-octeontx2-linuxapp-gcc
[14] arm64-octeontx2-linux-gcc
[15] arm64-stingray-linuxapp-gcc
[16] arm64-stingray-linux-gcc
[17] arm64-thunderx2-linuxapp-gcc
[18] arm64-thunderx2-linux-gcc
[19] arm64-thunderx-linuxapp-gcc
[20] arm64-thunderx-linux-gcc
[21] arm64-xgene1-linuxapp-gcc
[22] arm64-xgene1-linux-gcc
[23] arm-armv7a-linuxapp-gcc
[24] arm-armv7a-linux-gcc
[25] i686-native-linuxapp-gcc
[26] i686-native-linuxapp-icc
[27] i686-native-linux-gcc
[28] i686-native-linux-icc
[29] ppc_64-power8-linuxapp-gcc
[30] ppc_64-power8-linux-gcc
[31] x86_64-native-bsdapp-clang
[32] x86_64-native-bsdapp-gcc
[33] x86_64-native-freebsd-clang
[34] x86_64-native-freebsd-gcc
[35] x86_64-native-linuxapp-clang
[36] x86_64-native-linuxapp-gcc
[37] x86_64-native-linuxapp-icc
[38] x86_64-native-linux-clang
[39] x86_64-native-linux-gcc
[40] x86_64-native-linux-icc
[41] x86_x32-native-linuxapp-gcc
[42] x86_x32-native-linux-gcc
----------------------------------------------------------
Step 2: Setup linux environment
----------------------------------------------------------
[43] Insert IGB UIO module
[44] Insert VFIO module
[45] Insert KNI module
[46] Setup hugepage mappings for non-NUMA systems
[47] Setup hugepage mappings for NUMA systems
[48] Display current Ethernet/Baseband/Crypto device settings
[49] Bind Ethernet/Baseband/Crypto device to IGB UIO module
[50] Bind Ethernet/Baseband/Crypto device to VFIO module
[51] Setup VFIO permissions
----------------------------------------------------------
Step 3: Run test application for linux environment
----------------------------------------------------------
[52] Run test application ($RTE_TARGET/app/test)
[53] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
----------------------------------------------------------
Step 4: Other tools
----------------------------------------------------------
[54] List hugepage info from /proc/meminfo
----------------------------------------------------------
Step 5: Uninstall and system cleanup
----------------------------------------------------------
[55] Unbind devices from IGB UIO or VFIO driver
[56] Remove IGB UIO module
[57] Remove VFIO module
[58] Remove KNI module
[59] Remove hugepage mappings
[60] Exit Script
Option:
开始编译
64位系统出入39
[58] Remove KNI module
[59] Remove hugepage mappings
[60] Exit Script
Option: 39
编译完成后会生成x86_64-native-linux-gcc文件夹
插入IGB_UIO模块
选择 43 插入 IGB_UIO 模块
Step 2: Setup linux environment
----------------------------------------------------------
[43] Insert IGB UIO module
[44] Insert VFIO module
[45] Insert KNI module
[46] Setup hugepage mappings for non-NUMA systems
[47] Setup hugepage mappings for NUMA systems
[48] Display current Ethernet/Baseband/Crypto device settings
[49] Bind Ethernet/Baseband/Crypto device to IGB UIO module
[50] Bind Ethernet/Baseband/Crypto device to VFIO module
[51] Setup VFIO permissions
----------------------------------------------------------
Step 3: Run test application for linux environment
----------------------------------------------------------
[52] Run test application ($RTE_TARGET/app/test)
[53] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
----------------------------------------------------------
Step 4: Other tools
----------------------------------------------------------
[54] List hugepage info from /proc/meminfo
----------------------------------------------------------
Step 5: Uninstall and system cleanup
----------------------------------------------------------
[55] Unbind devices from IGB UIO or VFIO driver
[56] Remove IGB UIO module
[57] Remove VFIO module
[58] Remove KNI module
[59] Remove hugepage mappings
[60] Exit Script
Option: 43
## ERROR: Target does not have the DPDK UIO Kernel Module.
To fix, please try to rebuild target.
Press enter to continue ...
执行此步骤会报错，错误的原因为我们的系统上没有uio这个内核模块，uio（Userspace I/O）是一种在用户空间与设备之间进行直接通信的机制，用于将网络设备与DPDK绑定以实现高性能数据包处理。因此我们需要先加载uio模块与igb_uio模块。
编译安装uio内核模块
确认系统已安装开发工具包和依赖库
sudo apt-get install libnuma-dev build-essential linux-headers-$(uname -r)
进入DPDK源码目录，并根据目标系统进行配置。
此处的T指定的值和我们开始在
make config T=x86_64-native-linux-gcc
执行配置命令后，使用make命令编译DPDK库及其驱动程序
make -j 4
加载uio内核模块。
sudo modprobe uio
sudo insmod /home/gillbert/dpdk-stable-19.08.2/build/kmod/igb_uio.ko
确认uio内核模块是否已成功加载。
root@dpdk:/home/gillbert/dpdk-stable-19.08.2# lsmod |grep uio
igb_uio 16384 0
uio 20480 1 igb_uio
重新插入igb_uio模块
在执行此步骤之前，我们需要重新选择39编译一下，然后再选择43插入IGB_UIO模块
Option: 43
Unloading any existing DPDK UIO module
Loading DPDK UIO module
Press enter to continue ...
插入 VFIO 模块
选择 44 插入 VFIO 模块
Option: 44
Unloading any existing VFIO module
Loading VFIO module
chmod /dev/vfio
OK
绑定 igb_uio 模块
选择 49 绑定 igb_uio 模块,输入49后我们再输入eth0网卡
Option: 49
Network devices using kernel driver
===================================
0000:02:02.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=eth1 drv=e1000 unused=igb_uio,vfio-pci *Active*
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=igb_uio,vfio-pci *Active*
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
==============================
No 'Misc (rawdev)' devices detected
===================================
Enter PCI address of device to bind to IGB UIO driver: eth0
Warning: routing table indicates that interface 0000:03:00.0 is active. Not modifying
OK
执行上述命令时，我们发现有个警告信息，提示我们这个网卡是活动的状态不能设置，因此我们需要将这个网卡先down掉再设置
ifconfig eth0 down
[60] Exit Script
Option: 49
Network devices using kernel driver
===================================
0000:02:02.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=eth1 drv=e1000 unused=igb_uio,vfio-pci *Active*
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=igb_uio,vfio-pci
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
==============================
No 'Misc (rawdev)' devices detected
===================================
Enter PCI address of device to bind to IGB UIO driver: eth0
OK
测试
运行testpmd进行测试
Option: 53
Enter hex bitmask of cores to execute testpmd app on
Example: to execute app on cores 0 to 7, enter 0xff
bitmask: 7
Launching app
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:02.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 15ad:7b0 net_vmxnet3
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 00:0C:29:E5:BD:78
Checking link statuses...
Done
查看端口信息
输入 show port info 0命令
testpmd> show port info 0
********************* Infos for port 0 *********************
MAC address: 00:0C:29:E5:BD:78
Device name: 0000:03:00.0
Driver name: net_vmxnet3
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 10000 Mbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 1
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
strip off
filter off
qinq(extend) off
Supported RSS offload flow types:
ipv4
ipv4-tcp
ipv6
ipv6-tcp
Minimum size of RX buffer: 1646
Maximum configurable length of RX packet: 16384
Current number of RX queues: 1
Max possible RX queues: 16
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 128
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 8
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 512
TXDs number alignment: 1
Max segment number per packet: 255
Max segment number per MTU/TSO: 16
编译helloword测试程序进行测试
进入 example/helloworl可以直接 make,也可以通过 gcc 命令编译
root@dpdk:/home/gillbert/dpdk-stable-19.08.2/examples/helloworld# gcc -o helloword main.c -I /usr/local/include/dpdk/ -ldpdk -lpthread -lnuma -ldl
root@dpdk:/home/gillbert/dpdk-stable-19.08.2/examples/helloworld# ./helloword -l 0-3 -n 4
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
hello from core 1
hello from core 2
hello from core 3
hello from core 0
重新加载DPDK环境
再次重启系统后，我们发现以前配置的DPDK环境都没有了，因此我们需要重新配置一下
加载内核模块
modprobe uio
sudo insmod /home/gillbert/dpdk-stable-19.08.2/build/kmod/igb_uio.ko
设置环境变量
export RTE_SDK=/home/gillbert/dpdk-stable-19.08.2/
export RTE_TARGET=x86_64-native-linux-gcc
运行初始化脚本
root@dpdk:/home/gillbert/dpdk-stable-19.08.2# ./usertools/dpdk-setup.sh
加载模块
依次输入运行43, 44
绑定网卡
再输入49,网卡名称我们输入eth0
测试
我们可以按着前面的测试步骤进行测试看下是否加载成功。
