Ubuntu配置DPDK环境

44 阅读10分钟

DPDK介绍

DPDK(Data Plane Development Kit)是一个用于高性能数据包处理的软件集合。

DPDK主要特点和功能如下:

  • 以加速网络数据包处理为目标,旨在实现最佳性能。
  • 直接从用户空间访问网络设备,无需操作系统缓冲和复制,降低CPU和内存消耗。
  • 提供专门针对数据平面的编程接口,比如流式网络 I/O 和内存管理。
  • 支持多种计算平台,如x86、ARM和POWER。常用作高速网络和数据中介软件。
  • 内含基于环形缓冲的框架和线程模型,用于无锁的包处理。
  • 常用模块包括Poll Mode Driver(PMD)和Memory Manager(MEMPOOL)等。
  • 主要应用场景包括高性能交换机、路由器、防火墙、TCP私有网关等。
  • 一些知名项目使用DPDK,如:Linux VRouter虚拟路由,Apache Cassandra等。
  • 出于性能考虑,DPDK应用通常需要绕过OS网络堆栈,实现零拷贝。

总之,DPDK是一个用于高性能数据包I/O和处理的重量级开源软件库,广泛应用于高性能网络应用和设备上。它利用零拷贝和无锁设计大幅提升网络数据处理能力。

创建虚拟机

我们使用Vmware创建一个Ubuntu的虚拟机,并添加一块网卡,其中桥接网卡作为DPDK运行的网卡,NAT网卡作为我们SSH连接使用的网卡

image.png

安装使用的Ubuntu版本

gillbert@dpdk:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.7 LTS
Release:	16.04
Codename:	xenial

安装必备的工具包

apt install build-essential python

修改虚拟机网卡信息

将 ethernet0.virtualDev 由 e1000 修改 vmxnet3,因为 vmware 的 vmxnet3 支持多队列网卡

image.png

修改系统的GRUB启动参数

  • 物理机: default_hugepages=1G hugepagesz=1G hugepages=20 isolcpus=0-7
  • 虚拟机: default_hugepages=1G hugepagesz=2M hugepages=1024 isolcpus=0-2

image.png

find_preseed=/preseed.cfg noprompt net.ifnames=0 biosdevname=0 default_hugepagesz=2M hugepagesz=2M hugepages=1024 isolcpus=0-2

更新grub

更新grub,然后重启系统

sudo update-grub

修改网卡配置文件

此时可以看到系统的网卡名变为了eth开头的网卡,因此我们需要修改网卡配置文件

gillbert@dpdk:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0c:29:e5:bd:78 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:e5:bd:82 brd ff:ff:ff:ff:ff:ff
    inet 192.168.222.132/24 brd 192.168.222.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fee5:bd82/64 scope link 
       valid_lft forever preferred_lft forever

image.png

查看系统是否支持多队列网卡

 cat /proc/interrupts  | grep eth0

image.png

可以看到eth0有4个网卡队列eth0-rxtx-0、eth0-rxtx-1、eth0-rxtx-2、eth0-rxtx-3,对应的中断号分别是56、57、58、59。这里虚拟机有多少个CPU就会有多少个队列。

开始编译DPDK

下载DPDK源码

wget https://fast.dpdk.org/rel/dpdk-19.08.2.tar.xz

解压源码

tar -xf dpdk-19.08.2.tar.xz

设置环境变量

export RTE_SDK=/home/gillbert/dpdk-stable-19.08.2/
export RTE_TARGET=x86_64-native-linux-gcc

执行DPDK提供的脚本,开始编译

./usertools/dpdk-setup.sh
------------------------------------------------------------------------------
 RTE_SDK exported as /home/gillbert/dpdk-stable-19.08.2
------------------------------------------------------------------------------
----------------------------------------------------------
 Step 1: Select the DPDK environment to build
----------------------------------------------------------
[1] arm64-armada-linuxapp-gcc
[2] arm64-armada-linux-gcc
[3] arm64-armv8a-linuxapp-clang
[4] arm64-armv8a-linuxapp-gcc
[5] arm64-armv8a-linux-clang
[6] arm64-armv8a-linux-gcc
[7] arm64-bluefield-linuxapp-gcc
[8] arm64-bluefield-linux-gcc
[9] arm64-dpaa2-linuxapp-gcc
[10] arm64-dpaa2-linux-gcc
[11] arm64-dpaa-linuxapp-gcc
[12] arm64-dpaa-linux-gcc
[13] arm64-octeontx2-linuxapp-gcc
[14] arm64-octeontx2-linux-gcc
[15] arm64-stingray-linuxapp-gcc
[16] arm64-stingray-linux-gcc
[17] arm64-thunderx2-linuxapp-gcc
[18] arm64-thunderx2-linux-gcc
[19] arm64-thunderx-linuxapp-gcc
[20] arm64-thunderx-linux-gcc
[21] arm64-xgene1-linuxapp-gcc
[22] arm64-xgene1-linux-gcc
[23] arm-armv7a-linuxapp-gcc
[24] arm-armv7a-linux-gcc
[25] i686-native-linuxapp-gcc
[26] i686-native-linuxapp-icc
[27] i686-native-linux-gcc
[28] i686-native-linux-icc
[29] ppc_64-power8-linuxapp-gcc
[30] ppc_64-power8-linux-gcc
[31] x86_64-native-bsdapp-clang
[32] x86_64-native-bsdapp-gcc
[33] x86_64-native-freebsd-clang
[34] x86_64-native-freebsd-gcc
[35] x86_64-native-linuxapp-clang
[36] x86_64-native-linuxapp-gcc
[37] x86_64-native-linuxapp-icc
[38] x86_64-native-linux-clang
[39] x86_64-native-linux-gcc
[40] x86_64-native-linux-icc
[41] x86_x32-native-linuxapp-gcc
[42] x86_x32-native-linux-gcc

----------------------------------------------------------
 Step 2: Setup linux environment
----------------------------------------------------------
[43] Insert IGB UIO module
[44] Insert VFIO module
[45] Insert KNI module
[46] Setup hugepage mappings for non-NUMA systems
[47] Setup hugepage mappings for NUMA systems
[48] Display current Ethernet/Baseband/Crypto device settings
[49] Bind Ethernet/Baseband/Crypto device to IGB UIO module
[50] Bind Ethernet/Baseband/Crypto device to VFIO module
[51] Setup VFIO permissions

----------------------------------------------------------
 Step 3: Run test application for linux environment
----------------------------------------------------------
[52] Run test application ($RTE_TARGET/app/test)
[53] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)

----------------------------------------------------------
 Step 4: Other tools
----------------------------------------------------------
[54] List hugepage info from /proc/meminfo

----------------------------------------------------------
 Step 5: Uninstall and system cleanup
----------------------------------------------------------
[55] Unbind devices from IGB UIO or VFIO driver
[56] Remove IGB UIO module
[57] Remove VFIO module
[58] Remove KNI module
[59] Remove hugepage mappings

[60] Exit Script

Option: 

开始编译

64位系统出入39

[58] Remove KNI module
[59] Remove hugepage mappings

[60] Exit Script

Option: 39

编译完成后会生成x86_64-native-linux-gcc文件夹

image.png

插入IGB_UIO模块

选择 43 插入 IGB_UIO 模块

 Step 2: Setup linux environment
----------------------------------------------------------
[43] Insert IGB UIO module
[44] Insert VFIO module
[45] Insert KNI module
[46] Setup hugepage mappings for non-NUMA systems
[47] Setup hugepage mappings for NUMA systems
[48] Display current Ethernet/Baseband/Crypto device settings
[49] Bind Ethernet/Baseband/Crypto device to IGB UIO module
[50] Bind Ethernet/Baseband/Crypto device to VFIO module
[51] Setup VFIO permissions

----------------------------------------------------------
 Step 3: Run test application for linux environment
----------------------------------------------------------
[52] Run test application ($RTE_TARGET/app/test)
[53] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)

----------------------------------------------------------
 Step 4: Other tools
----------------------------------------------------------
[54] List hugepage info from /proc/meminfo

----------------------------------------------------------
 Step 5: Uninstall and system cleanup
----------------------------------------------------------
[55] Unbind devices from IGB UIO or VFIO driver
[56] Remove IGB UIO module
[57] Remove VFIO module
[58] Remove KNI module
[59] Remove hugepage mappings

[60] Exit Script

Option: 43

## ERROR: Target does not have the DPDK UIO Kernel Module.
       To fix, please try to rebuild target.

Press enter to continue ...

执行此步骤会报错,错误的原因为我们的系统上没有uio这个内核模块,uio(Userspace I/O)是一种在用户空间与设备之间进行直接通信的机制,用于将网络设备与DPDK绑定以实现高性能数据包处理。因此我们需要先加载uio模块与igb_uio模块。

编译安装uio内核模块

确认系统已安装开发工具包和依赖库
sudo apt-get install libnuma-dev build-essential linux-headers-$(uname -r)

进入DPDK源码目录,并根据目标系统进行配置。

此处的T指定的值和我们开始在

make config T=x86_64-native-linux-gcc

执行配置命令后,使用make命令编译DPDK库及其驱动程序
make -j 4
加载uio内核模块。
sudo modprobe uio
sudo insmod /home/gillbert/dpdk-stable-19.08.2/build/kmod/igb_uio.ko
确认uio内核模块是否已成功加载。
root@dpdk:/home/gillbert/dpdk-stable-19.08.2# lsmod |grep uio
igb_uio                16384  0
uio                    20480  1 igb_uio

重新插入igb_uio模块

在执行此步骤之前,我们需要重新选择39编译一下,然后再选择43插入IGB_UIO模块


Option: 43

Unloading any existing DPDK UIO module
Loading DPDK UIO module

Press enter to continue ...

插入 VFIO 模块

选择 44 插入 VFIO 模块

Option: 44

Unloading any existing VFIO module
Loading VFIO module
chmod /dev/vfio
OK

绑定 igb_uio 模块

选择 49 绑定 igb_uio 模块,输入49后我们再输入eth0网卡


Option: 49


Network devices using kernel driver
===================================
0000:02:02.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=eth1 drv=e1000 unused=igb_uio,vfio-pci *Active*
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=igb_uio,vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

Enter PCI address of device to bind to IGB UIO driver: eth0
Warning: routing table indicates that interface 0000:03:00.0 is active. Not modifying
OK

执行上述命令时,我们发现有个警告信息,提示我们这个网卡是活动的状态不能设置,因此我们需要将这个网卡先down掉再设置

ifconfig eth0 down
[60] Exit Script

Option: 49


Network devices using kernel driver
===================================
0000:02:02.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=eth1 drv=e1000 unused=igb_uio,vfio-pci *Active*
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=igb_uio,vfio-pci 

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

Enter PCI address of device to bind to IGB UIO driver: eth0
OK

测试

运行testpmd进行测试

image.png

Option: 53


  Enter hex bitmask of cores to execute testpmd app on
  Example: to execute app on cores 0 to 7, enter 0xff
bitmask: 7
Launching app
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:02.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15ad:7b0 net_vmxnet3
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)
Port 0: 00:0C:29:E5:BD:78
Checking link statuses...
Done

查看端口信息

输入 show port info 0命令

testpmd> show port info 0

********************* Infos for port 0  *********************
MAC address: 00:0C:29:E5:BD:78
Device name: 0000:03:00.0
Driver name: net_vmxnet3
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 10000 Mbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 1
Maximum number of MAC addresses of hash filtering: 0
VLAN offload: 
  strip off 
  filter off 
  qinq(extend) off 
Supported RSS offload flow types:
  ipv4
  ipv4-tcp
  ipv6
  ipv6-tcp
Minimum size of RX buffer: 1646
Maximum configurable length of RX packet: 16384
Current number of RX queues: 1
Max possible RX queues: 16
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 128
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 8
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 512
TXDs number alignment: 1
Max segment number per packet: 255
Max segment number per MTU/TSO: 16

编译helloword测试程序进行测试

进入 example/helloworl可以直接 make,也可以通过 gcc 命令编译

root@dpdk:/home/gillbert/dpdk-stable-19.08.2/examples/helloworld# gcc -o helloword main.c -I /usr/local/include/dpdk/ -ldpdk -lpthread -lnuma -ldl
root@dpdk:/home/gillbert/dpdk-stable-19.08.2/examples/helloworld# ./helloword -l 0-3 -n 4
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
hello from core 1
hello from core 2
hello from core 3
hello from core 0

重新加载DPDK环境

再次重启系统后,我们发现以前配置的DPDK环境都没有了,因此我们需要重新配置一下

加载内核模块

modprobe uio
sudo insmod /home/gillbert/dpdk-stable-19.08.2/build/kmod/igb_uio.ko

设置环境变量

export RTE_SDK=/home/gillbert/dpdk-stable-19.08.2/
export RTE_TARGET=x86_64-native-linux-gcc

运行初始化脚本

root@dpdk:/home/gillbert/dpdk-stable-19.08.2# ./usertools/dpdk-setup.sh 

加载模块

依次输入运行43, 44

绑定网卡

再输入49,网卡名称我们输入eth0

测试

我们可以按着前面的测试步骤进行测试看下是否加载成功。

参考文章