[development][dpdk][pktgen] 网卡收发包性能测试
一： 多队列及中断信息收集
[root@T185 ~]# ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 512 RX Mini: 0 RX Jumbo: 0 TX: 512
[root@T185 ~]# ls /sys/class/net/eth0/queues/ rx-0 rx-10 rx-12 rx-14 rx-16 rx-18 rx-2 rx-21 rx-23 rx-4 rx-6 rx-8 tx-0 tx-10 tx-12 tx-14 tx-16 tx-18 tx-2 tx-21 tx-23 tx-4 tx-6 tx-8 rx-1 rx-11 rx-13 rx-15 rx-17 rx-19 rx-20 rx-22 rx-3 rx-5 rx-7 rx-9 tx-1 tx-11 tx-13 tx-15 tx-17 tx-19 tx-20 tx-22 tx-3 tx-5 tx-7 tx-9
[root@T185 ~]# cat /proc/interrupts |less -S [root@T185 ~]# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 0: 186 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 8: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi 12: 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 16: 256 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi ehci_hcd:usb1 23: 298 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi ehci_hcd:usb2 88: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 89: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 90: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 91: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 92: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 93: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 94: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 95: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 96: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 97: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 98: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 99: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 100: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 101: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 102: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge aerdrv 103: 24847 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ahci 104: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge isci-msix 105: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge isci-msix 106: 3319852 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-0 107: 981541 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-1 108: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-2 109: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-3 110: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-4 111: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-5 112: 669706 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-6 113: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-7 114: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-8 115: 28009 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-9 116: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-10 117: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-11 118: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-12 119: 28069 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-13 120: 338786 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-14 121: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-15 122: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-16 123: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-17 124: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-18 125: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-19 126: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-20 127: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-21 128: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-22 129: 28004 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-TxRx-23 130: 59 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0 131: 85702 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-0 132: 84484 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-1 133: 89210 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-2 134: 86210 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-3 135: 86086 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-4 136: 89464 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-5 137: 94737 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-6 138: 86082 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-7 139: 84812 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-8 140: 86437 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-9 141: 83901 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-10 142: 90394 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-11 143: 86578 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-12 144: 89534 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-13 145: 92897 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-14 146: 97958 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-15 147: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-16 148: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-17 149: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-18 150: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-19 151: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-20 152: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-21 153: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-22 154: 26913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-TxRx-23 155: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1 157: 97476 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth3 NMI: 736 48 9 4 2 1 133 2 14 4 13 12 6 102 67 2 1 1 1 163 3 640 550 2718 Non-maskable interrupts LOC: 892399 69189 85565 60959 35207 24688 159437 24227 114514 64519 50015 24473 19392 198772 108184 34002 21177 18652 18684 250274 43307 42755165 42680063 2737380 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 736 48 9 4 2 1 133 2 14 4 13 12 6 102 67 2 1 1 1 163 3 640 550 2718 Performance monitoring interrupts IWI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 2281297 404101 4908 813 277 323 594 864 4512 1682 768 369 109 3271 222 508 86 81 145 7654 369 964 460 202 Rescheduling interrupts CAL: 325 389 365 373 385 385 741 491 1434 921 652 488 384 357 387 385 383 384 395 3966 679 586 1914 427 Function call interrupts TLB: 19 5 18 157 3 6 11 7 12 78 2 3 12 781 637 30 25 12 7 807 686 7816 4244 50309 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 217 Machine check polls ERR: 0 MIS: 0 [root@T185 ~]#interrupts
二： 设置CPU隔离
在grub.conf 中，增加isolcpus参数。
三： 修改收发包队列数
[root@T185 tong]# ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 64 RX Mini: 0 RX Jumbo: 0 TX: 512 [root@T185 tong]# ethtool -G eth0 rx 1 tx 1 [root@T185 tong]# ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 64 RX Mini: 0 RX Jumbo: 0 TX: 64ethtool
不能设成1, 最小64．所以，对于中断并没有什么影响．这好像不对，不知道这是什么鬼。
下载最新的驱动，https://downloadcenter.intel.com/download/14687 并查看README
Command Line Parameters ----------------------- If the driver is built as a module, the following optional parameters are used by entering them on the command line with the modprobe command using this syntax: modprobe ixgbe [<option>=<VAL1>,<VAL2>,...] There needs to be a <VAL#> for each network port in the system supported by this driver. The values will be applied to each instance, in function order. For example: modprobe ixgbe InterruptThrottleRate=16000,16000 。。。 。。。 Multiqueue ---------- Valid Range: 0, 1 0 = Disables Multiple Queue support 1 = Enabled Multiple Queue support (a prerequisite for RSS)README
README里给的参数竟然不对，尼玛
[root@T185 tong]# modprobe -r ixgbe [root@T185 tong]# modprobe ixgbe MQ=1,0 [root@T185 tong]# ls /sys/class/net/eth0/queues/ rx-0 rx-10 rx-12 rx-14 rx-16 rx-18 rx-2 rx-21 rx-23 rx-4 rx-6 rx-8 tx-0 tx-10 tx-12 tx-14 tx-16 tx-18 tx-2 tx-21 tx-23 tx-4 tx-6 tx-8 rx-1 rx-11 rx-13 rx-15 rx-17 rx-19 rx-20 rx-22 rx-3 rx-5 rx-7 rx-9 tx-1 tx-11 tx-13 tx-15 tx-17 tx-19 tx-20 tx-22 tx-3 tx-5 tx-7 tx-9 [root@T185 tong]# ls /sys/class/net/eth1/queues/ rx-0 tx-0 [root@T185 tong]#
四：中断绑定：
ｃｐｕ结构：
[root@T185 tong]# ./cpu_layout.py ============================================================ Core and Socket Information (as reported by '/proc/cpuinfo') ============================================================ cores = [0, 1, 2, 3, 4, 5] sockets = [0, 1] Socket 0 Socket 1 -------- -------- Core 0 [0, 12] [6, 18] Core 1 [1, 13] [7, 19] Core 2 [2, 14] [8, 20] Core 3 [3, 15] [9, 21] Core 4 [4, 16] [10, 22] Core 5 [5, 17] [11, 23] [root@T185 tong]#
另外两个工具：lstopo，和hwloc。
中断绑定：
#! /bin/bash function bind() # $lcpus, $irq_list, $set { for i in $2 do echo -n $i ": " if [ $3 -eq 1 ] then echo $1 > /proc/irq/$i/smp_affinity_list echo "Done!" else cat /proc/irq/$i/smp_affinity_list fi done } function usage() { echo "Usage:" echo -e "\tirq_bind.sh 0(print)/1(set) " echo } if [ $# -ne 1 ] then usage exit fi if [ \( "$1" != "1" \) -a \( "$1" != "0" \) ] then usage exit fi irq_list=`cat /proc/interrupts |grep eth0 |awk -F: '{print $1}'` bind "1,13" "$irq_list" $1 irq_list=`cat /proc/interrupts |grep eth1 |awk -F: '{print $1}'` bind "2,14" "$irq_list" $1
注意： 网卡驱动重新加载过了之后，需要重新对中断进行绑定。
五： 发包：
tcpreplay -i eth0 -l0 -L10000000 -p400000 -K syy/50000.pcap
六： pktgen-dpdk
http://pktgen-dpdk.readthedocs.io/en/latest/
版本太多，版本间的修改又都比较大。最后pktgen和dpdk都用了当前的最新版本。重要编译通过了，且运行正常了。
tong@zw:~/src$ ls dpdk-17.05 dpdk-17.05.tar.gz pktgen-3.3.3 pktgen-3.3.3.tar.gz
6.1 正常编译dpdk，并正确设置环境变量
make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
make
tong@zw:~/src$ tail -n2 ~/.bashrc export RTE_SDK=~/src/dpdk-17.05/ export RTE_TARGET=x86_64-native-linuxapp-gcc
6.2 编译并设置 pktgen
make sysctl -n vm.nr_hugepages=8192 vim cfg/default.cfg ./tools/run.py -s hugepages per socket 4096 Set /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages to 4096 Bind devices to DPDK: ['03:00.3']
sudo mount -t hugetlbfs nodev /mnt/huge
6.3 运行
http://pktgen-dpdk.readthedocs.io/en/latest/usage_pktgen.html#usage-pktgen
最终配置如下：
tong@zw:~/src/pktgen-3.3.3$ cat cfg/default.cfg # Setup configuration setup = { 'devices': [ '03:00.3' ], 'opts': [ '-b igb_uio' ] } # Run command and options run = { 'dpdk': [ '-l 4,5', '-n 1', '--proc-type auto', '--log-level 7', '--file-prefix pg' ], 'blacklist': [ ], 'pktgen': [ '-T', '-P', '--crc-strip', '-m [5].0', ], 'misc': [ '-f themes/black-yellow.theme' ] } tong@zw:~/src/pktgen-3.3.3$cfg/default.cfg
tong@zw:~/src/pktgen-3.3.3$ ./tools/run.py cfg/default.cfg
6.4 使用
Pktgen:/> set 0 count 5 Pktgen:/> set 0 size 64 Pktgen:/> start 0 Pktgen:/> set 0 size 512 Pktgen:/> start 0 Pktgen:/> set 0 size 70 Pktgen:/> start 0 Pktgen:/>
发了三次，每次五个包，长度分别为 64 / 512 / 70
详细的参数参考：http://pktgen-dpdk.readthedocs.io/en/latest/commands.html#runtime-options-and-commands
也可以在Pktgen的命令行里 help
还可以用LUA进行配置，但是我不会http://pktgen-dpdk.readthedocs.io/en/latest/lua.html
6.5 其他
但是所有的 TCP包都是SYN的，不能握手。与我的测试目的还是有些区别。
6.6 发送pcap
在命令行使用 -s P:PCAP_file 参数。
在pktgen中使用如下命令：
Pktgen:/> enable 0 pcap
使用page pcap命令可以查看pcap详细内容：
Pktgen:/> page pcap
之后使用正常的命令，便可以发送数据包了。
七。 pktgen-linux
据说之支持 UDP，不支持TCP。
八。测试数据
[development][dpdk][pktgen] 网卡收发包性能测试-详细数据
九。 使用gprof查找性能瓶颈。
9.1 使用gcc -pg编译
9.2 正常运行程序，并正常退出，系统会生成一个gmon.out文件。
9.3 使用如下命令查看输出
gprof app_name gmon.out
9.4 通过观察日志，找到性能瓶颈，是一个链表遍历查找的函数。
优质内容筛选与推荐>>1、Part 4 Data Directory & Control files
2、51定时器延时函数（精确延时）
3、Sass新手指南
4、webtrends
5、前端用到的设计模式之开闭原则. 里氏代换原则