PcapPlusPlus DPDK and PF_RING wrappers

DPDK support

The Data Plane Development Kit (DPDK) is a set of data plane libraries and network interface controller drivers for fast packet processing. The DPDK provides a programming framework for Intel x86 processors and enables faster development of high speed data packet networking applications. It is provided and supported under the open source BSD license (taken from Wikipedia )

DPDK provides packet processing in line rate using kernel bypass for a large range of network interface cards. Notice that not every NIC supports DPDK as the NIC needs to support the kernel bypass feature. You can read more about DPDK in DPDK's web-site and get a list of supported NICs here.

Also, you can get more details about DPDK and PcapPlusPlus wrap for DPDK in the documentation of DpdkDevice.h and DpdkDeviceList.h header files.

Download and install

Download and install instructions for DPDK are on this page: http://dpdk.org/download

Please make sure to install DPDK in the following way:

make config T=[platform_type_string] && make

So what does PcapPlusPlus offer for DPDK?
  • An easy-to-use C++ wrapper (as DPDK is written in C) that encapsulates DPDK's main functionality but doesn't hit packet procssing performance
  • Encapsulation of DPDK's initialization process - both outside and inside the application - using simple scripts and methods
  • A C++ class wrapper for DPDK's packet struct (mbuf) which offers most common functionality
  • Seamless integration to other PcapPlusPlus capabilities, for example: receivce packets with DPDK, parse them with Packet++ protocol layers and save them to a pcap file
PcapPlusPlus configuration for DPDK
  1. Download and compile DPDK on your system (see the link above)
  2. Note that PcapPlusPlus supports DPDK versions from 16.11 to 19.02 (inclusive). Previous (and maybe also newer) versions won't work
  3. Once DPDK compiles successfully you need to run PcapPlusPlus configure-linux.sh and type "y" in "Compile PcapPlusPlus with DPDK?"
  4. configure-linux.sh will ask for DPDK's path (i.e /home/user/dpdk-stable-16.11.1)
  5. Then you can compile PcapPlusPlus as usual (using make, see below)
DPDK initialization with PcapPlusPlus

DPDK has 2 steps of initialization: one that configures Linux to support DPDK applications and the other at application startup that initializes DPDK. PcapPlusPlus wraps both of them with easy-to-use interfaces:

Before application is run

DPDK requires several Linux configurations to run:

  1. DPDK uses the Linux huge-pages mechanism for faster virtual to physical page conversion resulting in better performance. So huge-pages must be set before a DPDK application is run
  2. DPDK uses a designated kernel module for the kernel-bypass mechanism. This module should be loaded into the kernel
  3. The user needs to state which NICs will move to DPDK control and which will stay under Linux control

PcapPlusPlus offers a simple script that automatically configures all of these. The script is under PcapPlusPlus root directory and is called setup-dpdk.sh. The script takes as an input the following parameters:

  1. -p : the amount of huge pages to allocate. By default each huge-page size is 2048KB
  2. -n : a comma-separated list of all NICs that will be unbinded from Linux and move to DPDK control. Only these NICs will be used by DPDK, the others will stay under Linux control. For example: eth0,eth1 will move these 2 interfaces under DPDK control - assuming this NIC is supported by DPDK. You can use the -h switch for help.

If everything went well the system is ready to run a DPDK application and the script output should look like this:

*******************************
PcapPlusPlus setup DPDK script 
*******************************

1. Reserve huge-pages - DONE!
2. Install kernel module - DONE!
3. Bind network adapters - DONE!

Network devices using DPDK-compatible driver
============================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller' drv=igb_uio unused=e1000
0000:00:08.0 '82545EM Gigabit Ethernet Controller (Copper)' drv=igb_uio unused=e1000

Network devices using kernel driver
===================================
0000:00:09.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio *Active*
0000:00:0a.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth3 drv=e1000 unused=igb_uio 

Other network devices
=====================

Setup DPDK completed
At application startup

before using DPDK inside the application DPDK should be configured on application startup. This configuration includes:

  1. Verify huge-pages, kernel module and NICs are set
  2. Initialize DPDK internal structures and memory, poll-mode-drivers etc.
  3. Prepare CPU cores that will be used by the application
  4. Initialize packet memory pool
  5. Configure each NIC controlled by DPDK

These steps are wrapped in one static method that should be called once in application startup:

DpdkDeviceList::initDpdk()

Tests and limitations
  • All unit-tests I perfromed are in Pcap++Test
  • In addition you try the DPDK example application (Examples/DpdkExample-FilterTraffic)
  • Supported DPDK versions are 16.11 to 18.02 (inclusive)
  • So far I managed to test the code on 2 virtual PMDs only:
    1. VMXNET3 - a VMWare guest driver
    2. E1000/EM - 1GbE Intel NIC but I tested it as virtual NIC in VirtualBox guest
  • I hope I'll be able to test it on some more (preferebly non-virtual) NICs soon - I'll update if/when I do
  • Operating systems and configurations tested:
    1. Ubuntu 14.04.1 LTS 64bit with kernel 3.x and gcc 4.8.x
    2. Ubuntu 12.04.5 LTS 32bit with kernel 3.x and gcc 4.6.x
    3. CentOS 7.1 64bit with kernel 3.x and gcc 4.8.x

PF_RING support

PcapPlusPlus provides a clean and simple C++ wrapper API for Vanilla PF_RING. Currently only Vanilla PF_RING is supported which provides significant performance improvement in comparison to libpcap or Linux kernel, but PF_RING DNA or ZC (which allows kernel bypass and zero-copy of packets from NIC to user-space) isn't supported. I hope I'll be able to add it in the future.

You can read more about PF_RING in ntop web-site: http://www.ntop.org/products/pf_ring/ and in PF_RING user guide: https://svn.ntop.org/svn/ntop/trunk/PF_RING/doc/UsersGuide.pdf

In order to compile PcapPlusPlus with PF_RING you need to:

  1. Download PF_RING from ntop's web-site: http://www.ntop.org/get-started/download/#PF_RING
  2. Note that I used PcapPlusPlus with PF_RING versions 6.2.0 - 6.4.1. I can't guarantee it'll work with previous or later versions
  3. Once PF_RING is compiled successfully, you need to run PcapPlusPlus configure-linux.sh and type "y" in "Compile PcapPlusPlus with PF_RING?"
  4. Then you can compile PcapPlusPlus as usual
  5. Before you activate any PcapPlusPlus application that uses PF_RING, don't forget to enable PF_RING kernel module. If you forget to do that, PcapPlusPlus will output an appropriate error on startup which will remind you:
sudo insmod <PF_RING_LOCATION>/kernel/pf_ring.ko