Run DPDK in Docker Container

1. Setup up a CentOS 7 host

Of course, this CentOS 7 host can also be a VM. And that's exactly how I setup up my host.

For example, you can use VMware Workstation in Windows to setup a CentOS 7 host.

After installing the OS for the host, you can install the essential tools by following commands:

// Update the kernel version to the latest
# yum update kernel kernel-devel kernel-headers

// Install more tools
# yum install git vim pciutils wget \
    gcc automake autoconf libtool make net-tools \
    libhugetlbfs-utils libpcap-devel

// Install docker and pull centos 7 docker images
# yum install docker
# chkconfig docker on
# service docker start
# docker pull centos:7

// Install pipework for adding multiple NICs
# git clone https://github.com/jpetazzo/pipework.git
# cp pipework/pipework /usr/bin

Note1: The kernel version of host should be the same with the docker image. Otherwise the compiling of DPDK may encounter errors.

Note2: The virtual driver of VMware host need to change to vmxnet3, for example

# vim centos7.vmx
ethernet0.virtualDev = "vmxnet3"

Note3: The VMWARE virtual host should setup 3 NICs.

2. Create DPDK images

This step is inspired by docker-dpdk, and change the version of DPDK and PKTGEN version to the latest.

# cat Dockerfile
# FROM registry.access.redhat.com/rhel7/rhel-tools
# MAINTAINER [email protected]
FROM centos:7

LABEL RUN docker run -it --privileged -v /sys/bus/pci/drivers:/sys/bus/pci/drivers -v /sys/kernel/mm/hugepages:/sys/kernel/mm/hugepages -v /sys/devices/system/node:/sys/devices/system/node -v /dev:/dev --name NAME -e NAME=NAME -e IMAGE=IMAGE IMAGE"

# Setup yum repos, or use subscription-manager

# Install DPDK support packages.
RUN yum install -y sudo libhugetlbfs-utils libpcap-devel \
    kernel kernel-devel kernel-headers

# Install more packages.
RUN yum install -y sudo pciutils vim wget \
    gcc automake autoconf libtool make net-tools \
    expect

# Build DPDK and pktgen-dpdk for x86_64-native-linuxapp-gcc.
WORKDIR /root
COPY ./build_dpdk.sh /root/build_dpdk.sh
COPY ./dpdk-profile.sh /etc/profile.d/
RUN /root/build_dpdk.sh

# Defaults to a bash shell, you could put your DPDK-based application here.
CMD ["/usr/bin/bash"]

# cat build_dpdk.sh
#!/bin/bash

################################################################################
#
#  build_dpdk.sh
#
#             - Build DPDK and pktgen-dpdk for 
#
#  Usage:     Adjust variables below before running, if necessary.
#
#  MAINTAINER:  [email protected]
#
#
################################################################################

################################################################################
#  Define Global Variables and Functions
################################################################################

URL=http://dpdk.org/browse/dpdk/snapshot/dpdk-2.2.0.tar.gz
BASEDIR=/root
VERSION=2.2.0
PACKAGE=dpdk
DPDKROOT=$BASEDIR/$PACKAGE-$VERSION
CONFIG=x86_64-native-linuxapp-gcc
export RTE_SDK=/root/dpdk-2.2.0


# Download/Build DPDK
cd $BASEDIR
wget -q $URL
tar xf dpdk-2.2.0.tar.gz
cd $DPDKROOT
make config T=$CONFIG
sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config
mkdir -p /opt/dpdk-2.2.0
make config T=$CONFIG DESTDIR=/opt/dpdk-2.2.0 install

# Download/Build pktgen-dpdk
URL=http://dpdk.org/browse/apps/pktgen-dpdk/snapshot/pktgen-2.9.12.tar.xz
BASEDIR=/root
VERSION=2.9.12
PACKAGE=pktgen
PKTGENROOT=$BASEDIR/$PACKAGE-$VERSION
cd $BASEDIR
wget -q $URL
tar xf pktgen-2.9.12.tar.xz

# Silence compiler info message
# sed -i '/Wwrite-strings$/ s/$/ -Wno-unused-but-set-variable/' $DPDKROOT/mk/toolchain/gcc/rte.vars.mk
cd $PKTGENROOT
make
ln -s $PKTGENROOT/app/app/$CONFIG/pktgen /usr/bin

3. Create DPDK container

After building the DPDK images, we need create more NICs to test DPDK applications. For example,

# cat create_dpdk_container.sh
#!/bin/bash

DockerImages="dpdk"
DockerContainer="dpdk_container"

// use --privileged for inseting kernel modules  
docker run -it -d --privileged -v /sys/bus/pci/drivers:/sys/bus/pci/drivers -v /sys/kernel/mm/hugepages:/sys/kernel/mm/hugepages -v /sys/devices/system/node:/sys/devices/system/node -v /dev:/dev --name $DockerContainer $DockerImages

# Add additional NICs for DPDK tests
sudo pipework docker0 -i eth1 $DockerContainer  172.17.0.22/16
sudo pipework docker0 -i eth2 $DockerContainer  172.17.0.23/16

4. Setup DPDK in container

The last step is setup environment for DPDK and use test-pmd to test the usability.

@host # docker attach dpdk_container
@dpdk container # menu  // setup linuxapp enviroment
==> choose [17] to insert igb_uio kernel module
==> choose [18] to insert vfio kernel module
==> choose [19] to insert KNI(kernel interface) kernel module
==> choose [21] to setup hugepage. (e.g. 512 * 2MB)
  Note: Please setup enough hugepages, otherwise may encounter "Creation of mbuf pool for socket 0 failed" error.
==> choose [28] to check hugepage settings.
==> choose [23] to bind divice to DPDK-compatible driver
  Note: In our case, the last two nic added by pipework can be used to bind
==> choose [22] display current ethernet device settings
==> choose [27] run testpmd application(e.g. use 0x3 as bitmask)
...  
testpmd> show

The above steps before [27] can be automated by the following script.

# cat setup-dpdk.sh
#!/bin/bash

RTE_SDK=/root/dpdk-2.2.0

echo -e "17\n \n 18\n \n 19\n \n 21\n 512\n \n 23\n 0000:0b:00.0\n \n 23\n 0000:13:00.0\n \n q\n" |
source $RTE_SDK/tools/setup.sh

Until now, a fully functional DPDK testbed has been setup in Docker container. Feel free to explore. :)

References

docker-dpdk
Can you run Intel's Data-plane Development Kit(DPDK) in a Docker container ? Yep.
pipework How to add multiple interfaces to docker container and set ip addresses manually? 7. Resolved Issues