..    Copyright (c) 2019 Intel
..
..  Licensed under the Apache License, Version 2.0 (the "License");
..  you may not use this file except in compliance with the License.
..  You may obtain a copy of the License at
..
..      http://www.apache.org/licenses/LICENSE-2.0
..
..  Unless required by applicable law or agreed to in writing, software
..  distributed under the License is distributed on an "AS IS" BASIS,
..  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
..  See the License for the specific language governing permissions and
..  limitations under the License.

.. |br| raw:: html

   <br />

Setup Configuration
===================

A.1 Setup Configuration
-----------------------
The configuration shown in Figure 26 shows how to set up a test
environment to execute xRAN scenarios where O-DU and 0-RU are simulated
using the sample application. This setup allows development and
prototyping as well as testing of xRAN specific functionality. The O-DU
side can be instantiated with a full 5G NR L1 reference as well. The
configuration differences of the 5G NR l1app configuration are provided
below. Steps for running the sample application on the O-DU side and
0-RU side are the same, except configuration file options may be
different.

.. image:: images/Setup-for-xRAN-Testing.jpg
  :width: 400
  :alt: Figure 26. Setup for xRAN Testing

Figure 26. Setup for xRAN Testing

.. image:: images/Setup-for-xRAN-Testing-with-PHY-and-Configuration-C3.jpg
  :width: 400
  :alt: Figure 27. Setup for xRAN Testing with PHY and Configuration C3

Figure 27. Setup for xRAN Testing with PHY and Configuration C3

A.2 Prerequisites
-----------------
Each server in Figure 26 requires the following:

-  Wolfpass server according to recommended BOM for FlexRAN such as
   Intel® Xeon® Skylake Gold 6148 FC-LGA3647 2.4 GHz 27.5 MB 150W 20
   cores (two sockets)

-  BIOS settings:

-  Intel(R) Virtualization Technology Enabled

-  Intel(R) VT for Directed I/O - Enabled

-  ACS Control - Enabled

-  Coherency Support - Disabled

-  Front Haul networking cards:

-  Intel® Ethernet Converged Network Adapter XL710-QDA2

-  Intel® Ethernet Converged Network Adapter XXV710-DA2

-  Intel® FPGA Programmable Acceleration Card (Intel® FPGA PAC) N3000

**The Front Haul NIC requires support for PTP HW timestamping.**

The recommended configuration for NICs is::


    ethtool -i enp216s0f0
    
    driver: i40e
    
    version: 2.9.21
    
    firmware-version: 6.80 0x80003d05 1.2007.0
    
    expansion-rom-version:
    
    bus-info: 0000:d8:00.0
    
    supports-statistics: yes
    
    supports-test: yes
    
    supports-eeprom-access: yes
    
    supports-register-dump: yes
    
    supports-priv-flags: yes
    
   [root@5gnr-sc12-xran testmac]# ethtool -T enp216s0f0::
   
   
   Time stamping parameters for enp216s0f0:
   
   Capabilities:
   
   hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
   
   software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
   
   hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
   
   software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
   
   software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
   
   hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
   
   PTP Hardware Clock: 2
   
   Hardware Transmit Timestamp Modes:
   
   off (HWTSTAMP_TX_OFF)
   
   on (HWTSTAMP_TX_ON)
   
   Hardware Receive Filter Modes:
   
   none (HWTSTAMP_FILTER_NONE)
   
   ptpv1-l4-sync (HWTSTAMP_FILTER_PTP_V1_L4_SYNC)
   
   ptpv1-l4-delay-req (HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ)
   
   ptpv2-l4-event (HWTSTAMP_FILTER_PTP_V2_L4_EVENT)
   
   ptpv2-l4-sync (HWTSTAMP_FILTER_PTP_V2_L4_SYNC)
   
   ptpv2-l4-delay-req (HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ)
   
   ptpv2-l2-event (HWTSTAMP_FILTER_PTP_V2_L2_EVENT)
   
   ptpv2-l2-sync (HWTSTAMP_FILTER_PTP_V2_L2_SYNC)
   
   ptpv2-l2-delay-req (HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ)
   
   ptpv2-event (HWTSTAMP_FILTER_PTP_V2_EVENT)
   
   ptpv2-sync (HWTSTAMP_FILTER_PTP_V2_SYNC)
   
   ptpv2-delay-req (HWTSTAMP_FILTER_PTP_V2_DELAY_REQ)

PTP Grand Master is required to be available in the network to provide
synchronization of both O-DU and RU to GPS time.

The software package includes Linux\* CentOS\* operating system and RT
patch according to *Cloud-Native*-platform *Setup* document (refer to
Table 2). Only real-time HOST is required.

1.Installing Intel® C++ Compiler v19.0.3 is preferred. or you could get
Intel® C++ Compiler through below link with community license,
however the version you could get is always latest version, the
verification for that version might not be performed yet, please
feedback through O-DU Low project WIKI page if you meet an issue. |br|
`https://software.intel.com/en-us/system-studio/choose-download <https://software.intel.com/en-us/system-studio/choose-download%20>`__

2.Download DPDK 18.08.

3.With FlexRAN BBDev patch as per release 20.02. (Note currently this may require a license from Intel)

4.Change DPDK files according to below diff information which relevant to O-RAN FH::

    diff --git a/drivers/net/i40e/i40e_ethdev.c
    b/drivers/net/i40e/i40e_ethdev.c
    
    index 85a6a86..236fbe0 100644
    
    --- a/drivers/net/i40e/i40e_ethdev.c
    
    +++ b/drivers/net/i40e/i40e_ethdev.c
    
    @@ -2207,7 +2207,7 @@ void i40e_flex_payload_reg_set_default(struct
    i40e_hw \*hw)
    
    /\* Map queues with MSIX interrupt \*/
    
    main_vsi->nb_used_qps = dev->data->nb_rx_queues -
    
    pf->nb_cfg_vmdq_vsi \* RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM;
    
    - i40e_vsi_queues_bind_intr(main_vsi, I40E_ITR_INDEX_DEFAULT);
    
    + i40e_vsi_queues_bind_intr(main_vsi, I40E_ITR_INDEX_NONE);
    
    i40e_vsi_enable_queues_intr(main_vsi);
    
    /\* Map VMDQ VSI queues with MSIX interrupt \*/
    
    @@ -2218,6 +2218,10 @@ void i40e_flex_payload_reg_set_default(struct
    i40e_hw \*hw)
    
    i40e_vsi_enable_queues_intr(pf->vmdq[i].vsi);
    
    }
    
    + i40e_aq_debug_write_global_register(hw,
    
    + 0x0012A504,
    
    + 0, NULL);
    
    +
    
    /\* enable FDIR MSIX interrupt \*/
    
    if (pf->fdir.fdir_vsi) {
    
    i40e_vsi_queues_bind_intr(pf->fdir.fdir_vsi,
    
    diff --git a/drivers/net/i40e/i40e_ethdev_vf.c
    b/drivers/net/i40e/i40e_ethdev_vf.c
    
    index 001c301..6f9ffdb 100644
    
    --- a/drivers/net/i40e/i40e_ethdev_vf.c
    
    +++ b/drivers/net/i40e/i40e_ethdev_vf.c
    
    @@ -640,7 +640,7 @@ struct rte_i40evf_xstats_name_off {
    
    map_info = (struct virtchnl_irq_map_info \*)cmd_buffer;
    
    map_info->num_vectors = 1;
    
    - map_info->vecmap[0].rxitr_idx = I40E_ITR_INDEX_DEFAULT;
    
    + map_info->vecmap[0].rxitr_idx = I40E_ITR_INDEX_NONE;
    
    map_info->vecmap[0].vsi_id = vf->vsi_res->vsi_id;
    
    /\* Alway use default dynamic MSIX interrupt \*/
    
    map_info->vecmap[0].vector_id = vector_id;
    
    diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
    b/drivers/net/ixgbe/ixgbe_ethdev.c
    
    index 26b1927..018eb8f 100644
    
    --- a/drivers/net/ixgbe/ixgbe_ethdev.c
    
    +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
    
    @@ -3705,7 +3705,7 @@ static int
    ixgbevf_dev_xstats_get_names(__rte_unused struct rte_eth_dev \*dev,
    
    \* except for 82598EB, which remains constant.
    
    \*/
    
    if (dev_conf->txmode.mq_mode == ETH_MQ_TX_NONE &&
    
    - hw->mac.type != ixgbe_mac_82598EB)
    
    + hw->mac.type != ixgbe_mac_82598EB && hw->mac.type !=
    ixgbe_mac_82599EB)
    
    dev_info->max_tx_queues = IXGBE_NONE_MODE_TX_NB_QUEUES;
    
    }
    
    dev_info->min_rx_bufsize = 1024; /\* cf BSIZEPACKET in SRRCTL register
    \*/
    
    diff --git a/lib/librte_eal/common/include/rte_dev.h
    b/lib/librte_eal/common/include/rte_dev.h
    
    old mode 100644
    
    new mode 100755

5.Build and install DPDK::

   [root@xran dpdk]# ./usertools/dpdk-setup.sh
   
   select [16] x86_64-native-linuxapp-icc
   
   select [19] Insert VFIO module
   
   exit [35] Exit Script

6.Make below file changes in dpdk that assure i40e to get best
latency of packet processing::

    --- i40e.h 2018-11-30 11:27:00.000000000 +0000
    
    +++ i40e_patched.h 2019-03-06 15:49:06.877522427 +0000
    
    @@ -451,7 +451,7 @@
    
    #define I40E_QINT_RQCTL_VAL(qp, vector, nextq_type) \\
    
    (I40E_QINT_RQCTL_CAUSE_ENA_MASK \| \\
    
    - (I40E_RX_ITR << I40E_QINT_RQCTL_ITR_INDX_SHIFT) \| \\
    
    + (I40E_ITR_NONE << I40E_QINT_RQCTL_ITR_INDX_SHIFT) \| \\
    
    ((vector) << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) \| \\
    
    ((qp) << I40E_QINT_RQCTL_NEXTQ_INDX_SHIFT) \| \\
    
    (I40E_QUEUE_TYPE_##nextq_type << I40E_QINT_RQCTL_NEXTQ_TYPE_SHIFT))
    
    --- i40e_main.c 2018-11-30 11:27:00.000000000 +0000
    
    +++ i40e_main_patched.c 2019-03-06 15:46:13.521518062 +0000
    
    @@ -15296,6 +15296,9 @@
    
    pf->hw_features \|= I40E_HW_HAVE_CRT_RETIMER;
    
    /\* print a string summarizing features \*/
    
    i40e_print_features(pf);
    
    +
    
    + /\* write to this register to clear rx descriptor \*/
    
    + i40e_aq_debug_write_register(hw, 0x0012A504, 0, NULL);
    
    return 0;
    
A.3 Configuration of System
---------------------------
1.Boot Linux with the following arguments::

    cat /proc/cmdline
    
    BOOT_IMAGE=/vmlinuz-3.10.0-957.10.1.rt56.921.el7.x86_64
    root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root
    rd.lvm.lv=centos/swap intel_iommu=on iommu=pt usbcore.autosuspend=-1
    selinux=0 enforcing=0 nmi_watchdog=0 softlockup_panic=0 audit=0
    intel_pstate=disable cgroup_memory=1 cgroup_enable=memory mce=off
    idle=poll hugepagesz=1G hugepages=40 hugepagesz=2M hugepages=0
    default_hugepagesz=1G isolcpus=1-19,21-39 rcu_nocbs=1-19,21-39
    kthread_cpus=0,20 irqaffinity=0,20 nohz_full=1-19,21-39
    
2.Download from Intel Website and install updated version of i40e
driver if needed. The current recommended version of i40e is x2.9.21.

3.Identify PCIe Bus address of the Front Haul NIC::

    lspci \|grep Eth
    
    19:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
    SFI/SFP+ Network Connection (rev 01)
    
    19:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
    SFI/SFP+ Network Connection (rev 01)
    
    41:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722
    for 10GBASE-T (rev 04)
    
    41:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722
    for 10GBASE-T (rev 04)
    
    d8:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710
    for 40GbE QSFP+ (rev 02) <<< port used for FH
    
    d8:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710
    for 40GbE QSFP+ (rev 02)
    
4.Identify the Ethernet device name::

    ethtool -i enp216s0f0
    
    driver: i40e
    
    version: 2.9.21
    
    firmware-version: 6.80 0x80003d05 1.2007.0
    
    expansion-rom-version:
    
    bus-info: 0000:d8:00.0
    
    supports-statistics: yes
    
    supports-test: yes
    
    supports-eeprom-access: yes
    
    supports-register-dump: yes
    
    supports-priv-flags: yes

5.Enable two virtual functions (VF) on the device::

    echo 2 > /sys/class/net/enp216s0f0/device/sriov_numvfs

More information about VFs supported by Intel NICs can be found at
https://doc.dpdk.org/guides/nics/intel_vf.html.

The resulting configuration can look like the listing below, where two
new VFs were added::

    lspci|grep Eth
    
    19:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
    SFI/SFP+ Network Connection (rev 01)
    
    19:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
    SFI/SFP+ Network Connection (rev 01)
    
    41:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722
    for 10GBASE-T (rev 04)
    
    41:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722
    for 10GBASE-T (rev 04)
    
    d8:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710
    for 40GbE QSFP+ (rev 02)
    
    d8:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710
    for 40GbE QSFP+ (rev 02)
    
    d8:02.0 Ethernet controller: Intel Corporation XL710/X710 Virtual
    Function (rev 02)
    
    d8:02.1 Ethernet controller: Intel Corporation XL710/X710 Virtual
    Function (rev 02)

6.Configure MAC address and VLAN settings for VFs for XRAN, based on
requirements for xRAN scenario and assignment of VLAN ID using IP
tool perform configuration of VF as shown below::

    [root@xran app]# ip link set enp216s0f0 vf 0 mac 00:11:22:33:44:66 vlan
    2
    
    [root@xran app]# ip link set enp216s0f0 vf 1 mac 00:11:22:33:44:66 vlan
    1
    
    [root@xran app]# ip link show
    
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode
    DEFAULT qlen 1
    
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    
    2: enp65s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
    UP mode DEFAULT qlen 1000
    
    link/ether a4:bf:01:3e:6b:79 brd ff:ff:ff:ff:ff:ff
    
    3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
    mode DEFAULT qlen 1000
    
    link/ether a4:bf:01:3e:6b:7a brd ff:ff:ff:ff:ff:ff
    
    4: enp25s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
    UP mode DEFAULT qlen 1000
    
    link/ether 90:e2:ba:d3:b2:ec brd ff:ff:ff:ff:ff:ff
    
    5: enp129s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
    state DOWN mode DEFAULT qlen 1000
    
    link/ether 3c:fd:fe:a8:e0:70 brd ff:ff:ff:ff:ff:ff
    
    6: enp129s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
    state DOWN mode DEFAULT qlen 1000
    
    link/ether 3c:fd:fe:a8:e0:71 brd ff:ff:ff:ff:ff:ff
    
    7: enp216s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
    UP mode DEFAULT qlen 1000
    
    link/ether 3c:fd:fe:9e:93:68 brd ff:ff:ff:ff:ff:ff
    
    vf 0 MAC 00:11:22:33:44:66, vlan 2, spoof checking on, link-state auto,
    trust off
    
    vf 1 MAC 00:11:22:33:44:66, vlan 1, spoof checking on, link-state auto,
    trust off
    
    8: enp25s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
    state DOWN mode DEFAULT qlen 1000
    
    link/ether 90:e2:ba:d3:b2:ed brd ff:ff:ff:ff:ff:ff
    
    9: enp216s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
    UP mode DEFAULT qlen 1000
    
    link/ether 3c:fd:fe:9e:93:69 brd ff:ff:ff:ff:ff:ff
    
    12: enp216s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
    UP mode DEFAULT qlen 1000
    
    link/ether 96:fa:4d:04:4d:87 brd ff:ff:ff:ff:ff:ff
    
    13: enp216s2f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
    state UP mode DEFAULT qlen 1000
    
    link/ether a6:67:49:bb:bd:5e brd ff:ff:ff:ff:ff:ff

After this step FH NIC is configured.

VF for C-plane is VF1 on PH enp216s0f0, it has ETH mac address
00:11:22:33:44:66 and VLAN tag 1. PCIe Bus address is VF1 is d8:02.1

VF for U-plane is VF0 on PH enp216s0f0, it has ETH mac address
00:11:22:33:44:66 and VLAN tag 2. PCIe Bus address is VF1 is d8:02.0

A.4 Install and Configure Sample Application
--------------------------------------------
To install and configure the sample application:

1. Set up the environment:

   export GTEST_ROOT=`pwd`/gtest-1.7.0
   
   export RTE_SDK=`pwd`/dpdk-18.08
   
   export RTE_TARGET=x86_64-native-linuxapp-icc
   
   export MLOG_DIR=`pwd`/flexran_l1_sw/libs/mlog
   
   export XRAN_DIR=`pwd`/flexran_xran

2. Compile xRAN library and test the application:

   [turner@xran home]$ cd $XRAN_DIR
   
   [turner@xran xran]$ ./build.sh
   
3. Configure the sample app.

IQ samples can be generated using Octave\* and script
libs/xran/app/gen_test.m. (CentOS\* has octave-3.8.2-20.el7.x86_64
compatible with get_test.m)

Other IQ sample test vectors can be used as well. The format of IQ
samples is binary int16_t I and Q for N slots of the OTA RF signal. For
example, for mmWave, it corresponds to 792RE*2*14symbol*8slots*10 ms =
3548160 bytes per antenna. Refer to comments in gen_test.m to correctly
specify the configuration for IQ test vector generation.

Update config_file_o_du.dat (or config_file_o_ru.dat) with a suitable
configuration for your scenario.

Update run_o_du.sh (run_o_ru.sh) with PCIe bus address of VF0 and VF1
used for U-plane and C-plane correspondingly::

    ./build/sample-app ./usecase/mu0_10mhz/config_file_o_du.dat 0000:d8:02.0
    0000:d8:02.1

4. Run application using run_o_du.sh (run_o_ru.sh).