While it would be great if we had direct, physical access to multiple compute nodes with fancy hardware (required for HPC research), more often than not we find ourselves having access to such machines on a cluster (such as the JLSE). So, installing software and libraries required for your work is a tad bit more tedious than what would've been a simple `yum install`. One such software suite is linux-rdma/perftest which is dependent on libraries contained in linux-rdma/rdma-core. Installing it in your all-access userspace is what this blog post will briefly cover.
Before anything, some overview: perftest is a test suite consisting of micro-benchmarks to measure the performance of message transfers and RDMA operations such as reads, writes, and atomics using InfiniBand hardware. This test suite requires core libraries such as libibverbs (the de-facto standard implementation of the Verbs API), librdmacm and libibumad. These libraries are now maintained in the linux-rdma/rdma-core git repository. Which is what we need to tackle first.
Building linux-rdma/rdma-core
After a `git clone` of rdma-core, simply run `./build.sh` inside the repository. If everything works, you can skip to the Building perftest section. When I ran `./build.sh`, for instance, it complained that `libnl-3` or `libnl-1` was missing. In your case, this could be any other library. I will be writing about the lessons from my story.
- First, I downloaded the source code for `libnl-3.x` and configured it to be installed in my userspace using `./configure --prefix=~/libnl-files && make install`
- export LD_LIBRARY_PATH=~/libnl-files/lib:$LD_LIBRARY_PATH. So that it knows where to find your library during runtime.
- export LIBRARY_PATH=$LD_LIBRARY_PATH. So it knows where to find your libraries during linking.
- export PKG_CONFIG_PATH=~libnl-files/lib/pkgconfig. This is because `build.sh` uses CMake
- `./build.sh`
... and that should successfully compile rdma-core (unless it complains about some other library that is missing for which you will have to repeat similar steps as above).
Building linux-rdma/perftest
First, set your environment variables to point to your freshly installed binaries:
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH=~/rdma-core/build/lib
- export LIBRARY_PATH=$LD_LIBRARY_PATH
Second, git clone the perftest suite and after the next three steps, you should be done!
- Run `./autogen.sh` inside the perftest suite
- Configure the Makefile to install in your userspace and build it `./configure --prefix=~/perftest-files && make install`
- Verify that the micro-benchmark binaries are seeing using your userspace rdma-core libraries using `ldd ~/perftest-files/bin/ib_send_bw`