Host-configs

To capture (and revision control) build options, third party library paths, etc., we recommend using CMake’s initial-cache file mechanism. This feature allows you to pass a file to CMake that provides variables to bootstrap the configuration process.

You can pass initial-cache files to cmake via the -C command line option.

cmake -C config_file.cmake

We call these initial-cache files host-config files since we typically create a file for each platform or for specific hosts, if necessary.

These files use standard CMake commands. CMake set() commands need to specify CACHE as follows:

set(CMAKE_VARIABLE_NAME {VALUE} CACHE PATH "")

Here is a snippet from a host-config file that specifies compiler details for using specific gcc (version 4.9.3 in this case) on the LLNL Pascal cluster.

Building and Testing on Pascal

Since compute nodes on the Pascal cluster have CPUs and GPUs, here is how you can use the host-config file to configure a build of the calc_pi project with MPI and CUDA enabled on Pascal:

# create build dir
mkdir build
cd build
# configure using host-config
cmake -C ../../host-configs/llnl/toss_3_x86_64_ib/gcc@4.9.3_nvcc.cmake  ..

After building (make), you can run make test on a batch node (where the GPUs reside) to run the unit tests that are using MPI and CUDA:

bash-4.1$ salloc -A <valid bank>
bash-4.1$ make
bash-4.1$ make test

Running tests...
Test project blt/docs/tutorial/calc_pi/build
    Start 1: test_1
1/8 Test #1: test_1 ...........................   Passed    0.01 sec
    Start 2: test_2
2/8 Test #2: test_2 ...........................   Passed    2.79 sec
    Start 3: test_3
3/8 Test #3: test_3 ...........................   Passed    0.54 sec
    Start 4: blt_gtest_smoke
4/8 Test #4: blt_gtest_smoke ..................   Passed    0.01 sec
    Start 5: blt_fruit_smoke
5/8 Test #5: blt_fruit_smoke ..................   Passed    0.01 sec
    Start 6: blt_mpi_smoke
6/8 Test #6: blt_mpi_smoke ....................   Passed    2.82 sec
    Start 7: blt_cuda_smoke
7/8 Test #7: blt_cuda_smoke ...................   Passed    0.48 sec
    Start 8: blt_cuda_runtime_smoke
8/8 Test #8: blt_cuda_runtime_smoke ...........   Passed    0.11 sec

100% tests passed, 0 tests failed out of 8

Total Test time (real) =   6.80 sec

Building and Testing on Ray

Here is how you can use the host-config file to configure a build of the calc_pi project with MPI and CUDA enabled on the LLNL BlueOS Ray cluster:

# create build dir
mkdir build
cd build
# configure using host-config
cmake -C ../../host-configs/llnl/blueos_3_ppc64le_ib_p9/clang@upstream_nvcc_xlf.cmake  ..

And here is how to build and test the code on Ray:

bash-4.2$ lalloc 1 -G <valid group>
bash-4.2$ make
bash-4.2$ make test

Running tests...
Test project projects/blt/docs/tutorial/calc_pi/build
    Start 1: test_1
1/7 Test #1: test_1 ...........................   Passed    0.01 sec
    Start 2: test_2
2/7 Test #2: test_2 ...........................   Passed    1.24 sec
    Start 3: test_3
3/7 Test #3: test_3 ...........................   Passed    0.17 sec
    Start 4: blt_gtest_smoke
4/7 Test #4: blt_gtest_smoke ..................   Passed    0.01 sec
    Start 5: blt_mpi_smoke
5/7 Test #5: blt_mpi_smoke ....................   Passed    0.82 sec
    Start 6: blt_cuda_smoke
6/7 Test #6: blt_cuda_smoke ...................   Passed    0.15 sec
    Start 7: blt_cuda_runtime_smoke
7/7 Test #7: blt_cuda_runtime_smoke ...........   Passed    0.04 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) =   2.47 sec

Building and Testing on Summit

Here is how you can use the host-config file to configure a build of the calc_pi project with MPI and CUDA enabled on the OLCF Summit cluster:

# load the cmake module
module load cmake
# create build dir
mkdir build
cd build
# configure using host-config
cmake -C ../../host-configs/olcf/summit/gcc@6.4.0_nvcc.cmake  ..

And here is how to build and test the code on Summit:

bash-4.2$ bsub -W 30 -nnodes 1 -P <valid project>  -Is /bin/bash
bash-4.2$ module load gcc cuda
bash-4.2$ make
bash-4.2$ make test

Running tests...
Test project /projects/blt/docs/tutorial/calc_pi/build
      Start  1: test_1
 1/11 Test  #1: test_1 ...........................   Passed    0.00 sec
      Start  2: test_2
 2/11 Test  #2: test_2 ...........................   Passed    1.03 sec
      Start  3: test_3
 3/11 Test  #3: test_3 ...........................   Passed    0.21 sec
      Start  4: blt_gtest_smoke
 4/11 Test  #4: blt_gtest_smoke ..................   Passed    0.00 sec
      Start  5: blt_fruit_smoke
 5/11 Test  #5: blt_fruit_smoke ..................   Passed    0.00 sec
      Start  6: blt_mpi_smoke
 6/11 Test  #6: blt_mpi_smoke ....................   Passed    0.76 sec
      Start  7: blt_cuda_smoke
 7/11 Test  #7: blt_cuda_smoke ...................   Passed    0.22 sec
      Start  8: blt_cuda_runtime_smoke
 8/11 Test  #8: blt_cuda_runtime_smoke ...........   Passed    0.07 sec
      Start  9: blt_cuda_version_smoke
 9/11 Test  #9: blt_cuda_version_smoke ...........   Passed    0.06 sec
      Start 10: blt_cuda_mpi_smoke
10/11 Test #10: blt_cuda_mpi_smoke ...............   Passed    0.80 sec
      Start 11: blt_cuda_gtest_smoke
11/11 Test #11: blt_cuda_gtest_smoke .............   Passed    0.21 sec

100% tests passed, 0 tests failed out of 11

Total Test time (real) =   3.39 sec

Example Host-configs

Basic TOSS3 (for example: Quartz) host-config that has C, C++, and Fortran Compilers along with MPI support:

gcc@8.3.1 host-config

Here are the full example host-config files for LLNL’s Pascal, Ray, and Quartz Clusters that uses the default compilers on the system:

gcc@4.9.3 host-config

More complicated BlueOS host-config that has C, C++, MPI, and CUDA support:

clang@upstream C++17 host-config

Here is a full example host-config file for an OSX laptop, using a set of dependencies built with Spack:

OSX clang@7.3.0 host-config
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Copyright (c) 2017-2023, Lawrence Livermore National Security, LLC and
# other BLT Project Developers. See the top-level LICENSE file for details
# 
# SPDX-License-Identifier: (BSD-3-Clause)

########################################################################
# host-config for naples
########################################################################

########################################################################
# Dependencies were built with spack (https://github.com/llnl/spack)
########################################################################
# spack install  cmake@3.8.2
# spack install  mpich 
# spack install  py-sphinx
# spack activate py-sphinx
# spack install  doxygen


########################################################################
# cmake path
########################################################################
# /Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/cmake-3.8.2-n2i4ijlet37i3jhmjfhzms2wo3b4ybcm/bin/cmake

########################################################################
# mpi from spack
########################################################################
set(ENABLE_MPI ON CACHE PATH "")

set(MPI_BASE_DIR "/Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/mpich-3.2-yc7ipshe7e3w4ohtgjtms2agecxruavw/bin" CACHE PATH "")

set(MPI_C_COMPILER   "${MPI_BASE_DIR}/mpicc" CACHE PATH "")
set(MPI_CXX_COMPILER "${MPI_BASE_DIR}/mpicxx" CACHE PATH "")
set(MPIEXEC          "${MPI_BASE_DIR}/mpiexec" CACHE PATH "")

########################################################################
# Cuda Support (standard osx cuda toolkit install)
########################################################################
set(ENABLE_CUDA ON CACHE BOOL "")

set(CUDA_TOOLKIT_ROOT_DIR "/Developer/NVIDIA/CUDA-8.0/" CACHE PATH "")
set(CUDA_BIN_DIR          "/Developer/NVIDIA/CUDA-8.0/bin/" CACHE PATH "")

########################################################################
# sphinx from spack
########################################################################
set(SPHINX_EXECUTABLE "/Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/python-2.7.13-jmhznopgz2j5zkmuzjygg5oyxnxtc653/bin/sphinx-build" CACHE PATH "")

########################################################################
# doxygen from spack
########################################################################
set(DOXYGEN_EXECUTABLE "/Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/doxygen-1.8.12-mji43fu4hxuu6js5irshpihkwwucn7rv/bin/doxygen" CACHE PATH "")