Host-configs¶
To capture (and revision control) build options, third party library paths, etc., we recommend using CMake’s initial-cache file mechanism. This feature allows you to pass a file to CMake that provides variables to bootstrap the configuration process.
You can pass initial-cache files to cmake via the -C
command line option.
cmake -C config_file.cmake
We call these initial-cache files host-config
files since we typically create
a file for each platform or for specific hosts, if necessary.
These files use standard CMake commands. CMake set()
commands need to specify
CACHE
as follows:
set(CMAKE_VARIABLE_NAME {VALUE} CACHE PATH "")
Here is a snippet from a host-config file that specifies compiler details for using specific gcc (version 4.9.3 in this case) on the LLNL Pascal cluster.
set(GCC_HOME "/usr/tce")
set(CMAKE_C_COMPILER "${GCC_HOME}/bin/gcc" CACHE PATH "")
set(CMAKE_CXX_COMPILER "${GCC_HOME}/bin/g++" CACHE PATH "")
# Fortran support
set(ENABLE_FORTRAN ON CACHE BOOL "")
set(CMAKE_Fortran_COMPILER "${GCC_HOME}/bin/gfortran" CACHE PATH "")
Building and Testing on Pascal¶
Since compute nodes on the Pascal cluster have CPUs and GPUs, here is how you
can use the host-config file to configure a build of the calc_pi
project with
MPI and CUDA enabled on Pascal:
# create build dir
mkdir build
cd build
# configure using host-config
cmake -C ../../host-configs/llnl/toss_3_x86_64_ib/gcc@4.9.3_nvcc.cmake ..
After building (make
), you can run make test
on a batch node (where the GPUs reside)
to run the unit tests that are using MPI and CUDA:
bash-4.1$ salloc -A <valid bank>
bash-4.1$ make
bash-4.1$ make test
Running tests...
Test project blt/docs/tutorial/calc_pi/build
Start 1: test_1
1/8 Test #1: test_1 ........................... Passed 0.01 sec
Start 2: test_2
2/8 Test #2: test_2 ........................... Passed 2.79 sec
Start 3: test_3
3/8 Test #3: test_3 ........................... Passed 0.54 sec
Start 4: blt_gtest_smoke
4/8 Test #4: blt_gtest_smoke .................. Passed 0.01 sec
Start 5: blt_fruit_smoke
5/8 Test #5: blt_fruit_smoke .................. Passed 0.01 sec
Start 6: blt_mpi_smoke
6/8 Test #6: blt_mpi_smoke .................... Passed 2.82 sec
Start 7: blt_cuda_smoke
7/8 Test #7: blt_cuda_smoke ................... Passed 0.48 sec
Start 8: blt_cuda_runtime_smoke
8/8 Test #8: blt_cuda_runtime_smoke ........... Passed 0.11 sec
100% tests passed, 0 tests failed out of 8
Total Test time (real) = 6.80 sec
Building and Testing on Ray¶
Here is how you can use the host-config file to configure a build of the calc_pi
project with MPI and CUDA
enabled on the LLNL BlueOS Ray cluster:
# create build dir
mkdir build
cd build
# configure using host-config
cmake -C ../../host-configs/llnl/blueos_3_ppc64le_ib_p9/clang@upstream_nvcc_xlf.cmake ..
And here is how to build and test the code on Ray:
bash-4.2$ lalloc 1 -G <valid group>
bash-4.2$ make
bash-4.2$ make test
Running tests...
Test project projects/blt/docs/tutorial/calc_pi/build
Start 1: test_1
1/7 Test #1: test_1 ........................... Passed 0.01 sec
Start 2: test_2
2/7 Test #2: test_2 ........................... Passed 1.24 sec
Start 3: test_3
3/7 Test #3: test_3 ........................... Passed 0.17 sec
Start 4: blt_gtest_smoke
4/7 Test #4: blt_gtest_smoke .................. Passed 0.01 sec
Start 5: blt_mpi_smoke
5/7 Test #5: blt_mpi_smoke .................... Passed 0.82 sec
Start 6: blt_cuda_smoke
6/7 Test #6: blt_cuda_smoke ................... Passed 0.15 sec
Start 7: blt_cuda_runtime_smoke
7/7 Test #7: blt_cuda_runtime_smoke ........... Passed 0.04 sec
100% tests passed, 0 tests failed out of 7
Total Test time (real) = 2.47 sec
Building and Testing on Summit¶
Here is how you can use the host-config file to configure a build of the calc_pi
project with MPI and CUDA
enabled on the OLCF Summit cluster:
# load the cmake module
module load cmake
# create build dir
mkdir build
cd build
# configure using host-config
cmake -C ../../host-configs/olcf/summit/gcc@6.4.0_nvcc.cmake ..
And here is how to build and test the code on Summit:
bash-4.2$ bsub -W 30 -nnodes 1 -P <valid project> -Is /bin/bash
bash-4.2$ module load gcc cuda
bash-4.2$ make
bash-4.2$ make test
Running tests...
Test project /projects/blt/docs/tutorial/calc_pi/build
Start 1: test_1
1/11 Test #1: test_1 ........................... Passed 0.00 sec
Start 2: test_2
2/11 Test #2: test_2 ........................... Passed 1.03 sec
Start 3: test_3
3/11 Test #3: test_3 ........................... Passed 0.21 sec
Start 4: blt_gtest_smoke
4/11 Test #4: blt_gtest_smoke .................. Passed 0.00 sec
Start 5: blt_fruit_smoke
5/11 Test #5: blt_fruit_smoke .................. Passed 0.00 sec
Start 6: blt_mpi_smoke
6/11 Test #6: blt_mpi_smoke .................... Passed 0.76 sec
Start 7: blt_cuda_smoke
7/11 Test #7: blt_cuda_smoke ................... Passed 0.22 sec
Start 8: blt_cuda_runtime_smoke
8/11 Test #8: blt_cuda_runtime_smoke ........... Passed 0.07 sec
Start 9: blt_cuda_version_smoke
9/11 Test #9: blt_cuda_version_smoke ........... Passed 0.06 sec
Start 10: blt_cuda_mpi_smoke
10/11 Test #10: blt_cuda_mpi_smoke ............... Passed 0.80 sec
Start 11: blt_cuda_gtest_smoke
11/11 Test #11: blt_cuda_gtest_smoke ............. Passed 0.21 sec
100% tests passed, 0 tests failed out of 11
Total Test time (real) = 3.39 sec
Example Host-configs¶
Basic TOSS3 (for example: Quartz) host-config that has C, C++, and Fortran Compilers along with MPI support:
gcc@8.3.1 host-config
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | # Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and # other BLT Project Developers. See the top-level LICENSE file for details # # SPDX-License-Identifier: (BSD-3-Clause) #------------------------------------------------------------------------------ # Example host-config file for the quartz cluster at LLNL #------------------------------------------------------------------------------ # # This file provides CMake with paths / details for: # C,C++, & Fortran compilers + MPI # #------------------------------------------------------------------------------ #------------------------------------------------------------------------------ # gcc@8.3.1 compilers #------------------------------------------------------------------------------ set(GCC_VERSION "gcc-8.3.1") set(GCC_HOME "/usr/tce/packages/gcc/${GCC_VERSION}") # c compiler set(CMAKE_C_COMPILER "${GCC_HOME}/bin/gcc" CACHE PATH "") # cpp compiler set(CMAKE_CXX_COMPILER "${GCC_HOME}/bin/g++" CACHE PATH "") # fortran support set(ENABLE_FORTRAN ON CACHE BOOL "") # fortran compiler set(CMAKE_Fortran_COMPILER "${GCC_HOME}/bin/gfortran" CACHE PATH "") #------------------------------------------------------------------------------ # MPI Support #------------------------------------------------------------------------------ set(ENABLE_MPI ON CACHE BOOL "") set(MPI_HOME "/usr/tce/packages/mvapich2/mvapich2-2.3-${GCC_VERSION}" CACHE PATH "") set(MPI_C_COMPILER "${MPI_HOME}/bin/mpicc" CACHE PATH "") set(MPI_CXX_COMPILER "${MPI_HOME}/bin/mpicxx" CACHE PATH "") set(MPI_Fortran_COMPILER "${MPI_HOME}/bin/mpif90" CACHE PATH "") set(MPIEXEC "/usr/bin/srun" CACHE PATH "") set(MPIEXEC_NUMPROC_FLAG "-n" CACHE PATH "") |
Here are the full example host-config files for LLNL’s Pascal, Ray, and Quartz Clusters that uses the default compilers on the system:
gcc@4.9.3 host-config
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | # Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and # other BLT Project Developers. See the top-level LICENSE file for details # # SPDX-License-Identifier: (BSD-3-Clause) #------------------------------------------------------------------------------ # Example host-config file for the Pascal cluster at LLNL #------------------------------------------------------------------------------ # This file provides CMake with paths / details for: # C,C++, & Fortran compilers + MPI & CUDA #------------------------------------------------------------------------------ #------------------------------------------------------------------------------ # gcc@4.9.3 compilers #------------------------------------------------------------------------------ # _blt_tutorial_compiler_config_start set(GCC_HOME "/usr/tce") set(CMAKE_C_COMPILER "${GCC_HOME}/bin/gcc" CACHE PATH "") set(CMAKE_CXX_COMPILER "${GCC_HOME}/bin/g++" CACHE PATH "") # Fortran support set(ENABLE_FORTRAN ON CACHE BOOL "") set(CMAKE_Fortran_COMPILER "${GCC_HOME}/bin/gfortran" CACHE PATH "") # _blt_tutorial_compiler_config_end #------------------------------------------------------------------------------ # MPI Support #------------------------------------------------------------------------------ # _blt_tutorial_mpi_config_start set(ENABLE_MPI ON CACHE BOOL "") set(MPI_HOME "/usr/tce/packages/mvapich2/mvapich2-2.3-gcc-4.9.3/") set(MPI_C_COMPILER "${MPI_HOME}/bin/mpicc" CACHE PATH "") set(MPI_CXX_COMPILER "${MPI_HOME}/bin/mpicxx" CACHE PATH "") set(MPI_Fortran_COMPILER "${MPI_HOME}/bin/mpif90" CACHE PATH "") # _blt_tutorial_mpi_config_end #------------------------------------------------------------------------------ # CUDA support #------------------------------------------------------------------------------ # _blt_tutorial_cuda_config_start set(ENABLE_CUDA ON CACHE BOOL "") set(CUDA_TOOLKIT_ROOT_DIR "/usr/tce/packages/cuda/cuda-10.1.168" CACHE PATH "") set(CMAKE_CUDA_COMPILER "${CUDA_TOOLKIT_ROOT_DIR}/bin/nvcc" CACHE PATH "") set(CMAKE_CUDA_HOST_COMPILER "${CMAKE_CXX_COMPILER}" CACHE PATH "") set(CMAKE_CUDA_ARCHITECTURES "70" CACHE STRING "") set(_cuda_arch "sm_${CMAKE_CUDA_ARCHITECTURES}") set(CMAKE_CUDA_FLAGS "-restrict -arch ${_cuda_arch} -std=c++11 --expt-extended-lambda -G" CACHE STRING "") set(CUDA_SEPARABLE_COMPILATION ON CACHE BOOL "") # _blt_tutorial_cuda_config_end |
More complicated BlueOS host-config that has C, C++, MPI, and CUDA support:
clang@upstream C++17 host-config
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | # Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and # other BLT Project Developers. See the top-level LICENSE file for details # # SPDX-License-Identifier: (BSD-3-Clause) #------------------------------------------------------------------------------ # Example host-config file for the blue_os cluster at LLNL #------------------------------------------------------------------------------ # # This file provides CMake with paths / details for: # C/C++: Clang with GCC 8.3.1 toolchain with C++17 support # Cuda # MPI # #------------------------------------------------------------------------------ #--------------------------------------- # Compilers #--------------------------------------- set(_CLANG_VERSION "clang-upstream-2019.08.15") set(_CLANG_DIR "/usr/tce/packages/clang/${_CLANG_VERSION}") set(_GCC_DIR "/usr/tce/packages/gcc/gcc-8.3.1") set(CMAKE_C_COMPILER "${_CLANG_DIR}/bin/clang" CACHE PATH "") set(CMAKE_CXX_COMPILER "${_CLANG_DIR}/bin/clang++" CACHE PATH "") set(BLT_CXX_STD "c++17" CACHE STRING "") set(CMAKE_C_FLAGS "--gcc-toolchain=${_GCC_DIR}" CACHE PATH "") set(CMAKE_CXX_FLAGS "--gcc-toolchain=${_GCC_DIR}" CACHE PATH "") set(BLT_EXE_LINKER_FLAGS " -Wl,-rpath,${_GCC_DIR}/lib" CACHE PATH "Adds a missing libstdc++ rpath") #--------------------------------------- # MPI #--------------------------------------- set(ENABLE_MPI ON CACHE BOOL "") set(_MPI_BASE_DIR "/usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-${_CLANG_VERSION}") set(MPI_C_COMPILER "${_MPI_BASE_DIR}/bin/mpicc" CACHE PATH "") set(MPI_CXX_COMPILER "${_MPI_BASE_DIR}/bin/mpicxx" CACHE PATH "") #------------------------------------------------------------------------------ # Cuda #------------------------------------------------------------------------------ set(ENABLE_CUDA ON CACHE BOOL "") set(CUDA_TOOLKIT_ROOT_DIR "/usr/tce/packages/cuda/cuda-11.1.1" CACHE PATH "") set(CMAKE_CUDA_COMPILER "${CUDA_TOOLKIT_ROOT_DIR}/bin/nvcc" CACHE PATH "") set(CMAKE_CUDA_HOST_COMPILER "${CMAKE_CXX_COMPILER}" CACHE PATH "") set(CMAKE_CUDA_ARCHITECTURES "70" CACHE STRING "") set(_cuda_arch "sm_${CMAKE_CUDA_ARCHITECTURES}") set(CMAKE_CUDA_FLAGS "-Xcompiler=--gcc-toolchain=${_GCC_DIR} -restrict -arch ${_cuda_arch} --expt-extended-lambda -G" CACHE STRING "") # nvcc does not like gtest's 'pthreads' flag set(gtest_disable_pthreads ON CACHE BOOL "") set(ENABLE_GTEST_DEATH_TESTS OFF CACHE BOOL "") # Very specific fix for working around CMake adding implicit link directories returned by the BlueOS # compilers to link CUDA executables set(BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE "/usr/tce/packages/gcc/gcc-4.9.3/lib64/gcc/powerpc64le-unknown-linux-gnu/4.9.3;/usr/tce/packages/gcc/gcc-4.9.3/lib64" CACHE STRING "") |
Here is a full example host-config file for an OSX laptop, using a set of dependencies built with Spack:
OSX clang@7.3.0 host-config
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | # Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and # other BLT Project Developers. See the top-level LICENSE file for details # # SPDX-License-Identifier: (BSD-3-Clause) ######################################################################## # host-config for naples ######################################################################## ######################################################################## # Dependencies were built with spack (https://github.com/llnl/spack) ######################################################################## # spack install cmake@3.8.2 # spack install mpich # spack install py-sphinx # spack activate py-sphinx # spack install doxygen ######################################################################## # cmake path ######################################################################## # /Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/cmake-3.8.2-n2i4ijlet37i3jhmjfhzms2wo3b4ybcm/bin/cmake ######################################################################## # mpi from spack ######################################################################## set(ENABLE_MPI ON CACHE PATH "") set(MPI_BASE_DIR "/Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/mpich-3.2-yc7ipshe7e3w4ohtgjtms2agecxruavw/bin" CACHE PATH "") set(MPI_C_COMPILER "${MPI_BASE_DIR}/mpicc" CACHE PATH "") set(MPI_CXX_COMPILER "${MPI_BASE_DIR}/mpicxx" CACHE PATH "") set(MPIEXEC "${MPI_BASE_DIR}/mpiexec" CACHE PATH "") ######################################################################## # Cuda Support (standard osx cuda toolkit install) ######################################################################## set(ENABLE_CUDA ON CACHE BOOL "") set(CUDA_TOOLKIT_ROOT_DIR "/Developer/NVIDIA/CUDA-8.0/" CACHE PATH "") set(CUDA_BIN_DIR "/Developer/NVIDIA/CUDA-8.0/bin/" CACHE PATH "") ######################################################################## # sphinx from spack ######################################################################## set(SPHINX_EXECUTABLE "/Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/python-2.7.13-jmhznopgz2j5zkmuzjygg5oyxnxtc653/bin/sphinx-build" CACHE PATH "") ######################################################################## # doxygen from spack ######################################################################## set(DOXYGEN_EXECUTABLE "/Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/doxygen-1.8.12-mji43fu4hxuu6js5irshpihkwwucn7rv/bin/doxygen" CACHE PATH "") |