Host-configs¶
To capture (and revision control) build options, third party library paths, etc., we recommend using CMake’s initial-cache file mechanism. This feature allows you to pass a file to CMake that provides variables to bootstrap the configuration process.
You can pass initial-cache files to cmake via the -C
command line option.
cmake -C config_file.cmake
We call these initial-cache files host-config
files since we typically create
a file for each platform or for specific hosts, if necessary.
These files use standard CMake commands. CMake set()
commands need to specify
CACHE
as follows:
set(CMAKE_VARIABLE_NAME {VALUE} CACHE PATH "")
Here is a snippet from a host-config file that specifies compiler details for using specific gcc (version 4.9.3 in this case) on the LLNL Pascal cluster.
set(GCC_HOME "/usr/tce")
set(CMAKE_C_COMPILER "${GCC_HOME}/bin/gcc" CACHE PATH "")
set(CMAKE_CXX_COMPILER "${GCC_HOME}/bin/g++" CACHE PATH "")
# Fortran support
set(ENABLE_FORTRAN ON CACHE BOOL "")
set(CMAKE_Fortran_COMPILER "${GCC_HOME}/bin/gfortran" CACHE PATH "")
Building and Testing on Pascal¶
Since compute nodes on the Pascal cluster have CPUs and GPUs, here is how you
can use the host-config file to configure a build of the calc_pi
project with
MPI and CUDA enabled on Pascal:
# create build dir
mkdir build
cd build
# configure using host-config
cmake -C ../../host-configs/llnl/toss_3_x86_64_ib/gcc@4.9.3_nvcc.cmake ..
After building (make
), you can run make test
on a batch node (where the GPUs reside)
to run the unit tests that are using MPI and CUDA:
bash-4.1$ salloc -A <valid bank>
bash-4.1$ make
bash-4.1$ make test
Running tests...
Test project blt/docs/tutorial/calc_pi/build
Start 1: test_1
1/8 Test #1: test_1 ........................... Passed 0.01 sec
Start 2: test_2
2/8 Test #2: test_2 ........................... Passed 2.79 sec
Start 3: test_3
3/8 Test #3: test_3 ........................... Passed 0.54 sec
Start 4: blt_gtest_smoke
4/8 Test #4: blt_gtest_smoke .................. Passed 0.01 sec
Start 5: blt_fruit_smoke
5/8 Test #5: blt_fruit_smoke .................. Passed 0.01 sec
Start 6: blt_mpi_smoke
6/8 Test #6: blt_mpi_smoke .................... Passed 2.82 sec
Start 7: blt_cuda_smoke
7/8 Test #7: blt_cuda_smoke ................... Passed 0.48 sec
Start 8: blt_cuda_runtime_smoke
8/8 Test #8: blt_cuda_runtime_smoke ........... Passed 0.11 sec
100% tests passed, 0 tests failed out of 8
Total Test time (real) = 6.80 sec
Building and Testing on Ray¶
Here is how you can use the host-config file to configure a build of the calc_pi
project with MPI and CUDA
enabled on the LLNL BlueOS Ray cluster:
# create build dir
mkdir build
cd build
# configure using host-config
cmake -C ../../host-configs/llnl/blueos_3_ppc64le_ib_p9/clang@upstream_nvcc_xlf.cmake ..
And here is how to build and test the code on Ray:
bash-4.2$ lalloc 1 -G <valid group>
bash-4.2$ make
bash-4.2$ make test
Running tests...
Test project projects/blt/docs/tutorial/calc_pi/build
Start 1: test_1
1/7 Test #1: test_1 ........................... Passed 0.01 sec
Start 2: test_2
2/7 Test #2: test_2 ........................... Passed 1.24 sec
Start 3: test_3
3/7 Test #3: test_3 ........................... Passed 0.17 sec
Start 4: blt_gtest_smoke
4/7 Test #4: blt_gtest_smoke .................. Passed 0.01 sec
Start 5: blt_mpi_smoke
5/7 Test #5: blt_mpi_smoke .................... Passed 0.82 sec
Start 6: blt_cuda_smoke
6/7 Test #6: blt_cuda_smoke ................... Passed 0.15 sec
Start 7: blt_cuda_runtime_smoke
7/7 Test #7: blt_cuda_runtime_smoke ........... Passed 0.04 sec
100% tests passed, 0 tests failed out of 7
Total Test time (real) = 2.47 sec
Building and Testing on Summit¶
Here is how you can use the host-config file to configure a build of the calc_pi
project with MPI and CUDA
enabled on the OLCF Summit cluster:
# load the cmake module
module load cmake
# create build dir
mkdir build
cd build
# configure using host-config
cmake -C ../../host-configs/olcf/summit/gcc@6.4.0_nvcc.cmake ..
And here is how to build and test the code on Summit:
bash-4.2$ bsub -W 30 -nnodes 1 -P <valid project> -Is /bin/bash
bash-4.2$ module load gcc cuda
bash-4.2$ make
bash-4.2$ make test
Running tests...
Test project /projects/blt/docs/tutorial/calc_pi/build
Start 1: test_1
1/11 Test #1: test_1 ........................... Passed 0.00 sec
Start 2: test_2
2/11 Test #2: test_2 ........................... Passed 1.03 sec
Start 3: test_3
3/11 Test #3: test_3 ........................... Passed 0.21 sec
Start 4: blt_gtest_smoke
4/11 Test #4: blt_gtest_smoke .................. Passed 0.00 sec
Start 5: blt_fruit_smoke
5/11 Test #5: blt_fruit_smoke .................. Passed 0.00 sec
Start 6: blt_mpi_smoke
6/11 Test #6: blt_mpi_smoke .................... Passed 0.76 sec
Start 7: blt_cuda_smoke
7/11 Test #7: blt_cuda_smoke ................... Passed 0.22 sec
Start 8: blt_cuda_runtime_smoke
8/11 Test #8: blt_cuda_runtime_smoke ........... Passed 0.07 sec
Start 9: blt_cuda_version_smoke
9/11 Test #9: blt_cuda_version_smoke ........... Passed 0.06 sec
Start 10: blt_cuda_mpi_smoke
10/11 Test #10: blt_cuda_mpi_smoke ............... Passed 0.80 sec
Start 11: blt_cuda_gtest_smoke
11/11 Test #11: blt_cuda_gtest_smoke ............. Passed 0.21 sec
100% tests passed, 0 tests failed out of 11
Total Test time (real) = 3.39 sec
Example Host-configs¶
Basic TOSS3 (for example: Quartz) host-config that has C, C++, and Fortran Compilers along with MPI support:
gcc@8.3.1 host-config
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | # Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and
# other BLT Project Developers. See the top-level LICENSE file for details
#
# SPDX-License-Identifier: (BSD-3-Clause)
#------------------------------------------------------------------------------
# Example host-config file for the quartz cluster at LLNL
#------------------------------------------------------------------------------
#
# This file provides CMake with paths / details for:
# C,C++, & Fortran compilers + MPI
#
#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
# gcc@8.3.1 compilers
#------------------------------------------------------------------------------
set(GCC_VERSION "gcc-8.3.1")
set(GCC_HOME "/usr/tce/packages/gcc/${GCC_VERSION}")
# c compiler
set(CMAKE_C_COMPILER "${GCC_HOME}/bin/gcc" CACHE PATH "")
# cpp compiler
set(CMAKE_CXX_COMPILER "${GCC_HOME}/bin/g++" CACHE PATH "")
# fortran support
set(ENABLE_FORTRAN ON CACHE BOOL "")
# fortran compiler
set(CMAKE_Fortran_COMPILER "${GCC_HOME}/bin/gfortran" CACHE PATH "")
#------------------------------------------------------------------------------
# MPI Support
#------------------------------------------------------------------------------
set(ENABLE_MPI ON CACHE BOOL "")
set(MPI_HOME "/usr/tce/packages/mvapich2/mvapich2-2.3-${GCC_VERSION}" CACHE PATH "")
set(MPI_C_COMPILER "${MPI_HOME}/bin/mpicc" CACHE PATH "")
set(MPI_CXX_COMPILER "${MPI_HOME}/bin/mpicxx" CACHE PATH "")
set(MPI_Fortran_COMPILER "${MPI_HOME}/bin/mpif90" CACHE PATH "")
set(MPIEXEC "/usr/bin/srun" CACHE PATH "")
set(MPIEXEC_NUMPROC_FLAG "-n" CACHE PATH "")
|
Here are the full example host-config files for LLNL’s Pascal, Ray, and Quartz Clusters that uses the default compilers on the system:
gcc@4.9.3 host-config
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | # Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and
# other BLT Project Developers. See the top-level LICENSE file for details
#
# SPDX-License-Identifier: (BSD-3-Clause)
#------------------------------------------------------------------------------
# Example host-config file for the Pascal cluster at LLNL
#------------------------------------------------------------------------------
# This file provides CMake with paths / details for:
# C,C++, & Fortran compilers + MPI & CUDA
#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
# gcc@4.9.3 compilers
#------------------------------------------------------------------------------
# _blt_tutorial_compiler_config_start
set(GCC_HOME "/usr/tce")
set(CMAKE_C_COMPILER "${GCC_HOME}/bin/gcc" CACHE PATH "")
set(CMAKE_CXX_COMPILER "${GCC_HOME}/bin/g++" CACHE PATH "")
# Fortran support
set(ENABLE_FORTRAN ON CACHE BOOL "")
set(CMAKE_Fortran_COMPILER "${GCC_HOME}/bin/gfortran" CACHE PATH "")
# _blt_tutorial_compiler_config_end
#------------------------------------------------------------------------------
# MPI Support
#------------------------------------------------------------------------------
# _blt_tutorial_mpi_config_start
set(ENABLE_MPI ON CACHE BOOL "")
set(MPI_HOME "/usr/tce/packages/mvapich2/mvapich2-2.3-gcc-4.9.3/")
set(MPI_C_COMPILER "${MPI_HOME}/bin/mpicc" CACHE PATH "")
set(MPI_CXX_COMPILER "${MPI_HOME}/bin/mpicxx" CACHE PATH "")
set(MPI_Fortran_COMPILER "${MPI_HOME}/bin/mpif90" CACHE PATH "")
# _blt_tutorial_mpi_config_end
#------------------------------------------------------------------------------
# CUDA support
#------------------------------------------------------------------------------
# _blt_tutorial_cuda_config_start
set(ENABLE_CUDA ON CACHE BOOL "")
set(CUDA_TOOLKIT_ROOT_DIR "/usr/tce/packages/cuda/cuda-10.1.168" CACHE PATH "")
set(CMAKE_CUDA_COMPILER "${CUDA_TOOLKIT_ROOT_DIR}/bin/nvcc" CACHE PATH "")
set(CMAKE_CUDA_HOST_COMPILER "${CMAKE_CXX_COMPILER}" CACHE PATH "")
set(CMAKE_CUDA_ARCHITECTURES "70" CACHE STRING "")
set(_cuda_arch "sm_${CMAKE_CUDA_ARCHITECTURES}")
set(CMAKE_CUDA_FLAGS "-restrict -arch ${_cuda_arch} -std=c++11 --expt-extended-lambda -G"
CACHE STRING "")
set(CUDA_SEPARABLE_COMPILATION ON CACHE BOOL "")
# _blt_tutorial_cuda_config_end
|
More complicated BlueOS host-config that has C, C++, MPI, and CUDA support:
clang@upstream C++17 host-config
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | # Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and
# other BLT Project Developers. See the top-level LICENSE file for details
#
# SPDX-License-Identifier: (BSD-3-Clause)
#------------------------------------------------------------------------------
# Example host-config file for the blue_os cluster at LLNL
#------------------------------------------------------------------------------
#
# This file provides CMake with paths / details for:
# C/C++: Clang with GCC 8.3.1 toolchain with C++17 support
# Cuda
# MPI
#
#------------------------------------------------------------------------------
#---------------------------------------
# Compilers
#---------------------------------------
set(_CLANG_VERSION "clang-upstream-2019.08.15")
set(_CLANG_DIR "/usr/tce/packages/clang/${_CLANG_VERSION}")
set(_GCC_DIR "/usr/tce/packages/gcc/gcc-8.3.1")
set(CMAKE_C_COMPILER "${_CLANG_DIR}/bin/clang" CACHE PATH "")
set(CMAKE_CXX_COMPILER "${_CLANG_DIR}/bin/clang++" CACHE PATH "")
set(BLT_CXX_STD "c++17" CACHE STRING "")
set(CMAKE_C_FLAGS "--gcc-toolchain=${_GCC_DIR}" CACHE PATH "")
set(CMAKE_CXX_FLAGS "--gcc-toolchain=${_GCC_DIR}" CACHE PATH "")
set(BLT_EXE_LINKER_FLAGS " -Wl,-rpath,${_GCC_DIR}/lib" CACHE PATH "Adds a missing libstdc++ rpath")
#---------------------------------------
# MPI
#---------------------------------------
set(ENABLE_MPI ON CACHE BOOL "")
set(_MPI_BASE_DIR "/usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-${_CLANG_VERSION}")
set(MPI_C_COMPILER "${_MPI_BASE_DIR}/bin/mpicc" CACHE PATH "")
set(MPI_CXX_COMPILER "${_MPI_BASE_DIR}/bin/mpicxx" CACHE PATH "")
#------------------------------------------------------------------------------
# Cuda
#------------------------------------------------------------------------------
set(ENABLE_CUDA ON CACHE BOOL "")
set(CUDA_TOOLKIT_ROOT_DIR "/usr/tce/packages/cuda/cuda-11.1.1" CACHE PATH "")
set(CMAKE_CUDA_COMPILER "${CUDA_TOOLKIT_ROOT_DIR}/bin/nvcc" CACHE PATH "")
set(CMAKE_CUDA_HOST_COMPILER "${CMAKE_CXX_COMPILER}" CACHE PATH "")
set(CMAKE_CUDA_ARCHITECTURES "70" CACHE STRING "")
set(_cuda_arch "sm_${CMAKE_CUDA_ARCHITECTURES}")
set(CMAKE_CUDA_FLAGS "-Xcompiler=--gcc-toolchain=${_GCC_DIR} -restrict -arch ${_cuda_arch} --expt-extended-lambda -G" CACHE STRING "")
# nvcc does not like gtest's 'pthreads' flag
set(gtest_disable_pthreads ON CACHE BOOL "")
set(ENABLE_GTEST_DEATH_TESTS OFF CACHE BOOL "")
# Very specific fix for working around CMake adding implicit link directories returned by the BlueOS
# compilers to link CUDA executables
set(BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE "/usr/tce/packages/gcc/gcc-4.9.3/lib64/gcc/powerpc64le-unknown-linux-gnu/4.9.3;/usr/tce/packages/gcc/gcc-4.9.3/lib64" CACHE STRING "")
|
Here is a full example host-config file for an OSX laptop, using a set of dependencies built with Spack:
OSX clang@7.3.0 host-config
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | # Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and
# other BLT Project Developers. See the top-level LICENSE file for details
#
# SPDX-License-Identifier: (BSD-3-Clause)
########################################################################
# host-config for naples
########################################################################
########################################################################
# Dependencies were built with spack (https://github.com/llnl/spack)
########################################################################
# spack install cmake@3.8.2
# spack install mpich
# spack install py-sphinx
# spack activate py-sphinx
# spack install doxygen
########################################################################
# cmake path
########################################################################
# /Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/cmake-3.8.2-n2i4ijlet37i3jhmjfhzms2wo3b4ybcm/bin/cmake
########################################################################
# mpi from spack
########################################################################
set(ENABLE_MPI ON CACHE PATH "")
set(MPI_BASE_DIR "/Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/mpich-3.2-yc7ipshe7e3w4ohtgjtms2agecxruavw/bin" CACHE PATH "")
set(MPI_C_COMPILER "${MPI_BASE_DIR}/mpicc" CACHE PATH "")
set(MPI_CXX_COMPILER "${MPI_BASE_DIR}/mpicxx" CACHE PATH "")
set(MPIEXEC "${MPI_BASE_DIR}/mpiexec" CACHE PATH "")
########################################################################
# Cuda Support (standard osx cuda toolkit install)
########################################################################
set(ENABLE_CUDA ON CACHE BOOL "")
set(CUDA_TOOLKIT_ROOT_DIR "/Developer/NVIDIA/CUDA-8.0/" CACHE PATH "")
set(CUDA_BIN_DIR "/Developer/NVIDIA/CUDA-8.0/bin/" CACHE PATH "")
########################################################################
# sphinx from spack
########################################################################
set(SPHINX_EXECUTABLE "/Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/python-2.7.13-jmhznopgz2j5zkmuzjygg5oyxnxtc653/bin/sphinx-build" CACHE PATH "")
########################################################################
# doxygen from spack
########################################################################
set(DOXYGEN_EXECUTABLE "/Users/harrison37/Work/blt_tutorial/tpls/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/doxygen-1.8.12-mji43fu4hxuu6js5irshpihkwwucn7rv/bin/doxygen" CACHE PATH "")
|