wiki:CMakeFAQ

TOC(PrerequisitesForCompiling,PlatformSpecificSetup,CMakeFAQ)?

DAKOTA CMake Hints

We currently require CMake version 2.8.9 or higher  CMake Downloads. Most developers will need to build from source on Linux. Do so using either an installed CMake or the traditional configure bootstrap process that comes with CMake sources. See the quick reference pages on the  CMake Documentation Site. For example, the following works on stock RHEL6 with no CMake installed:

# If you want to build the Curses-based CMake GUI ccmake, consider 
# installing packages: ncurses, ncurses-devel, ncurses-libs 
[export http_proxy=wwwproxy.domain.com:80]
wget http://www.cmake.org/files/v2.8/cmake-2.8.9.tar.gz
tar xzf cmake-2.8.9.tar.gz 
cd cmake-2.8.9
./bootstrap --prefix=$HOME/local --parallel=4
make -j 4
make install
export PATH=$HOME/local/bin:$PATH

For general CMake guidance, see

Managing Your Build Configuration

There are several ways to set and manage the CMake configuration settings within your build trees. They can be used in combination as well.

  1. Write any settings (CMake definitions using set) in a file such as the example file dakota/config/BuildSetup.cmake. You can invoke CMake to use these settings with
    cmake -C BuildSetup.cmake path/to/source
    
    So for example you could create GeorgesDebug.cmake and GeorgesOptimized.cmake corresponding to two build trees with various options. NOTE, in particular, one key variable that MUST be set if you're on RHEL6 with system-provided Boost libs and using CMake 2.8.6 (or newer) is:
     -DBoost_NO_BOOST_CMAKE=ON
    
  1. You can set options when invoking CMake at the command line, e.g.,
    cmake -DOPTION=VALUE path/to/source 
    # or alternately,
    cmake -DOPTION:TYPE=VALUE
    
  2. After running CMake once in a build tree, when in DAKOTA_BUILD you can run ccmake DAKOTA_SOURCE to see and modify the currently defined cache variables. Pressing t toggles advanced mode to show more variables.

Build and Debug Options

Optimization and Debugging Symbols

The easiest way to manage the optimization and debug symbol inclusion for your builds is with the CMake build type. For example set

CMAKE_BUILD_TYPE=DEBUG              # defaults to -g
CMAKE_BUILD_TYPE=RELEASE            # defaults to -O3 -DNDEBUG
CMAKE_BUILD_TYPE=RELWITHDEBINFO     # defaults to -O2 -g
CMAKE_BUILD_TYPE=DISTRIBUTION       # custom setting -O2; DAKOTA optimized builds

You may also manually set the relevant flags at configure time and these will supplement any set by the CMAKE_BUILD_TYPE configurations shown immediately above.

CMAKE_C_FLAGS=-O2
CMAKE_CXX_FLAGS=-O2
CMAKE_Fortran_FLAGS=-O2

Originally this was reported to work as well:

export CFLAGS=-O2
export CXXFLAGS=-O2
export FFLAGS=-O2
export FCFLAGS=-O2

then proceed as above.

Developer Build Macros

These shortcuts are designed to help routine development:

  • -C config/DakotaDev.cmake: enable MPI, docs, and specification maintenance
  • DevDebug:BOOL=ON to turn on most common developer options, with debug build type, static libs, and bounds checking (equivalent to cmake -C config/DakotaDebug.cmake -C config/DakotaDev.cmake)
  • DevDistro:BOOL=ON to turn on most common developer options and distribution build type, shared libs, no bounds checking (equivalent to cmake -C config/DakotaDistro.cmake -C config/DakotaDev.cmake)

Take care to add additional options or configuration files in the right place before or after these to get the right behavior.

Verbosity of make

To see compile/link commands issued:

make -jN VERBOSE=1

Enabling Optional Features

MPI

For Dakota newer than 6.1:

The minimum for compiling Dakota with MPI is to make sure appropriate MPI binaries and libraries are on necessary PATHs and setting DAKOTA_HAVE_MPI:BOOL=TRUE

The recommended practice is to also specify a C++ compiler wrapper:

DAKOTA_HAVE_MPI:BOOL=TRUE
MPI_CXX_COMPILER:FILEPATH="/path/to/mpicxx"

By default, the MPI compiler wrapper will be used by CMake to find MPI, but not actually used for compilation. Rather the detected defines, includes, and libraries will be added to relevant MPI-enabled targets in the build.

If a compiler wrapper isn't available, or specific MPI includes and libraries are needed, specify appropriate values for MPI_CXX_LIBRARIES and MPI_CXX_INCLUDE_PATH. This will circumvent any autodetection. If you want to force Dakota to use the MPI wrappers and NOT apply the detected libraries and headers, set

DAKOTA_HAVE_MPI:BOOL=TRUE
CMAKE_CXX_COMPILER:FILEPATH="/path/to/mpicxx"
MPI_CXX_COMPILER:FILEPATH="/path/to/mpicxx"
DAKOTA_MPI_WRAPPER_ONLY:BOOL=TRUE

Dakota uses the native CMake MPI probe to find and configure MPI. For more information, see:  http://www.cmake.org/cmake/help/v2.8.9/cmake.html#module:FindMPI

For Dakota 6.1 and older, historical guidance on MPI settings: The recommended practice for building with MPI is to set options similar to

DAKOTA_HAVE_MPI:BOOL=TRUE 
CMAKE_CXX_COMPILER:FILEPATH="/path/to/mpicxx" 

however, several alternatives to setting the CXX compiler are possible, including:

  • set MPI_CXX_COMPILER in CMake cache to a specific MPI C++ wrapper (Trilinos)
  • set PATH in environment to include MPI C++ wrapper
  • set MPI_LIBRARY and MPI_INCLUDE_PATH in CMake cache
  • set MPI_BASE_DIR in CMake cache (Trilinos)
  • set MPIHOME in environment

Documentation

Documentation build requires appropriate versions of Perl, Doxygen, and LaTeX and only works on Unix: ENABLE_DAKOTA_DOCS:BOOL=TRUE

Graphics

DAKOTA X11/Motif-based graphics default to ON, but are turned off if necessary libraries aren't found. This can be force disabled with the setting HAVE_X_GRAPHICS:BOOL=FALSE

Specification Maintenance

DAKOTA specification maintenance mode generates nidrgen and Dt binaries in packages/nidr and when dakota.input.nspec changes, will use them to update relevant generated files in DAKOTA_SOURCE/src since currently they need to be checked in. To enable specification maintenance, use the option ENABLE_SPEC_MAINT:BOOL=TRUE

Static Libraries

DAKOTA defaults to shared object builds, but you can override if you need to link against static libraries without position-independent code, e.g,. an installed libblas.a or libmpi.a. Building with static libraries will drastically increase the link time for executables.

# Build shared libraries ONLY
set(BUILD_STATIC_LIBS OFF CACHE BOOL "Set to ON to build static libraries" FORCE)
set(BUILD_SHARED_LIBS ON CACHE BOOL "Set to ON to build DSO libraries" FORCE)
# Build static libraries ONLY
set(BUILD_STATIC_LIBS ON CACHE BOOL "Set to ON to build static libraries" FORCE)
set(BUILD_SHARED_LIBS OFF CACHE BOOL "Set to ON to build DSO libraries" FORCE)

QUESO for Bayesian Calibration

Using QUESO in Dakota additionally requires

  • Enabling MPI with DAKOTA_HAVE_MPI and other related MPI settings
  • Boost 1.49 or newer, including the program_options compiled library
  • GSL 1.15 or newer, which means that resulting binaries will include GPL licensed components

Make the following CMake settings:

HAVE_QUESO:BOOL=ON
DAKOTA_HAVE_GSL:BOOL=ON
# Optionally, to work around a Dakota 6.1 bug:
HAVE_QUESO_GPMSA:BOOL=OFF

If the program "gsl-config" is not on your PATH, you'll either need to put it there, or set the environment variable export GSL_CONFIG=/usr/local/gsl/1.15/bin/gsl-config or similar. You may also need to set export BOOST_ROOT=/usr/local/boost/1.49.0 or similar to make sure that QUESO and Dakota are built with the same Boost.

Finding Third-Party Libraries (TPLs)

Boost

Building DAKOTA with CMake requires a sufficiently new Boost, 1.49 as of this writing. (DAKOTA will no longer build with a provided in-source Boost distribution.) It requires binary library components such as, but not necessarily limited to, signals, regex, and filesystem. As of CMake-2.8.6 a new cache variable must be defined in order to use system installed boost library components on RHEL6 -- please add the following to properly initialize your CMake Cache:

 -DBoost_NO_BOOST_CMAKE=ON

You can specify your own Boost by setting, e.g.,

-DBoost_INCLUDE_DIR=/path/to/boost/1.49.0/include

 Boost downloads

Note that with recent versions of CMAKE, e.g. CMAKE 2.8.8, it is best to compile with the following: first export a global variable BOOST_ROOT (e.g. export BOOST_ROOT=/path/to/boost). Then, in the configuration, set:

-DBoost_NO_SYSTEM_PATHS=ON 

In this case, especially if you have multiple versions of Boost and you do NOT want to point to the system Boost which may be an older version, use the Boost_NO_SYSTEM_PATHS option and do not explicitly set the Boost_INCLUDE_DIR or Boost_LIBRARY_DIR.

Linear Algebra (BLAS/LAPACK)

  • You can use definitions such as -DBLAS_LIBS=path/to/libblas.* -DLAPACK_LIBS=path/to/liblapack.* or set in a CMake config file with something like
    set(BLAS_LIBS "$ENV{BLAS_LIBS}" CACHE STRING "BLAS libs" FORCE)
    set(LAPACK_LIBS "$ENV{LAPACK_LIBS}" CACHE STRING "LAPACK libs" FORCE)
    
  • Known issues with systems providing, e.g., libblas.so.3. For now a workaround is to symlink libblas.so -> libblas.so.3 and similar for the lapack library.
  • Not all DAKOTA's TPLs treat BLAS and LAPACK with the same cache variables. To use specific versions of these libraries, the following is likely necessary to handle HOPSPACK LAPACK dependency on BLAS:
    set(LAPACK_ADD_LIBS ${BLAS_LIBS}}
    

Development Process

Routine development typically looks like

  • Modify files in DAKOTA_SOURCE, potentially adding new sources or libraries to CMakeLists.txt
  • make in DAKOTA_BUILD (if configuration-related CMakeLists.txt files have changed, make will automatically re-run cmake, so you shouldn't need to do so manually)
  • If you need to modify build settings such as optimization, compilers, or features, you will need to re-run cmake or ccmake (in DAKOTA_BUILD) to change the settings, then make.

Debugging Process

Debugging with GDB using the default CMake build settings can be challenging due to shared libraries. At least for binaries built with -g, gdb is able to correctly resolve the location of source files, even when out-of-source. However, with shared library builds, even the files in DAKOTA_SOURCE/src are compiled into a libdakota_src.so convenience library which is linked into main. Until the executable is run, these symbols aren't available to the debugger and even when they are they get lost among many available symbols when using autocomplete.

Here are two suggested workarounds/workflows:

  1. Current recommended if viable: Configure your build to use static libraries (see options above); TO DO: embed this as part of CMAKE_BUILD_TYPE=DEBUG macro.
  2. Use shared object libraries, which aren't loaded until you run the binary in gdb, with some care and patience, since searching for symbols is costly:
    1. Start gdb, e.g., gdb ./dakota in DAKOTA_BUILD/src
    2. Set a breakpoint in main, e.g., break main.C:59
    3. Haven't gotten this to work: optionally specify a regular expression filter on which shared objects to load, which can drastically improve search time for symbols, using the command sharedlibrary regex. If you don't do this, using tab completion on a file name can take a long time. For example sharedlibrary pecos
    4. Run the binary to load symbols from shared libraries before issuing any breakpoint or watch commands, especially if you regularly use tab completion on names.
    5. Optionally see the loaded shared libs with info sharedlibrary
    6. You should now be able to do, e.g., break ParallelLibrary.C:65, though you'll wait for a search period, and then continue.

Optionally enable a richer GDB experience by appending -ggdb to the CMAKE_*_FLAGS cited above.

Testing Process

For all testing: the absolute path to BINARY_DIR/test must be on your PATH before testing. Relative does not suffice due to multi-part drivers like textbook_3pc.

Recommended Process

  • Files required for testing, e.g., dakota_*.in, *.dat, various scripts, will be copied from test/ in the source tree to per-test subdirectories in test/ in the build tree at configure time. (If a needed file isn't copied, it must be called out in test/CMakeLists.txt. If a file is added, you must rerun CMake to update dependencies)
  • Test executables (drivers) are built in test/ in the binary tree at make time
  • There are a couple of ways to perform testing:
    1. Change into test in the build tree and run dakota_test.perl as usual, specifying the name of an input file. While the file won't exist in build/test, it will be found from the source tree.
    2. Preferred: Run ctest, either from the top level or (preferrably) from the test directory (see  http://cmake.org/cmake/help/v2.8.5/ctest.html for options)
      • You may run select tests using globs or regular expressions, e.g., -R dakota_ pdakota_. Note that parallel tests are named separately as pdakota_
      • You may run test files concurrently with -j N
      • There are some make targets to help generate the usual diffs files from the subdirectories. A typical full process might look like:
        cd build && make
        cd test
        # clean removes all diffs files, otherwise files will be appended to: 
        make dakota-diffs-clean
        ctest -j 4 -R dakota_ -R pdakota_
        make dakota-diffs
        
      • The usual dakota_diffs.out and dakota_pdiffs.out will results, as will a new file dakota_results.log, with a concise summary.
  • If you change a source dir test input file (or add one?) re-run make at least in test in build tree

Alternate, In-Source Process

  • If you prefer testing in the source directory, you can do so by setting your PATH appropriately, e.g., export PATH=/path/to/dakotabuild/test:/path/to/dakotabuild/src:$PATH
  • Then run dakota_test.perl as usual in the source directory test area