wiki:Development/NightlyTests

Testing Philosophy

Our goal is to support a variety of different categories of software tests within Acro, including:

  • Unit Tests - Tests that independently exercise C++ component interfaces
  • Functionality Tests - Tests that validate the behavior of intra-package interfaces (these are notionally testing a higher-level concepts than are addressed by Unit Tests).
  • Integration Tests - Tests that validate the behavior of inter-package interfaces
  • Regression Tests - Tests that use simple 'diffs' of output (e.g. code that can be driven within shell scripts with config files).
  • Performance Tests - Tests that evaluate runtime, cost and/or the number of iterations over a suite of test problems. (Test comparisons are made with respect to a history of previous test results.)
  • Benchmark Tests - Test that evaluate the absolute runtime, cost and/or number of iterations over a suite of test problems. (Test comparisons are made with respect to published results and/or third-party solvers.)

The following table summarizes the types of tests that are supported within Acro, and the maturity of Acro's testing capabilities.

Build Tests

Code-Coverage Results

DAKOTA Integration Tests