wiki:Documentation/runtests

Version 2 (modified by wehart, 10 years ago) (diff)

--

runtests

The runtests script is a generic driver for running a heterogeneous set of testing tools in a large software project.

Usage

This is still evolving. The following use cases are being used to motivate the development of this tool.

Example1

cd acro
bin/runtests

This runs all acro tests.

Example2

cd acro/packages/colin
lbin runtests

This runs only COLIN tests. (The use of 'lbin' here is a convenience. The 'runtests' command is installed in acro/bin)

Example3

cd acro
runtests --package colin

This runs only COLIN tests.

Example4

cd acro
runtests --tag smoke

This runs only smoke tests.

Example5

cd acro
runtests --tag smoke --tag parallel

This runs all parallel smoke tests.

Example6

cd acro
runtests --tester cxxtest

This runs all cxxtest tests.

Example7

cd acro
runtests --timelimit

This specifies a timelimit for the completion of all tests.

Example8

cd acro
runtests -j 4

This specifies that up to 4 tests will be run concurrently (on the local machine).

Example9

Currently, runtests is configured statically, in the FAST python software. I plan to develop a  YAML configuration file for runtests. I propose that the default runtests configuration file will be acro/admin/runtests.yaml, and a user can specify a configuration file as follows:

cd acro
runtests --config myconfig.yaml

 YAML is a human friendly data serialization standard for all programming languages. I think that the following YAML file describes the current configuration of runtests:

#
# Declare categories with nested maps
#
categories:
  nproc:
    serial:
      doc = Tests that run on a single processor
      default = true
    parallel:
      doc = Tests that run on multiple processors

  scope:
    smoke:
      doc = Tests that execute quickly and exercise basic functionality
      default = true
    nightly:
      doc = Tests that are suitable for testing nightly builds
    performance:
      doc = Tests that measure code performance
    acceptance:
      doc = Tests used to manager releases

  granularity:
    unit:
      doc = Tests that validate the operation of independent software components
      default = true
    system:
      doc = Tests that validate the system-level operation of software

#
# Declare tests as a sequence of mappings
#
  # UTILIB
-
  type:    cxxtest
  package: utilib
  dir:     ${root}/packages/utilib/test/unit
-
  type:    exact
  package: utilib
  dir:     ${root}/packages/utilib/test/studies

  # COLIN
-
  type:    cxxtest
  package: colin
  dir:     ${root}/packages/colin/test/unit
-
  type:    exact
  package: colin
  dir:     ${root}/packages/colin/test/studies
-
  type:    nose
  package: colin
  dir:     ${root}/packages/colin/test/driver

  # PEBBL
-
  type:    exact
  package: pebbl
  dir:     ${root}/packages/pebbl/test

  # PICO
-
  type:    exact
  package: pico
  dir:     ${root}/packages/pico/test
-
  type:    nose
  package: pico
  dir:     ${root}/packages/pico/test/driver
-
  type:    nose
  package: pico
  dir:     ${root}/packages/pico/test/sucasa

  # Interfaces
-
  type:    exact
  package: interfaces
  dir:     ${root}/packages/interfaces/test

  # SCOLIB
-
  type:    exact
  package: scolib
  dir:     ${root}/packages/scolib/test

If you like, YAML also supports formats that do not rely on whitespace to define blocks.