Subversion Repositories openrisc

         xml:id="manual.intro.setup.test" xreflabel="Testing">

      ISO C++

      test

      testsuite

      performance

      conformance

      ABI

      exception safety

The libstdc++ testsuite includes testing for standard conformance,

regressions, ABI, and performance.

  The directory libsrcdir/testsuite contains the

  individual test cases organized in sub-directories corresponding to

  chapters of the C++ standard (detailed below), the dejagnu test

  harness support files, and sources to various testsuite utilities

  that are packaged in a separate testing library.

  All test cases for functionality required by the runtime components

  of the C++ standard (ISO 14882) are files within the following

  directories.

17_intro

18_support

19_diagnostics

20_util

21_strings

22_locale

23_containers

25_algorithms

26_numerics

27_io

28_regex

29_atomics

30_threads

      In addition, the following directories include test files:

tr1               Tests for components as described by the Technical Report on Standard Library Extensions (TR1).

backward          Tests for backwards compatibility and deprecated features.

demangle          Tests for __cxa_demangle, the IA 64 C++ ABI demangler

ext               Tests for extensions.

performance       Tests for performance analysis, and performance regressions.

      Some directories don't have test files, but instead contain

      auxiliary information:

config            Files for the dejagnu test harness.

lib               Files for the dejagnu test harness.

libstdc++*        Files for the dejagnu test harness.

data              Sample text files for testing input and output.

util              Files for libtestc++, utilities and testing routines.

      Within a directory that includes test files, there may be

      additional subdirectories, or files.  Originally, test cases

      were appended to one file that represented a particular section

      of the chapter under test, and was named accordingly. For

      instance, to test items related to  21.3.6.1 -

      basic_string::find [lib.string::find] in the standard,

      the following was used:

21_strings/find.cc

      However, that practice soon became a liability as the test cases

      became huge and unwieldy, and testing new or extended

      functionality (like wide characters or named locales) became

      frustrating, leading to aggressive pruning of test cases on some

      platforms that covered up implementation errors. Now, the test

      suite has a policy of one file, one test case, which solves the

      above issues and gives finer grained results and more manageable

      error debugging. As an example, the test case quoted above

      becomes:

21_strings/basic_string/find/char/1.cc

21_strings/basic_string/find/char/2.cc

21_strings/basic_string/find/char/3.cc

21_strings/basic_string/find/wchar_t/1.cc

21_strings/basic_string/find/wchar_t/2.cc

21_strings/basic_string/find/wchar_t/3.cc

      All new tests should be written with the policy of one test

      case, one file in mind.

      In addition, there are some special names and suffixes that are

      used within the testsuite to designate particular kinds of

      tests.

   _xin.cc

      This test case expects some kind of interactive input in order

      to finish or pass. At the moment, the interactive tests are not

      run by default. Instead, they are run by hand, like:

g++ 27_io/objects/char/3_xin.cc

cat 27_io/objects/char/3_xin.in | a.out

     .in

      This file contains the expected input for the corresponding 

      _xin.cc test case.

   _neg.cc

      This test case is expected to fail: it's a negative test. At the

      moment, these are almost always compile time errors.

   char

      This can either be a directory name or part of a longer file

      name, and indicates that this file, or the files within this

      directory are testing the char instantiation of a

      template.

   wchar_t

      This can either be a directory name or part of a longer file

      name, and indicates that this file, or the files within this

      directory are testing the wchar_t instantiation of

      a template. Some hosts do not support wchar_t

      functionality, so for these targets, all of these tests will not

      be run.

   thread

      This can either be a directory name or part of a longer file

      name, and indicates that this file, or the files within this

      directory are testing situations where multiple threads are

      being used.

   performance

      This can either be an enclosing directory name or part of a

      specific file name. This indicates a test that is used to

      analyze runtime performance, for performance regression testing,

      or for other optimization related analysis. At the moment, these

      test cases are not run by default.

      You can check the status of the build without installing it

      using the dejagnu harness, much like the rest of the gcc

      tools.

    make check

   in the libbuilddir directory.

   or

    make check-target-libstdc++-v3

   in the gccbuilddir directory.

       These commands are functionally equivalent and will create a

       'testsuite' directory underneath

       libbuilddir containing the results of the

       tests. Two results files will be generated: 

       libstdc++.sum, which is a PASS/FAIL summary for each

       test, and libstdc++.log which is a log of

       the exact command line passed to the compiler, the compiler

       output, and the executable output (if any).

       Archives of test results for various versions and platforms are

       available on the GCC website in the build

       status section of each individual release, and are also

       archived on a daily basis on the gcc-testresults

       mailing list. Please check either of these places for a similar

       combination of source version, operating system, and host CPU.

      There are several options for running tests, including testing

      the regression tests, testing a subset of the regression tests,

      testing the performance tests, testing just compilation, testing

      installed tools, etc. In addition, there is a special rule for

      checking the exported symbols of the shared library.

      To debug the dejagnu test harness during runs, try invoking with a

      specific argument to the variable RUNTESTFLAGS, as below.

make check-target-libstdc++-v3 RUNTESTFLAGS="-v"

      or

make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"

      To run a subset of the library tests, you will need to generate

      the testsuite_files file by running

      make testsuite_files in the

      libbuilddir/testsuite directory, described

      below.  Edit the file to remove the tests you don't want and

      then run the testsuite as normal.

      There are two ways to run on a simulator: set up DEJAGNU to point to a

      specially crafted site.exp, or pass down --target_board flags.

    Example flags to pass down for various embedded builds are as follows:

      --target=powerpc-eabism (libgloss/sim)

make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"

--target=calmrisc32 (libgloss/sid)

make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"

--target=xscale-elf (newlib/sim)

make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"

      Also, here is an example of how to run the libstdc++ testsuite

      for a multilibed build directory with different ABI settings:

make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'

      You can run the tests with a compiler and library that have

      already been installed.  Make sure that the compiler (e.g.,

      g++) is in your PATH.  If you are

      using shared libraries, then you must also ensure that the

      directory containing the shared version of libstdc++ is in your

      LD_LIBRARY_PATH, or equivalent.  If your GCC source

      tree is at /path/to/gcc, then you can run the tests

      as follows:

runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite

      The testsuite will create a number of files in the directory in

      which you run this command,.  Some of those files might use the

      same name as files created by other testsuites (like the ones

      for GCC and G++), so you should not try to run all the

      testsuites in parallel from the same directory.

      In addition, there are some testing options that are mostly of

      interest to library maintainers and system integrators. As such,

      these tests may not work on all cpu and host combinations, and

      may need to be executed in the

      libbuilddir/testsuite directory.  These

      options include, but are not necessarily limited to, the

      following:

   make testsuite_files

    Five files are generated that determine what test files

    are run. These files are:

         testsuite_files

         This is a list of all the test cases that will be run. Each

         test case is on a separate line, given with an absolute path

         from the libsrcdir/testsuite directory.

         testsuite_files_interactive

         This is a list of all the interactive test cases, using the

         same format as the file list above. These tests are not run

         by default.

         testsuite_files_performance

         This is a list of all the performance test cases, using the

         same format as the file list above. These tests are not run

         by default.

         testsuite_thread

         This file indicates that the host system can run tests which

         involved multiple threads.

         testsuite_wchar_t

         This file indicates that the host system can run the wchar_t

         tests, and corresponds to the macro definition 

         _GLIBCXX_USE_WCHAR_T in the file c++config.h.

   make check-abi

     The library ABI can be tested. This involves testing the shared

     library against an ABI-defining previous version of symbol

     exports.

   make check-compile

     This rule compiles, but does not link or execute, the

     testsuite_files test cases and displays the

     output on stdout.

   make check-performance

     This rule runs through the

     testsuite_files_performance test cases and

     collects information for performance analysis and can be used to

     spot performance regressions. Various timing information is

     collected, as well as number of hard page faults, and memory

     used. This is not run by default, and the implementation is in

     flux.

      We are interested in any strange failures of the testsuite;

      please email the main libstdc++ mailing list if you see

      something odd or have questions.

      To run the libstdc++ test suite under the debug mode, edit

      libstdc++-v3/scripts/testsuite_flags to add the

      compile-time flag -D_GLIBCXX_DEBUG to the

      result printed by the --build-cxx

      option. Additionally, add the

      -D_GLIBCXX_DEBUG_PEDANTIC flag to turn on

      pedantic checking. The libstdc++ test suite should produce

      precisely the same results under debug mode that it does under

      release mode: any deviation indicates an error in either the

      library or the test suite.

      The parallel

      mode can be tested in much the same manner, substituting

      -D_GLIBCXX_PARALLEL for

      -D_GLIBCXX_DEBUG in the previous paragraph.

      Or, just run the testsuites with CXXFLAGS

      set to -D_GLIBCXX_DEBUG or

      -D_GLIBCXX_PARALLEL.

    The first step in making a new test case is to choose the correct

    directory and file name, given the organization as previously

    described.

    All files are copyright the FSF, and GPL'd: this is very

    important.  The first copyright year should correspond to the date

    the file was checked in to SVN.

     As per the dejagnu instructions, always return 0 from main to

     indicate success.

   A bunch of utility functions and classes have already been

   abstracted out into the testsuite utility library, 

   libtestc++. To use this functionality, just include the

   appropriate header file: the library or specific object files will

   automatically be linked in as part of the testsuite run.

   For a test that needs to take advantage of the dejagnu test

   harness, what follows below is a list of special keyword that

   harness uses. Basically, a test case contains dg-keywords (see

   dg.exp) indicating what to do and what kinds of behavior are to be

   expected.  New test cases should be written with the new style

   DejaGnu framework in mind.

    To ease transition, here is the list of dg-keyword documentation

    lifted from dg.exp.

# The currently supported options are:

#

# dg-prms-id N

#       set prms_id to N

#

# dg-options "options ..." [{ target selector }]

#       specify special options to pass to the tool (eg: compiler)

#

# dg-do do-what-keyword [{ target/xfail selector }]

#       `do-what-keyword' is tool specific and is passed unchanged to

#       ${tool}-dg-test.  An example is gcc where `keyword' can be any of:

#       preprocess|compile|assemble|link|run

#       and will do one of: produce a .i, produce a .s, produce a .o,

#       produce an a.out, or produce an a.out and run it (the default is

#       compile).

#

# dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]

#       indicate an error message <regexp> is expected on this line

#       (the test fails if it doesn't occur)

#       Linenum=0 for general tool messages (eg: -V arg missing).

#       "." means the current line.

#

# dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]

#       indicate a warning message <regexp> is expected on this line

#       (the test fails if it doesn't occur)

#

# dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]

#       indicate a bogus error message <regexp> use to occur here

#       (the test fails if it does occur)

#

# dg-build regexp comment [{ target/xfail selector }]

#       indicate the build use to fail for some reason

#       (errors covered here include bad assembler generated, tool crashes,

#       and link failures)

#       (the test fails if it does occur)

#

# dg-excess-errors comment [{ target/xfail selector }]

#       indicate excess errors are expected (any line)

#       (this should only be used sparingly and temporarily)

#

# dg-output regexp [{ target selector }]

#       indicate the expected output of the program is <regexp>

#       (there may be multiple occurrences of this, they are concatenated)

#

# dg-final { tcl code }

#       add some tcl code to be run at the end

#       (there may be multiple occurrences of this, they are concatenated)

#       (unbalanced braces must be \-escaped)

#

# "{ target selector }" is a list of expressions that determine whether the

# test succeeds or fails for a particular target, or in some cases whether the

# option applies for a particular target.  If the case of `dg-do' it specifies

# whether the test case is even attempted on the specified target.

#

# The target selector is always optional.  The format is one of:

#

# { xfail *-*-* ... } - the test is expected to fail for the given targets

# { target *-*-* ... } - the option only applies to the given targets

#

# At least one target must be specified, use *-*-* for "all targets".

# At present it is not possible to specify both `xfail' and `target'.

# "native" may be used in place of "*-*-*".

Example 1: Testing compilation only

// { dg-do compile }

Example 2: Testing for expected warnings on line 36, which all targets fail

// { dg-warning "string literals" "" { xfail *-*-* } 36 }

Example 3: Testing for expected warnings on line 36

// { dg-warning "string literals" "" { target *-*-* } 36 }

Example 4: Testing for compilation errors on line 41

// { dg-do compile }

// { dg-error "no match for" "" { target *-*-* } 41 }

Example 5: Testing with special command line settings, or without the

use of pre-compiled headers, in particular the stdc++.h.gch file. Any

options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set

up in the normal.exp file.

// { dg-options "-O0" { target *-*-* } }

    More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.

    Underlying details of testing for conformance and regressions are

    abstracted via the GNU Dejagnu package. This is similar to the

    rest of GCC.

This is information for those looking at making changes to the testsuite

structure, and/or needing to trace dejagnu's actions with --verbose.  This

will not be useful to people who are "merely" adding new tests to the existing

structure.

The first key point when working with dejagnu is the idea of a "tool".

Files, directories, and functions are all implicitly used when they are

named after the tool in use.  Here, the tool will always be "libstdc++".

The lib subdir contains support routines.  The

lib/libstdc++.exp file ("support library") is loaded

automagically, and must explicitly load the others.  For example, files can

be copied from the core compiler's support directory into lib.

Some routines in lib/libstdc++.exp are callbacks, some are

our own.  Callbacks must be prefixed with the name of the tool.  To easily

distinguish the others, by convention our own routines are named "v3-*".

The next key point when working with dejagnu is "test files".  Any

directory whose name starts with the tool name will be searched for test files.

(We have only one.)  In those directories, any .exp file is

considered a test file, and will be run in turn.  Our main test file is called

normal.exp; it runs all the tests in testsuite_files using the

callbacks loaded from the support library.

The config directory is searched for any particular "target

board" information unique to this library.  This is currently unused and sets

only default variables.

   The testsuite directory also contains some files that implement

   functionality that is intended to make writing test cases easier,

   or to avoid duplication, or to provide error checking in a way that

   is consistent across platforms and test harnesses. A stand-alone

   executable, called abi_check, and a static

   library called libtestc++ are

   constructed. Both of these items are not installed, and only used

   during testing.

  These files include the following functionality:

       testsuite_abi.h,

       testsuite_abi.cc,

       testsuite_abi_check.cc

        Creates the executable abi_check.

        Used to check correctness of symbol versioning, visibility of

        exported symbols, and compatibility on symbols in the shared

        library, for hosts that support this feature. More information

        can be found in the ABI documentation here

       testsuite_allocator.h,

       testsuite_allocator.cc

        Contains specialized allocators that keep track of construction

        and destruction. Also, support for overriding global new and

        delete operators, including verification that new and delete

        are called during execution, and that allocation over max_size

        fails.

       testsuite_character.h

        Contains std::char_traits and

        std::codecvt specializations for a user-defined

        POD.

       testsuite_hooks.h,

       testsuite_hooks.cc

       A large number of utilities, including:

         VERIFY

         set_memory_limits

         verify_demangle

         run_tests_wrapped_locale

         run_tests_wrapped_env

         try_named_locale

         try_mkfifo

         func_callback

         counter

         copy_tracker

         copy_constructor

         assignment_operator

         destructor

         pod_char, pod_int and associated char_traits specializations

         testsuite_io.h

       Error, exception, and constraint checking for

       std::streambuf, std::basic_stringbuf, std::basic_filebuf.

         testsuite_iterators.h

         Wrappers for various iterators.

         testsuite_performance.h

       A number of class abstractions for performance counters, and

       reporting functions including:

         time_counter

         resource_counter

         report_performance

         Testing is composed of running a particular test sequence,

         and looking at what happens to the surrounding code when

         exceptions are thrown. Each test is composed of measuring

         initial state, executing a particular sequence of code under

         some instrumented conditions, measuring a final state, and

         then examining the differences between the two states.

         Test sequences are composed of constructed code sequences

         that exercise a particular function or member function, and

         either confirm no exceptions were generated, or confirm the

         consistency/coherency of the test subject in the event of a

         thrown exception.

         Random code paths can be constructed using the basic test

         sequences and instrumentation as above, only combined in a

         random or pseudo-random way.

        To compute the code paths that throw, test instruments

         are used that throw on allocation events

         (__gnu_cxx::throw_allocator_random

         and __gnu_cxx::throw_allocator_limit)

         and copy, assignment, comparison, increment, swap, and

         various operators

         (__gnu_cxx::throw_type_random

         and __gnu_cxx::throw_type_limit). Looping

         through a given test sequence and conditionally throwing in

         all instrumented places.  Then, when the test sequence

         completes without an exception being thrown, assume all

         potential error paths have been exercised in a sequential

         manner.

         Ad Hoc

         For example,

         testsuite/23_containers/list/modifiers/3.cc.

         Policy Based Data Structures

         For example, take the test

         functor rand_reg_test in

         in testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc. This uses container_rand_regression_test in

testsuite/util/regression/rand/assoc/container_rand_regression_test.h.

         Which has several tests for container member functions,

Includes control and test container objects. Configuration includes

random seed, iterations, number of distinct values, and the

probability that an exception will be thrown. Assumes instantiating

container uses an extension

allocator, __gnu_cxx::throw_allocator_random,

as the allocator type.

         C++11 Container Requirements.