Subversion Repositories or1k

This is dejagnu.info, produced by makeinfo version 4.0 from
dejagnu.texi.

START-INFO-DIR-ENTRY
* DejaGnu: (dejagnu).            The GNU testing framework.
END-INFO-DIR-ENTRY

   Copyright (C) 92, 93, 94, 95, 1996 Free Software Foundation, Inc.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the entire resulting derived work is distributed under the terms
of a permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions.


File: dejagnu.info,  Node: Top,  Next: Overview,  Up: (dir)

DejaGnu
*******

   DejaGnu is a framework for running test suites on software tools.

   This file describes version 1.3 of DejaGnu.

* Menu:

* Overview::                    What is DejaGnu?
* What is New::                 What is new in this release.
* Invoking runtest::            Using `runtest', the main test driver
* Customizing::                 Setting `runtest' defaults
* Internals::                   The DejaGnu implementation
* Tests::                       How to write a test case
* Extending::                   New tools, new targets, and new hosts
* Installation::                Configuring and Installing DejaGnu
* Index::                       Index


File: dejagnu.info,  Node: Overview,  Next: What is New,  Prev: Top,  Up: Top

What is DejaGnu?
****************

   DejaGnu is a framework for testing other programs.  Its purpose is to
provide a single front end for all tests.  Beyond this, DejaGnu offers
several advantages for testing:

  1. The flexibility and consistency of the DejaGnu framework make it
     easy to write tests for any program.

  2. DejaGnu provides a layer of abstraction which allows you to write
     tests that are portable to any host or target where a program must
     be tested.   For instance, a test for GDB can run (from any Unix
     based host) on any target architecture that DejaGnu supports.
     Currently DejaGnu runs tests on several single board computers,
     whose operating software ranges from just a boot monitor to a
     full-fledged, Unix-like realtime OS.

  3. All tests have the same output format.  This makes it easy to
     integrate testing into other software development processes.
     DejaGnu's output is designed to be parsed by other filtering
     script, and it is also human readable.

   DejaGnu is written in `expect', which in turn uses "Tcl"--Tool
command language.

   Running tests requires two things: the testing framework, and the
test suites themselves.  Tests are usually written in `expect' using
Tcl, but you can also use a Tcl script to run a test suite that is not
based on `expect'.  (`expect' script filenames conventionally use
`.exp' as a suffix; for example, the main implementation of the DejaGnu
test driver is in the file `runtest.exp'.)

* Menu:

* Running Tests::               A first look at running DejaGnu tests
* Sample Test::                 What does a DejaGnu test case look like?
* Design Goals::                Goals behind DejaGnu
* Posix::                       DejaGnu conforms to POSIX 1003.3
* Future Directions::           Where is DejaGnu going?
* Tcl and Expect::              Reading more about Tcl and Expect


File: dejagnu.info,  Node: What is New,  Next: Invoking runtest,  Prev: Overview,  Up: Top

What is new in this release ?
*****************************

   This release has a number of substantial changes over version 1.2.
The most visible change is that the version of expect and Tcl included
in the release are up-to-date with the current stable net releases.
Other changes are:

  1. The config sub-system in DejaGnu has been completely redesigned.
     It now supports testing on remote hosts as well as remote targets.

  2. More builtin support for building target binaries with the correct
     linker flags. Currently this only works with GCC, preferably with a
     target support by `libgloss'.

  3. Lots of little bug fixes from a year of heavy use here at Cygnus
     Support.

  4. DejaGnu now uses `autoconf' for configuration.

  5. New test cases for DejaGnu have been added for the new features,
     plus the "-tool" option bug in the 1.2 testsuite has been fixed.

  6. The `--tool' option is now optional.

  7. `runtest' when searching for test drivers ignores all directories
     named SCCS, RCS, and CVS.

  8. There is now a generic keyword based test harness that uses
     comments in source code to control how each test case gets built
     and run.

  9. There is now some support for running a testsuite with multiple
     passes and multiple targets.



File: dejagnu.info,  Node: Running Tests,  Next: Sample Test,  Up: Overview

Running existing tests
======================

   To run tests from an existing collection, first use `configure' as
usual to set up the source directory containing the tests.  Then try
running

     make check

   If the `check' target exists, it usually saves you some trouble--for
instance, it can set up any auxiliary programs or other files needed by
the tests.

   Once you have run `make check' to build any auxiliary files, you
might want to call the test driver `runtest' directly to repeat the
tests.  You may also have to call `runtest' directly for test
collections with no `check' target in the `Makefile'.

   Typically, you must use two command-line options: `--tool', to
specify which set of tests to run(1), and `--srcdir', to specify where
to find test directories.

   For example, if the directory `gdb/testsuite' contains a collection
of DejaGnu tests for GDB, you can run them like this:

     eg$ cd gdb/testsuite
     eg$ runtest --tool gdb
_Test output follows, ending with:_

                === gdb Summary ===
     
     # of expected passes 508
     # of expected failures 103
     /usr/latest/bin/gdb version 4.14.4 -nx

   You can use the option `--srcdir' to point to some other directory
containing a collection of tests:

     eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite

   These examples assume a "native" configuration, where the same
computer runs both `runtest' and the tests themselves.  When you have a
"cross" configuration, the tests run on a different computer,
controlled by the host running `runtest'.  In this situation, you need
the option `--name' to specify the network address for the other
computer:

     eg$ runtest --tool gdb --name vx9.munist.com

   If you always use the same option values, you can record them in a
file called `site.exp', rather than typing them each time.  *Note
Setting defaults for `runtest' options: Config Values.

   By default, `runtest' prints only the names of the tests it runs,
output from any tests that have unexpected results, and a summary
showing how many tests passed and how many failed.  To display output
from all tests (whether or not they behave as expected), use the
`--all' option.  For more verbose output about processes being run,
communication, and so on, use `--verbose'. To see even more output, use
multiple `--verbose' options.  *Note Using `runtest': Invoking runtest,
for a more detailed explanation of each `runtest' option.

   Test output goes into two files in your current directory: summary
output in `TOOL.sum', and detailed output in `TOOL.log'.  (TOOL refers
to the collection of tests; for example, after a run with `--tool gdb',
look for output files `gdb.sum' and `gdb.log'.)  *Note The files
DejaGnu writes: Output Files.

   ---------- Footnotes ----------

   (1) `--tool' selects a particular suite of tests, _not_ the name of
the executable program to run.  *Note Configuration dependent values:
Config Values, for information on the variables that you can use to
specify the names of programs to run.


File: dejagnu.info,  Node: Sample Test,  Next: Design Goals,  Prev: Running Tests,  Up: Overview

What does a DejaGnu test look like?
===================================

   Each DejaGnu test is an `expect' script; the tests vary widely in
complexity, depending on the nature of the tool and the feature tested.

   Here is a very simple GDB test--one of the simplest tests shipped
with DejaGnu (extracted from `gdb.t00/echo.exp'):(1)

     # send a string to the GDB stdin:
     send "echo Hello world!\n"
     
     # inspect the GDB stdout for the correct reply,
     # and determine whether the test passes or fails:
     expect {
       -re "Hello world.*$prompt $"    { pass "Echo test" }
       -re "$prompt $"                 { fail "Echo test" }
       timeout                         { fail "(timeout) Echo test" }
       }

   Though brief, this example is a complete test.  It illustrates some
of the main features of DejaGnu test scripts:

   * The test case does not start the tested program (GDB in this case);
     all test scripts for interactive tools can assume the
     corresponding tool is running.

   * Comments start with `#'.

   * The main commands you use to control a tested program are `send'
     (to give it commands) and `expect' (to analyze its responses).

   * The `expect' command uses a list of pairs; a pattern (regular
     expression if `-re' specified), followed by an action to run if the
     pattern matches output from the program.  Only the action for the
     _first_ matching pattern will execute.

   * Test cases use the commands `pass' and `fail' to record the test
     outcome.

   ---------- Footnotes ----------

   (1) More recent GDB tests use the `gdb_test' procedure.  An
equivalent test using that procedure is ` gdb_test "echo Hello world!"
"Hello world!" '


File: dejagnu.info,  Node: Design Goals,  Next: Posix,  Prev: Sample Test,  Up: Overview

Design goals
============

   DejaGnu grew out of the internal needs of Cygnus Support.  Cygnus
maintains and enhances a variety of free programs in many different
environments, and we needed a testing tool that:

   * is useful to developers while fixing bugs;

   * automates running many tests during a software release process;

   * is portable among a variety of host computers;

   * supports cross-development testing;

   * permits testing interactive programs, like GDB; and

   * permits testing batch oriented programs, like GCC.

   Some of the requirements proved challenging.  For example,
interactive programs do not lend themselves very well to automated
testing.  But all the requirements are important: for instance, it is
imperative to make sure that GDB works as well when cross-debugging as
it does in a native configuration.

   Probably the greatest challenge was testing in a cross-development
environment (which can be a real nightmare).  Most cross-development
environments are customized by each developer.  Even when buying
packaged boards from vendors there are many differences.  The
communication interfaces vary from a serial line to ethernet.  DejaGnu
was designed with a modular communication setup, so that each kind of
communication can be added as required, and supported thereafter.  Once
a communication procedure is coded, any test can use it.  Currently
DejaGnu can use `rsh', `rlogin', `telnet', `tip', `kermit', and
`mondfe' for remote communications.

   Julia Menapace first coined the term "Deja Gnu" to describe an
earlier testing framework at Cygnus Support.  When we replaced it with
the Expect-based framework, it was like DejaGnu all over again...


File: dejagnu.info,  Node: Posix,  Next: Future Directions,  Prev: Design Goals,  Up: Overview

A POSIX conforming test framework
=================================

   DejaGnu conforms to the POSIX standard for test frameworks.

   POSIX standard 1003.3 defines what a testing framework needs to
provide, in order to permit the creation of POSIX conformance test
suites. This standard is primarily oriented to running POSIX
conformance tests, but its requirements also support testing of features
not related to POSIX conformance.  POSIX 1003.3 does not specify a
particular testing framework, but at this time there is only one other
POSIX conforming test framework: TET.(1)

   The POSIX documentation refers to "assertions".  An assertion is a
description of behavior.  For example, if a standard says "The sun
shall shine", a corresponding assertion might be "The sun is shining."
A test based on this assertion would pass or fail depending on whether
it is daytime or nighttime.  It is important to note that the standard
being tested is never 1003.3; the standard being tested is some other
standard, for which the assertions were written.

   As there is no test suite to test _testing frameworks_ for POSIX
1003.3 conformance, verifying conformance to this standard is done by
repeatedly reading the standard and experimenting.  One of the main
things 1003.3 does specify is the set of allowed output messages, and
their definitions.  Four messages are supported for a required feature
of POSIX conforming systems, and a fifth for a conditional feature.
DejaGnu supports the use of all five output messages; in this sense a
test suite that uses exactly these messages can be considered POSIX
conforming.  These definitions specify the output of a test case:

`PASS'
     A test has succeeded.  That is, it demonstrated that the assertion
     is true.

`XFAIL'
     POSIX 1003.3 does not incorporate the notion of expected failures,
     so `PASS', instead of `XPASS', must also be returned for test
     cases which were expected to fail and did not.  This means that
     `PASS' is in some sense more ambiguous than if `XPASS' is also
     used.  For information on `XPASS' and `XFAIL', see *Note Using
     `runtest': Invoking runtest.

`FAIL'
     A test _has_ produced the bug it was intended to capture.  That is,
     it has demonstrated that the assertion is false.  The `FAIL'
     message is based on the test case only.  Other messages are used to
     indicate a failure of the framework.

     As with `PASS', POSIX tests must return `FAIL' rather than `XFAIL'
     even if a failure was expected.

`UNRESOLVED'
     A test produced indeterminate results.  Usually, this means the
     test executed in an unexpected fashion; this outcome requires that
     a human being go over results, to determine if the test should
     have passed or failed.  This message is also used for any test
     that requires human intervention because it is beyond the
     abilities of the testing framework.  Any unresolved test should
     resolved to `PASS' or `FAIL' before a test run can be considered
     finished.

     Note that for POSIX, each assertion must produce a test result
     code.  If the test isn't actually run, it must produce `UNRESOLVED'
     rather than just leaving that test out of the output.  This means
     that you have to be careful when writing tests, to not carelessly
     use tcl statements like `return'--if you alter the flow of control
     of the tcl code you must insure that every test still produces
     some result code.

     Here are some of the ways a test may wind up `UNRESOLVED':

        * A test's execution is interrupted.

        * A test does not produce a clear result. This is usually
          because there was an `ERROR' from DejaGnu while processing
          the test, or because there were three or more `WARNING'
          messages. Any `WARNING' or `ERROR' messages can invalidate
          the output of the test.  This usually requires a human being
          to examine the output to determine what really happened--and
          to improve the test case.

        * A test depends on a previous test, which fails.

        * The test was set up incorrectly.

`UNTESTED'
     A test was not run.  This is a placeholder, used when there is no
     real test case yet.

The only remaining output message left is intended to test features that
are specified by the applicable POSIX standard as conditional:

`UNSUPPORTED'
     There is no support for the tested case.  This may mean that a
     conditional feature of an operating system, or of a compiler, is
     not implemented.  DejaGnu also uses this message when a testing
     environment (often a "bare board" target) lacks basic support for
     compiling or running the test case.  For example, a test for the
     system subroutine `gethostname' would never work on a target board
     running only a boot monitor.

   DejaGnu uses the same output procedures to produce these messages for
all test suites, and these procedures are already known to conform to
POSIX 1003.3.  For a DejaGnu test suite to conform to POSIX 1003.3, you
must avoid the `setup_xfail' procedure as described in the `PASS'
section above, and you must be careful to return `UNRESOLVED' where
appropriate, as described in the `UNRESOLVED' section above.

   ---------- Footnotes ----------

   (1) TET was created by Unisoft for a consortium comprised of X/Open,
Unix International, and the Open Software Foundation.


File: dejagnu.info,  Node: Future Directions,  Next: Tcl and Expect,  Prev: Posix,  Up: Overview

Future directions
=================

   In the near future, there are two parallel directions for DejaGnu
development.  The first is to add support for more hosts and targets.

   The second would permit testing programs with a more complex
interface, whether text based or GUI based.  Two components already
exist: a Tcl based X window toolkit, and a terminal package for
`expect'. Both of these could be merged into DejaGnu in a way that
permits testing programs that run in each environment.

   Meanwhile, we hope DejaGnu enables the creation of test suites for
conformance to ANSI C and C++, to POSIX, and to other standards.  We
encourage you to make any test suites you create freely available,
under the same terms as DejaGnu itself.


File: dejagnu.info,  Node: Tcl and Expect,  Prev: Future Directions,  Up: Overview

Tcl and Expect
==============

   Tcl was introduced in a paper by John K. Ousterhout at the 1990
Winter Usenix conference, `Tcl: An Embeddable Command Language'.  That
paper is included in PostScript form in the `doc' subdirectory of the
Tcl distribution. The version of Tcl included in DejaGnu at this time is
Tcl 7.4p3.

   Don Libes introduced `expect' in his paper `expect: Curing Those
Uncontrollable Fits of Interaction' at the 1990 Summer Usenix
conference.  The paper is included in PostScript form in the `expect'
distribution (as are several other papers about `expect'). The version
of expect included in DejaGnu at this time is expect 5.18.0.


File: dejagnu.info,  Node: Invoking runtest,  Next: Customizing,  Prev: What is New,  Up: Top

Using `runtest'
***************

   `runtest' is the executable test driver for DejaGnu.  You can
specify two kinds of things on the `runtest' command line: command line
options, and Tcl variables for the test scripts.  The options are
listed alphabetically below.

   `runtest' returns an exit code of `1' if any test has an unexpected
result; otherwise (if all tests pass or fail as expected) it returns
`0' as the exit code.

   `runtest' flags the outcome of each test as one of these cases.
(*Note A POSIX conforming test framework: Posix, for a discussion of
how POSIX specifies the meanings of these cases.)

`PASS'
     The most desirable outcome: the test succeeded, and was expected to
     succeed.

`XPASS'
     A pleasant kind of failure: a test was expected to fail, but
     succeeded.  This may indicate progress; inspect the test case to
     determine whether you should amend it to stop expecting failure.

`FAIL'
     A test failed, although it was expected to succeed.  This may
     indicate regress; inspect the test case and the failing software
     to locate the bug.

`XFAIL'
     A test failed, but it was expected to fail.  This result indicates
     no change in a known bug.  If a test fails because the operating
     system where the test runs lacks some facility required by the
     test, the outcome is `UNSUPPORTED' instead.

`UNRESOLVED'
     Output from a test requires manual inspection; the test suite
     could not automatically determine the outcome.  For example, your
     tests can report this outcome is when a test does not complete as
     expected.

`UNTESTED'
     A test case is not yet complete, and in particular cannot yet
     produce a `PASS' or `FAIL'.  You can also use this outcome in dummy
     "tests" that note explicitly the absence of a real test case for a
     particular property.

`UNSUPPORTED'
     A test depends on a conditionally available feature that does not
     exist (in the configured testing environment).  For example, you
     can use this outcome to report on a test case that does not work
     on a particular target because its operating system support does
     not include a required subroutine.

   `runtest' may also display the following messages:

`ERROR'
     Indicates a major problem (detected by the test case itself) in
     running the test. This is usually an unrecoverable error, such as
     a missing file or loss of communication to the target.  (POSIX
     test suites should not emit this message; use `UNSUPPORTED',
     `UNTESTED', or `UNRESOLVED' instead, as appropriate.)

`WARNING'
     Indicates a possible problem in running the test. Usually warnings
     correspond to recoverable errors, or display an important message
     about the following tests.

`NOTE'
     An informational message about the test case.

   This is the full set of command line options that `runtest'
recognizes.  Arguments may be abbreviated to the shortest unique string.

     runtest --tool TOOL  [ TESTSUITE.exp ... ]
     [ TESTSUITE.exp="testfile1 ..." ]
     [ TCLVAR=VALUE... ]
     [ --all ]  [ --baud BAUD-RATE ]  [ --connect TYPE ]
     [ --debug ]  [ --help ]  [ --host STRING ]
     [ --mail "NAME ..." ]  [ --name STRING ]
     [ --name NAME ]  [ --outdir PATH ]
     [ --objdir PATH ]  [ --reboot ]
     [ --srcdir PATH ]  [ --strace N ]
     [ --target STRING --build STRING ]
     [ -v | --verbose ]  [ -V | --version ]  [ --DN ]

`--tool TOOL'
     TOOL specifies what set of tests to run, and what initialization
     module to use.  TOOL is used _only_ for these two purposes: it is
     _not_ used to name the executable program to test.  Executable
     tool names (and paths) are recorded in `site.exp' (*note
     Configuration dependent values: Config Values.), and you can
     override them by specifying Tcl variables on the command line.

     For example, including `--tool gcc' on the `runtest' command line
     runs tests from all test subdirectories whose names match `gcc.*',
     and uses one of the initialization modules named
     `config/*-gcc.exp'.  To specify the name of the compiler (perhaps
     as an alternative path to what `runtest' would use by default), use
     `GCC=BINNAME' on the `runtest' command line.

`TESTSUITE.exp ...'
     Specify the names of testsuites to run.  By default, `runtest'
     runs all tests for the tool, but you can restrict it to particular
     testsuites by giving the names of the `.exp' `expect' scripts that
     control them.

     TESTSUITE.exp may not include path information; use plain
     filenames.

`TESTFILE.exp="testfile1 ..."'
     Specify a subset of tests in a suite to run.  For compiler or
     assembler tests, which often use a single `.exp' script covering
     many different source files, this option allows you to further
     restrict the tests by listing particular source files to compile.
     Some tools even support wildcards here.  The wildcards supported
     depend upon the tool, but typically they are `?', `*', and
     `[chars]'.

`TCLVAR=VALUE'
     You can define Tcl variables for use by your test scripts in the
     same style used with `make' for environment variables.  For
     example, `runtest GDB=gdb.old' defines a variable called `GDB';
     when your scripts refer to `$GDB' in this run, they use the value
     `gdb.old'.

     The default Tcl variables used for most tools are defined in the
     main DejaGnu `Makefile'; their values are captured in the
     `site.exp' file.  *Note Configuration dependent values: Config
     Values.

`--all'
     Display all test output.  By default, `runtest' shows only the
     output of tests that produce unexpected results; that is, tests
     with status `FAIL' (unexpected failure), `XPASS' (unexpected
     success), or `ERROR' (a severe error in the test case itself).
     Specify `--all' to see output for tests with status `PASS'
     (success, as expected) `XFAIL' (failure, as expected), or
     `WARNING' (minor error in the test case itself).

`--baud BAUD-RATE'
`-b BAUD-RATE'
     Set the default baud rate to something other than 9600.  (Some
     serial interface programs, like `tip', use a separate
     initialization file instead of this value.)

`--connect TYPE'
     Connect to a target testing environment as specified by TYPE, if
     the target is not the computer running `runtest'.  For example, use
     `--connect' to change the program used to connect to a "bare
     board" boot monitor.  The choices for TYPE in the DejaGnu 1.0
     distribution are `rlogin', `telnet', `rsh', `tip', `kermit', and
     `mondfe'.

     The default for this option depends on the configuration (*note
     Remote targets supported: Cross Targets.).  The default is chosen
     to be the most convenient communication method available, but
     often other alternatives work as well; you may find it useful to
     try alternative connect methods if you suspect a communication
     problem with your testing target.

`--debug'
     Turns on the `expect' internal debugging output.  Debugging output
     is displayed as part of the `runtest' output, and logged to a file
     called `dbg.log'.  The extra debugging output does _not_ appear on
     standard output, unless the verbose level is greater than 2 (for
     instance, to see debug output immediately, specify `--debug -v
     -v').  The debugging output shows all attempts at matching the test
     output of the tool with the scripted patterns describing expected
     output.  The output generated with `--strace' also goes into
     `dbg.log'.

`--help'
`-he'
     Prints out a short summary of the `runtest' options, then exits
     (even if you also specify other options).

`--host STRING'
     STRING is a full configuration "triple" name as used by
     `configure'.  Use this option to override the default string
     recorded by your configuration's choice of host.  This choice does
     not change how anything is actually configured unless -build is
     also specified; it affects _only_ DejaGnu procedures that compare
     the host string with particular values.  The procedures `ishost',
     `istarget', `isnative', and `setup_xfail' are affected by
     `--host'. In this usage, `host' refers to the machine that the
     tests are to be run on, which may not be the same as the `build'
     machine. If `--build' is also specified, then `--host' refers to
     the machine that the tests wil, be run on, not the machine DejaGnu
     is run on.

`--build STRING'
     STRING is a full configuration "triple" name as used by
     `configure'. This is the type of machine DejaGnu and the tools to
     be tested are built on. For a normal cross this is the same as the
     host, but for a canadian cross, they are seperate.

`--name NAME'
     NAME is a name for the particular testing target machine (for
     cross testing).  If the testing target has IP network support (for
     example, `RPC' or `NFS'), this is the network name for the target
     itself.  (NAME is _not the configuration string_ you specify as a
     target with `configure'; the `--name' option names a particular
     target, rather than describing a class of targets.)  For targets
     that connect in other ways, the meaning of the NAME string depends
     on the connection method.  *Note Remote targets supported: Cross
     Targets.

`--name STRING'
     Specify a network name of testing target or its host.  The
     particular names that are meaningful with `--name' will depend on
     your site configuration, and on the connection protocol: for
     example, `tip' connections require names from a serial line
     configuration file (usually called `/etc/remote'), while `telnet'
     connections use IP hostnames.

`--objdir PATH'
     Use PATH as the top directory containing any auxiliary compiled
     test code. This defaults to `.'.  Use this option to locate
     pre-compiled test code.  You can normally prepare any auxiliary
     files needed with `make'.

`--outdir PATH'
     Write output logs in directory PATH.  The default is `.', the
     directory where you start `runtest'.  This option affects only the
     summary and the detailed log files `TOOL.sum' and `TOOL.log'.  The
     DejaGnu debug log `dbg.log' always appears (when requested) in the
     local directory.

`--reboot'
     Reboot the target board when `runtest' initializes.  Usually, when
     running tests on a separate target board, it is safer to reboot
     the target to be certain of its state.  However, when developing
     test scripts, rebooting takes a lot of time.

`--srcdir PATH'
     Use PATH as the top directory for test scripts to run.  `runtest'
     looks in this directory for any subdirectory whose name begins
     with the toolname (specified with `--tool').  For instance, with
     `--tool gdb', `runtest' uses tests in subdirectories `gdb.*' (with
     the usual shell-like filename expansion).  If you do not use
     `--srcdir', `runtest' looks for test directories under the current
     working directory.

`--strace N'
     Turn on internal tracing for `expect', to N levels deep. By
     adjusting the level, you can control the extent to which your
     output expands multi-level Tcl statements.  This allows you to
     ignore some levels of `case' or `if' statements.  Each procedure
     call or control structure counts as one "level".

     The output is recorded in the same file, `dbg.log', used for output
     from `--debug'.

`--target STRING'
     Use this option to override the default setting (running native
     tests).  STRING is a full configuration "triple" name(1) as used
     by `configure'.  This option changes the configuration `runtest'
     uses for the default tool names, and other setup information.
     *Note Using `configure': (configure.info)Using configure, for
     details about `configure' names.

`--verbose'
`-v'
     Turns on more output.  Repeating this option increases the amount
     of output displayed.  Level one (`-v') is simply test output. Level
     two (`-v -v') shows messages on options, configuration, and process
     control.  Verbose messages appear in the detailed (`*.log') log
     file, but not in the summary (`*.sum') log file.

`--version'
`-V'
     Prints out the version numbers of DejaGnu, `expect' and Tcl, and
     exits without running any tests.

`-D0'
`-D1'
     Start the internal Tcl debugger.  The Tcl debugger supports
     breakpoints, single stepping, and other common debugging
     activities.  (See `A Debugger for Tcl Applications' by Don Libes.
     (2))

     If you specify `-D1', the `expect' shell stops at a breakpoint as
     soon as DejaGnu invokes it.

     If you specify `-D0', DejaGnu starts as usual, but you can enter
     the debugger by sending an interrupt (e.g. by typing <C-c>).

   ---------- Footnotes ----------

   (1) Configuration triples have the form `CPU-VENDOR-OS'.

   (2) Distributed in PostScript form with `expect' as the file
`expect/tcl-debug.ps'.


File: dejagnu.info,  Node: Customizing,  Next: Internals,  Prev: Invoking runtest,  Up: Top

Setting `runtest' defaults
**************************

   The site configuration file, `site.exp', captures
configuration-dependent values and propagates them to the DejaGnu test
environment using Tcl variables.  This ties the DejaGnu test scripts
into the `configure' and `make' programs.

   DejaGnu supports more than one `site.exp' file.  The multiple
instances of `site.exp' are loaded in a fixed order built into DejaGnu
(the more local last). The first file loaded is the optional
`~/.dejagnurc', then the local files, and finally the global file.

  1. There is am optional "master" `site.exp', capturing configuration
     values that apply to DejaGnu across the board, in each
     configuration-specific subdirectory of the DejaGnu library
     directory.  `runtest' loads these values first. *Note Configuring
     and Installing DejaGnu: Installation.  The master `site.exp'
     contains the default values for all targets and hosts supported by
     DejaGnu. This master file is identified by setting the environment
     variable `DEJAGNU' to the name of the file. This is also refered
     to as the "global" config file.

  2. Any directory containing a configured test suite also has a
     `site.exp', capturing configuration values specific to the tool
     under test.  Since `runtest' loads these values last, the
     individual test configuration can either rely on and use, or
     override, any of the global values from the "master" `site.exp'.

     You can usually generate or update the testsuite `site.exp' by
     typing `make site.exp' in the test suite directory, after the test
     suite is configured.

  3. You can also have a file in your home directory called
     `.dejagnurc'. This gets loaded first before the other config
     files. Usually this is used for personal stuff, like setting
     `all_flag' so all the output gets printed, or verbosity levels.

   You can further override the default values in a user-editable
section of any `site.exp', or by setting variables on the `runtest'
command line.

* Menu:

* Config Values::               Variables used in the configuration file.
* Master Config File::          The master configuration file.
* Local Config File::           The local configuration file.
* Personal Config File::        The personal configuration file.


File: dejagnu.info,  Node: Config Values,  Next: Master Config File,  Up: Customizing

Config Variables
----------------

   DejaGnu uses a named array in Tcl to hold all the info for each
machine. In the case of a canadian cross, this means host information as
well as target information. The named array is called `target_info',
and it has two indices. The following fields are part of the array.

`name'
     The name of the target. (mostly for error messages) This should
     also be the string used for this target's array.  It should also
     be the same as the linker script so we can find them dynamically.
     This should be the same as the argument used for `push_target{}'.

`ldflags'
     This is the linker flags required to produce a fully linked
     executable. For `libgloss' supported targets this is usually just
     the name of the linker script.

`config'
     The target canonical for this target. This is used by some init
     files to make sure the target is supported.

`cflags'
     The flags required to produce an object file from a source file.

`connect'
     This is the connectmode for this target. This is for both IP and
     serial connections. Typically this is either `telnet', `rlogin',
     or `rsh'.

`target'
     This is the hostname of the target. This is for TCP/IP based
     connections, and is also used for version of tip that use
     /etc/remote.

`serial'
     This is the serial port. This is typically /dev/tty? or com?:.

`netport'
     This is the IP port. This is commonly used for telneting to target
     boards that are connected to a terminal server. In that case the
     IP port specifies the which serial port to use.

`baud'
     This is the baud rate for a serial port connection.

`x10'
     This is the parameters for an x10 controller. These are simple
     devices that let us power cycle or reset a target board remotely.

`fileid'
     This is the fileid or spawn id of of the connection.

`prompt'
     a glob style pattern to recognize the prompt.

`abbrev'
     abbreviation for tool init files.

`ioport'
     This is the port for I/O on dual port systems. In this
     configuration, the main serial port `0' is usually used for stdin
     and stdout, which the second serial port can be used for debugging.

   The first index into the array is the same value as used in the
`name' field. This is usually a short version of the name of the target
board. For an example, here's the settings I use for  my `Motorola's'
`IDP' board and my `Motorola' 6U VME `MVME135-1' board. (both m68k
targets)

     # IDP board
     set target_info(idp,name)       "idp"
     set target_info(idp,ldflags)    "-Tidp.ld"
     set target_info(idp,config)     m68k-unknown-aout
     set target_info(idp,cflags)     ""
     set target_info(idp,connect)    telnet
     set target_info(idp,target)     "s7"
     set target_info(idp,serial)     "tstty7"
     set target_info(idp,netport)    "wharfrat:1007"
     set target_info(idp,baud)       "9600"
     # MVME 135 board
     set target_info(idp,name)       "mvme"
     set target_info(idp,ldflags)    "-Tmvme.ld"
     set target_info(idp,config)     m68k-unknown-aout
     set target_info(idp,cflags)     ""
     set target_info(idp,connect)    telnet
     set target_info(idp,target)     "s8"
     set target_info(idp,serial)     "tstty8"
     set target_info(idp,netport)    "wharfrat:1008"
     set target_info(idp,baud)       "9600"

   DejaGnu can use this information to switch between multiple targets
in one test run. This is done through the use of the `push_target'
procedure, which is discussed elsewhere.

   This array can also hold information for a remote host, which is used
when testing a candain cross. In this case, the only thing different is
the index is just `host'. Here's the settings I use to run tests on my
NT machine while running DejaGnu on a Unix machine. (in this case a
Linux box)

     set target_info(host,name)      "nt-host"
     set target_info(host,config)    "386-unknown-winnt"
     set target_info(host,connect)   "telnet"
     set target_info(host,target)    "ripple"

   There is more info on how to use these variables in the sections on
the config files. *Note Configuration Files: Master Config File.

   In the user editable second section of `site.exp', you can not only
override the configuration variables captured in the first section, but
also specify default values for all the `runtest' command line options.
Save for `--debug', `--help', and `--version', each command line
option has an associated Tcl variable.  Use the Tcl `set' command to
specify a new default value (as for the configuration variables).  The
following table describes the correspondence between command line
options and variables you can set in `site.exp'.  *Note Running the
Tests: Invoking runtest, for explanations of the command-line options.

     runtest      Tcl
     option     variable     description
     __________ ________     ___________________________________________
     
     --all      all_flag     display all test results if set
     
     --baud     baud         set the default baud rate to something other
                             than 9600.
     --connect  connectmode  `rlogin', `telnet', `rsh',
                             `kermit', `tip', or `mondfe'
     
     --outdir   outdir       directory for `TOOL.sum' and `TOOL.log'
     
     --objdir   objdir       directory for pre-compiled binaries
     
     --reboot   reboot       reboot the target if set to `"1"';
                             do not reboot if set to `"0"' (the default)
     
     --srcdir   srcdir       directory of test subdirectories
     
     --strace   tracelevel   a number: Tcl trace depth
     
     --tool     tool         name of tool to test; identifies init, test subdir
     
     --verbose  verbose      verbosity level.  As option, use multiple times;
                             as variable, set a number, 0 or greater
     --target   target_triplet The canonical configuration string for the target.
     --host     host_triplet The canonical configuration string for the host.
     --build    build_triplet The canonical configuration string for the
                             build host.


File: dejagnu.info,  Node: Master Config File,  Next: Local Config File,  Prev: Config Values,  Up: Customizing

Master Config File
------------------

   The master config file is where all the target specific config
variables get set for a whole site get set. The idea is that for a
centralized testing lab where people have to share a target between
multiple developers. There are settings for both remote targets and
remote hosts.  Here's an example of a Master Config File (also called
the Global config file) for a _canadian cross_. A canadian cross is
when you build and test a cross compiler on a machine other than the
one it's to be hosted on.

   Here we have the config settings for our California office. Note that
all config values are site dependant. Here we have two sets of values
that we use for testing m68k-aout cross compilers. As both of these
target boards has a different debugging protocol, we test on both of
them in sequence.

     global CFLAGS
     global CXXFLAGS
     
     case "$target_triplet" in {
         { "native" } {
             set target_abbrev unix
         }
         { "m68*-unknown-aout" } {
             set target_abbrev               "rom68k"
             # IDP target                    # IDP board with rom68k monitor
             set target_info(idp,name)       "idp"
             set target_info(idp,ldflags)    "-Tidp.ld"
             set target_info(idp,config)     m68k-unknown-aout
             set target_info(idp,cflags)     ""
             set target_info(idp,connect)    telnet
             set target_info(idp,target)     "s7"
             set target_info(idp,serial)     "tstty12"
             set target_info(idp,netport)    "truckin:1007"
             set target_info(idp,baud)       "9600"
             # MVME target                   # Motorola MVME 135 with BUG monitor
             set target_info(mvme,name)      "mvme"
             set target_info(mvme,ldflags)   "-Tmvme.ld"
             set target_info(mvme,config)    m68k-unknown-aout
             set target_info(mvme,cflags)    ""
             set target_info(mvme,connect)   telnet
             set target_info(mvme,target)    "s4"
             set target_info(mvme,serial)    "tstty8"
             set target_info(mvme,netport)   "truckin:1004"
             set target_info(mvme,baud)      "9600"
          }
     }

   In this case, we have support for several remote hosts for our
m68k-aout cross compiler. Typically the remote Unix hosts run DejaGnu
locally, but we also use them for debugging the testsuites when we find
problems in running on remote hosts. Expect won't run on NT, so DejaGnu
is run on the local build machine, and it'll connect to the NT host and
run all the tests for this cross compiler on that host.

     case "$host_triplet" in {
         "native" {
         }
         "i?86-*-linux*" {                   # Linux host
             set target_info(host,name)      "linux-host"
             set target_info(host,config)    $host_triplet
             set target_info(host,connect)   rlogin
             set target_info(host,target)    chinadoll
         }
         "i?86-*-winnt                       # NT host
             set target_info(host,name)      "nt-host"
             set target_info(host,config)    i386-unknown-winnt
             set target_info(host,connect)   telnet
             set target_info(host,target)    ripple
         }
         "hppa*-hp-hpux*" {                  # HP-UX host
             set target_info(host,name)      "hpux-host"
             set target_info(host,config)    $host_triplet
             set target_info(host,connect)   rlogin
             set target_info(host,target)    slipknot
             }
         "sparc-sun-sunos*" {                # SunOS (sun4)
             set target_info(host,name)      "sunos-host"
             set target_info(host,config)    $host_triplet
             set target_info(host,connect)   rlogin
             set target_info(host,target)    darkstar
         }
     }


File: dejagnu.info,  Node: Local Config File,  Next: Personal Config File,  Prev: Master Config File,  Up: Customizing

Local Config File
-----------------

   It is usually more convenient to keep these "manual overrides" in the
`site.exp' local to each test directory, rather than in the "master"
`site.exp' in the DejaGnu library.

   All local `site.exp' usually files have two sections, separated by
comment text. The first section is the part that is generated by
`make'. It is essentially a collection of Tcl variable definitions
based on `Makefile' environment variables. Since they are generated by
`make', they contain the values as specified by `configure'.  (You can
also customize these values by using the `--site' option to
`configure'.)  In particular, this section contains the `Makefile'
variables for host and target configuration data. Do not edit this
first section; if you do, your changes are replaced next time you run
`make'.

   The first section starts with:

     ## these variables are automatically generated by make ##
     # Do not edit here. If you wish to override these values
     # add them to the last section

   In the second section, you can override any default values (locally
to DejaGnu) for all the variables.  The second section can also contain
your preferred defaults for all the command line options to `runtest'.
This allows you to easily customize `runtest' for your preferences in
each configured test-suite tree, so that you need not type options
repeatedly on the command line.  (The second section may also be empty,
if you do not wish to override any defaults.)

   The first section ends with this line:

     ## All variables above are generated by configure. Do Not Edit ##

   You can make any changes under this line. If you wish to redefine a
variable in the top section, then just put a duplicate value in this
second section. Usually the values defined in this config file are
related to the configuration of the test run. This is the ideal place to
set the variables `host_triplet', `build_triplet', `target_triplet'.
All other variables are tool dependant. ie for testing a compiler, the
value for CC might be set to a freshly built binary, as opposed to one
in the user's path.


File: dejagnu.info,  Node: Personal Config File,  Prev: Local Config File,  Up: Customizing

Personal Config File
--------------------

   The personal config file is used to customize `runtest's' behaviour
for each person. It's typically used to set the user prefered setting
for verbosity, and any experimental Tcl procedures. My personal
`~/.dejagnurc' file looks like:

     set all_flag 1
     set RLOGIN /usr/ucb/rlogin
     set RSH /usr/ucb/rsh

   Here I set `all_flag' so I see all the test cases that PASS along
with the ones that FAIL. I also set RLOGIN and `RSH' to the BSD
version. I have `kerberos' installed, and when I rlogin to a target
board, it usually isn't supported. So I use the non secure versions of
these programs rather than the default that's in my path.


File: dejagnu.info,  Node: Internals,  Next: Tests,  Prev: Customizing,  Up: Top

The DejaGnu Implementation
**************************

   DejaGnu is entirely written in `expect', which uses Tcl as a command
language.  `expect' serves as a very programmable shell; you can run
any program, as with the usual Unix command shells--but once the
program is started, your `expect' script has fully programmable control
of its input and output.  This does not just apply to the programs
under test; `expect' can also run any auxiliary program, such as `diff'
or `sh', with full control over its input and output.

   DejaGnu itself is merely a framework for the set of test suites
distributed separately for each GNU tool.  Future releases of GNU tools
will include even more tests, developed throughout the free software
community.

   `runtest' is the glue to tie together and manage the test scripts.
The `runtest' program is actually a simple Bourne shell script that
locates a copy of the `expect' shell and then starts the main Tcl code,
`runtest.exp'. `runtest.exp' itself has these essential functions:

  1. Parse the command line options, load the library files, and load
     the default configuration files.

  2. Locating the individual test scripts.  `runtest.exp' locates the
     tests by exploiting a straightforward naming convention based on
     the string you specify with the `--tool' option.

  3. Providing an extended test environment, by defining additional Tcl
     procedures beyond those already in `expect'.

  4. Locating target-dependent functions, to standardize the test
     environment across a wide variety of test platforms.

* Menu:

* Names::                       Conventions for using tool names
* Init Module::                 Initialization module
* DejaGnu Builtins::            DejaGnu provides these Tcl procedures
* Target Dependent::            Procedures supplied by the init module
* Cross Targets::               Remote targets supported
* Input Files::                 The files DejaGnu depends on
* Output Files::                The files DejaGnu produces