Subversion Repositories or1k

        1998

        Free Software Foundation, Inc.">

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This file documents the GNU Testing Framework ``DejaGnu''

Copyright (C) 92, 93, 94, 95, 96, 97, 98, 1999 Free Software Foundation, Inc.

This text may be freely distributed under the terms of the GNU

General Public License.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

">

Copyright 92, 93, 94, 95, 96, 97, 98, 1999 Free Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of

this manual provided the copyright notice and this permission notice

are preserved on all copies.

Permission is granted to copy and distribute modified versions of this

manual under the conditions for verbatim copying, provided also that

the entire resulting derived work is distributed under the terms of a

permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual

into another language, under the above conditions for modified versions.

">

]>

    The GNU Testing Framework

    1998 Nov 24

     &version

     for circulation within Cygnus

        Rob Savoye

          Free Software Foundation

      rob@welcomehome.org

    &dejagnu-copyright;

         0.1

        1998-11

        rob@welcomehome.org

        Initial version after conversion to DocBook.

    This document attempts to describe the functionality of

    DejaGnu, the GNU Testing Framework. DejaGnu is entirely written in

    Expect, which uses

    Tcl as a command

    language. Expect serves as a very

    programmable shell; you can run any program, as with the usual

    Unix command shells---but once the program is started, your

    test script has fully programmable control of

    its input and output.  This does not just apply to the programs

    under test; expect can also run any auxiliary

    program, such as diff or sh,

    with full control over its input and output.

    DejaGnu itself is merely a framework for creation of a test

    suites. Test suites are distributed separately for each GNU

    tool.

    DejaGnu is a framework for

        testing other programs.  Its purpose is to provide a single

        front end for all tests. Think of it as a custom library of

        Tcl procedures crafted to support writing a test harness. A

        Test Harness is the testing

        infrastructure that is created to support a specific program

        or tool. Each program can have multiple test suites, all

        supported by a single test harness. DejaGnu is written in

        Expect, which in turn uses

        Tcl -- Tool command

        language. There is more information on Tcl at the 

        URL="http://www.scriptics.com">Scriptics web site, and the

        Expect web site is at 

        URL="http://expect.nist.gov">NIST.

        DejaGnu offers several advantages for testing:

      The flexibility and consistency of the DejaGnu

        framework make it easy to write tests for any program, with

        either batch oriented, or interactive programs.

      DejaGnu provides a layer of abstraction which

        allows you to write tests that are portable to any host or

        target where a program must be tested. For instance, a test

        for GDB can run (from any Unix

        based host) on any target architecture that DejaGnu

        supports. Currently DejaGnu runs tests on many single board

        computers, whose operating software ranges from just a boot

        monitor to a full-fledged, Unix-like realtime OS.

      All tests have the same output format. This

        makes it easy to integrate testing into other software

        development processes. DejaGnu's output is designed to be

        parsed by other filtering script, and it is also human

        readable.

      Using Tcl and expect, it's easy to create wrappers

      for existing test suites. By incorporating existing tests under

      DejaGnu, it's easier to have a single set of report analyse

      programs..

    Running tests requires two things: the testing framework, and

    the test suites themselves. Tests are usually written in

    Expect using Tcl, but you can also use a

    Tcl script to run a test suite that is not based on

    Expect.

    (expect script filenames conventionally

    use .exp as a suffix; for example, the main

    implementation of the DejaGnu test driver is in the file

    runtest.exp.)

    Julia Menapace first coined the term ``Deja Gnu'' to describe an

    earlier testing framework at Cygnus Support she had written for

    GDB. When we replaced it with the Expect-based

    framework, it was like DejaGnu all over again... But more importantly, it

    was also named after my daughter,

    URL="mailto:deja@welcomehome.org">Deja Snow Savoye (now 9

    years old in Dec of 1998), who was a toddler during DejaGnu's

    creation.

    This release has a number of substantial changes over version

    1.3. The most visible change is that the version of Expect and Tcl

    included in the release are up-to-date with the current stable net

    releases. The biggest change is years of modifications to the

    target configuration system, used for cross testing. While this

    greatly improved cross testing, is has made that subsystem very

    complicated. The goal is to have this entirely rewritten using

    iTcl by the next release. Other changes

    are:

      More builtin support for building target binaries

        with the correct linker flags. Currently this only works with

        GCC as the cross compiler,

        preferably with a target supported by

        .

      Lots of little bug fixes from years of heavy

        use at Cygnus Solutions.

      DejaGnu now uses

      Automake for Makefile

      configuration.

      Updated documentation, now in SGML

        (using the 

        URL="http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro.html">free

        GNU DocBook tools) format.

      NT support. There is beta level support for NT

      that is still a work in progress. This requires the 

      URL=httpd://sourceware.cygnus.com>Cygwin POSIX system

      for NT.

      To use DejaGnu on NT, you need to first install the

        Cygwin

        release. This works as of the B20.1 release. Cygwin is a POSIX

        system for NT. This covers both utility programs, and a libray

        that adds POSIX system calls to NT. Among them is pseudo tty

        support for NT that emulates the POSIX pty standard. The

        latest Cygwin is always available from 

        URL="http://sourceware.cygnus.com">this location. This

        works well enough to run "make check" of

        the GNU development tree on NT after a native build. But the

        nature of pty's on NT is still evolving. Your mileage may

        vary...

    DejaGnu grew out of the internal needs of Cygnus Solutions. (then

    Cygnus Support). Cygnus maintains and enhances a variety of free programs

    in many different environments, and we needed a testing tool that:

      is useful to developers while fixing bugs.

      automates running many tests during a software

      release process.

      is portable among a variety of host

      computers.

      supports cross-development testing.

      permits testing interactive programs, like

      GDB; and 

      permits testing batch oriented programs, like

      GCC.

    Some of the requirements proved challenging.  For example,

    interactive programs do not lend themselves very well to automated testing.

    But all the requirements are important: for instance, it is imperative to

    make sure that GDB works as well when cross-debugging

    as it does in a native configuration. 

    Probably the greatest challenge was testing in a cross-development

    environment (which can be a real nightmare).  Most cross-development

    environments are customized by each developer.  Even when buying packaged

    boards from vendors there are many differences.  The communication

    interfaces vary from a serial line to ethernet.  DejaGnu was designed with

    a modular communication setup, so that each kind of communication can be

    added as required, and supported thereafter.  Once a communication

    procedure is coded, any test can use it.  Currently DejaGnu can use

    rsh, rlogin,

    telnet, tip,

    kermit, and mondfe for remote

    communications.

      DejaGnu conforms to the POSIX 1003.3 standard for test

      frameworks. I was also a member of that committe.

      The POSIX standard 1003.3 defines what a testing framework needs to

      provide, in order to permit the creation of POSIX conformance test

      suites. This standard is primarily oriented to running POSIX conformance

      tests, but its requirements also support testing of features not related

      to POSIX conformance.  POSIX 1003.3 does not specify a particular testing

      framework, but at this time there is only one other POSIX conforming test

      framework: TET. TET was created by Unisoft for a consortium comprised of

      X/Open, Unix International, and the Open Software Foundation.

      The POSIX documentation refers to assertions.

      An assertion is a description of behavior.  For example, if a standard

      says ``The sun shall shine'', a corresponding assertion might be ``The

      sun is shining.''  A test based on this assertion would pass or fail

      depending on whether it is daytime or nighttime.  It is important to note

      that the standard being tested is never 1003.3; the standard being tested

      is some other standard, for which the assertions were written.

      As there is no test suite to test testing frameworks for POSIX

      1003.3 conformance, verifying conformance to this standard is done by

      repeatedly reading the standard and experimenting.  One of the main

      things 1003.3 does specify is the set of allowed output messages, and

      their definitions.  Four messages are supported for a required feature of

      POSIX conforming systems, and a fifth for a conditional feature. DejaGnu

      supports the use of all five output messages; in this sense a test suite

      that uses exactly these messages can be considered POSIX conforming.

      These definitions specify the output of a test case:

          PASS

          A test has succeeded.  That is, it demonstrated that

          the assertion is true.

          XFAIL

          POSIX 1003.3 does not incorporate the notion of

          expected failures, so PASS, instead of

          XPASS, must also be returned for test cases

          which were expected to fail and did not.  This means that

          PASS is in some sense more ambiguous than if

          XPASS is also used.

          FAIL

          A test has produced the bug it was intended to

          capture.  That is, it has demonstrated that the assertion is false.

          The FAIL message is based on the test case only.

          Other messages are used to indicate a failure of the framework. As

          with PASS, POSIX tests must return

          FAIL rather than XFAIL even

          if a failure was expected.

          UNRESOLVED

          A test produced indeterminate results.  Usually, this

          means the test executed in an unexpected fashion; this outcome

          requires that a human being go over results, to determine if the test

          should have passed or failed.  This message is also used for any test

          that requires human intervention because it is beyond the abilities

          of the testing framework.  Any unresolved test should resolved to

          PASS or FAIL before a test

          run can be considered finished.

          Note that for POSIX, each assertion must produce a test result

          code.  If the test isn't actually run, it must produce

          UNRESOLVED rather than just leaving that test

          out of the output.  This means that you have to be careful when

          writing tests, to not carelessly use tcl statements like

          return---if you alter the flow of control of the

          tcl code you must insure that every test still produces some result

          code.

          Here are some of the ways a test may wind up

          UNRESOLVED:

            A test's execution is interrupted.

            A test does not produce a clear result. This is

            usually because there was an ERROR from

            DejaGnu while processing the test, or because there were three or

            more WARNING messages. Any

            WARNING or ERROR messages

            can invalidate the output of the test.  This usually requires a

            human being to examine the output to determine what really

            happened---and to improve the test case.

            A test depends on a previous test, which

            fails.

            The test was set up incorrectly.

            UNTESTED

            A test was not run.  This is a placeholder, used

            when there is no real test case yet.

          The only remaining output message left is intended to test

          features that are specified by the applicable POSIX standard as

          conditional:

            UNSUPPORTED

            There is no support for the tested case.  This may

            mean that a conditional feature of an operating system, or of a

            compiler, is not implemented.  DejaGnu also uses this message when

            a testing environment (often a ``bare board'' target) lacks basic

            support for compiling or running the test case.  For example, a

            test for the system subroutine gethostname

            would never work on a target board running only a boot monitor.

        DejaGnu uses the same output procedures to produce these messages

        for all test suites, and these procedures are already known to conform

        to POSIX 1003.3.  For a DejaGnu test suite to conform to POSIX 1003.3,

        you must avoid the setupxfail} procedure as

        described in the PASS section above, and you must

        be careful to return UNRESOLVED where appropriate,

        as described in the UNRESOLVED section

        above.

  &user;

  &ref;