Subversion Repositories or1k

                        The Insight Testsuite
                        ---------------------
                   Keith Seitz (keiths@cygnus.com)
                             May 1, 2001

RUNNING THE TESTSUITE

The Insight testsuite is run in much the same way that gdb's testsuites
are run. The one big difference is the environment variable GDB_DISPLAY,
which governs what display should be used for the tests.

When GDB_DISPLAY is not set in the user's environment, the Insight testsuite
will attempt to run Xvfb, an X server with a virtual frame buffer. Using
Xvfb, the testsuite can run without interuppting the user.

When Xvfb is not available, the testsuite will mark the Insight tests
"untested" and print out some appropriate warning to the testsuite log
file.

If GDB_DISPLAY is set in the user's environment, the testsuite will attempt
to use this display for the tests. If this display is a desktop display,
it is very likely that any interaction between the user and his desktop
will interfere with the tests. Some tests warp the cursor, i.e., they
force the mouse to move on the screen. If you don't want this to happen to
you, put Xvfb in your path.

On Cygwin systems, Xvfb is not supported. Only two choices are available in
this environment: run the testsuites using the desktop or do not run the
testsuite. To run the testsuite on Cygwin, just define the environment
variable GDB_DISPLAY to anything.

The examples below summarize the usage of the environment variable GDB_DISPLAY
on unix/X-Windows hosts and Cygwin hosts. In all examples, assume that DISPLAY
set to the local workstation's main display (:0).

To run the testsuite using Xvfb -- unix only (Xvfb must be in PATH):
$ make check

To run the testsuite using a given display (either the desktop or a peviously
started Xvfb):
$ GDB_DISPLAY=$DISPLAY make check

To run the testsuite on Cygwin:
$ GDB_DISPLAY=foo make check


TESTSUITE INFRASTRUCTURE

The rest of this document deals with writing tests for Insight. This reading
is only noteworthy for developers and contributors.

The Insight testsuite consists of two large portions of code: code which is
run in dejagnu and code which runs in Insight's built-in Tcl interpreter. Files
containing dejagnu code (those files ending in ".exp" in the testsuite directory)
are "glue code" between gdb's dejagnu testsuite and Insight's Tcl testsuite.

Dejagnu testsuite files are considered "drivers" for any particular set of
tests, since they allow dejagnu to control Insight's Tcl testsuite.


Dejagnu Testsuite Infrastructure

The dejagnu code is responsible for doing several things. Some of the more
important responsibilities include:

o Initializing the display
o Determining if tests should be run
o Results accounting
o Compiling testcases in various languages
o Repoting results to gdb's testsuite

There are various functions defined to facilitate the writing of tests. These
functions currently reside in gdb's gdb.exp (src/gdb/testsuite/lib/gdb.exp) and
include:

Pulic functions:
proc gdbtk_initialize_display {}

  gdbtk_initialize_display should be the first function called from the
  (dejagnu) test file. It initializes the DISPLAY variable on unix systems
  and determines (for all host systems) whether or not the testsuite should
  run. It returns 1 if the test should run. If tests should not run, it
  marks the test as "untested" and leaves a suitable message about why
  the test should not run. If gdbtk_initialize_display returns zero, a test
  should simply exit.

proc gdbtk_start {test}

  This function marks the start of a test and will execute Insight for
  testing. The TEST parameter should be the file name of the Tcl test
  file to load into Insight's Tcl interpreter. It returns a list of
  test results suitable for passing to gdbtk_done or gdbtk_analyze_results.
  See gdbtk_analyze_results for more information on the format of results.

  gdbtk_start is responsible for communicating target startup information
  to Insight, so that Insight's testsuite may be run on any target supported
  by gdb. It does this by setting several environment variables just before
  it starts Insight. These environment variables are:

  OBJDIR
    The object directory of the dejagnu testsuite (i.e.,
    objdir/gdb/testsuite).
  SRCDIR
    The dejagnu source directory in which the tests are located (i.e,
    src/gdb/testsuite)
  SUBDIR
    The dejagnu testsuite subdirectory for the test (i.e., gdb.gdbtk)
  GDBTK_LIBRARY
    The location of the Insight tcl library (i.e., src/gdb/gdbtk/library)
  TCL_LIBRARY
    The location of the Tcl library (i.e., src/tcl/library)
  TK_LIBRARY
    The location of the Tk library (i.e., src/tk/library)
  TIX_LIBRARY
    The location of the Tix library (i.e., src/tix/library)
  ITCL_LIBRARY
    The location fo the Itcl librarry (i.e., src/itcl/itcl/library)
  CYGNUS_GUI_LIBRARY
    The location of libgui (i.e., src/libgui/library)
  DEFS
    The location of the testsuite definitions file (i.e.,
    src/gdb/testsuite/gdb.gdbtk/defs)

  Note that *_LIBRARY and DEFS are converted to absolute tcl-style
  paths. On unix, this means that GDBTK_LIBRARY would point to, for example,
  /home/keiths/insight/src/gdb/gdbtk/library. On Cygwin it would point to
  C:/cygwin/home/keiths/insight/src/gdb/gdbtk/library. This is because of
  a descrepency between Cygwin's posix paths and Tcl's not-quite-posix
  paths.

proc gdbtk_analyze_results {results}
  This function translates the list of results in RESULTS into dejagnu
  results, reporting the number of failures, errors, passes, and expected
  failures and passes. It currently does not deal with "untested" and other
  test statuses from dejagnu since Insight's tcl testsuite does not
  issue such results.

  The format of the results expected by gdbtk_analyze_results is simple:
  it is a list of {status name description msg}. "status" is the execution
  status of one of the tcl tests run. This can be "PASS", "FAIL", "ERROR",
  "XFAIL", or "XPASS".

  "name" is the name of the test, and it is reported in all testsuite
  results in dejagnu. This speeds location of the failing test. This
  "name" may also be given to Insight's testsuite, telling it to
  only run this test. See "do_tests" in Tcl Testsuite Infrastructure
  for more information.

  "description" is a textual description of the test given by "name".

  "msg" is currently not used.

proc gdbtk_done {{results {}}}
  gdbtk_done takes any RESULTS and passes it gdbtk_analyze_results for
  outputting to the dejagnu part of the testsuite. It may be called
  without an argument, in which case it will only terminate Xvfb if it
  was started. Tests must call gdbtk_done _once_ at the end of their
  test drivers.

Private functions:
proc _gdbtk_export_target_info
  This functin exports information about the target into the environment
  so that Insight's testsuite may run programs on any supported gdb
  target. This target information is passed into the Tcl testsuite
  via an environment variable, TARGET_INFO, which is really a Tcl array,
  i.e., the array is constructed in tcl and exported into the environment
  with Tcl's "array get" command).

  There are four elements to the array:
  TARGET_INFO(init)   - (optional) A list of commands to execute in gdb
                        to initialize the session. This usually includes
                        setting baud rates and remote protocol options.
  TARGET_INFO(target) - (required) The complete "target" command to connect
                        to the given target.
  TARGET_INFO(load)   - (optional) The complete "load" command to load an
                        executable into a target.
  TARGET_INFO(run)    - (required) The complete "run" command, sans arguments,
                        to start a process on the target. For remote targets,
                        this is usually just "continue".
  
proc _gdbtk_xvfb_init
  This procedure actually determines whether the an Insight test should
  run and what DISPLAY it should use for that test. It is called by
  gdbtk_initialize_display to do most of the dirty work.

  It has a simple heuristic: If GDB_DISPLAY is not set and Xvfb is available
  (on unix), it starts Xvfb, using the current process id as the screen number.
  If Xvfb is not available and GDB_DISPLAY was not set, it skips the tests.

proc _gdbtk_xvfb_exit
  _gdbtk_xvfb_exit will kill any previously started Xvfb.

Private globals:
global _xvfb_spawn_id
  This variable holds the spawn_id of any Xvfb process started
  by the testsuite (or it is left undefined).

global _using_windows
  A global variable which indicates whether the testsuite is running
  on cygwin. Unfortunately, as of writing, the global variable
  tcl_platform(platform) is "unix" on Cygwin, so it is not possible
  to rely on this for platform-dependent operations.

  Instead, this variable is set by gdbtk_initialize_display. The test
  it uses to determine if Cygwin is being used: it looks for the program
  cygpath in the PATH. Therefore, cygpath is REQUIRED to run the testsuite
  on Cygwin. (gdbtk_start also uses cygpath to determine Windows
  pathnames for Cygwin.)


Testsuite Driver Basics

Given the above interfaces for connecting Insight's Tcl testsuite and
gdb's dejagnu testsuite, the basic testsuite driver file should look
(minimally) like this:

File: mytest.exp
1   if {[gdbtk_initialize_display]} {
2     # We found a display to use
3     gdb_exit;   # Make sure any previous gdb is gone
4     set results [gdbtk_start mytest.test]
5
6     # Done!
7     gdbtk_done [split $results \n]
8   }

Line 1 calls gdbtk_initialize_display to ascertain whether there is a display
to use for the test. This could use an existing display (if GDB_DISPLAY is
set in the environment) or gdbk_initialize_display could startup an Xvfb
for use by the testsuite.

Line 3 forces any previously executing gdb to terminate.

Line 4 signals the start of the test run. "mytest.test" is the name of the
Tcl test file to execute in Insight's built-in Tcl interpreter. The output
of gdbtk_start_test is all of the results of the Tcl test from Insight, which
is subsequently passed to gdbk_analyze_results via gdbtk_done on Line 7.

Note how nothing happens if gdbtk_initialize_display returns false.


Tcl Testsuite Infrastructure

The heart of Insight's testsuite is its Tcl testsuite. It is these tests
which run directly in Insight's Tcl interpreter and allow test writers
access to Insight's internals. Tcl testsuite files have the filename suffix
".test" to distinguish them from their driver files, which end in ".exp".

The design of the Insight Tcl testsuite parallels Tcl's testsuite. It has
many powerful features, including the ability to run ANY test in a given
Tcl test file. See the description of utility routines below for more
information about running any set of tests from a file.

The bulk of the code implementing the Tcl testsuite infrastructure in
Insight is contained in the testsuite definitions file, "defs", located
in src/gdb/testsuite/gdb.gdbtk. This file contains routines necessary
to write tests for Insight.

Public functions:
proc gdbtk_read_defs {}
  This function, located in Insight's core Tcl library, attempts to load
  the testsuite definitions file. If it fails, it will either pop up
  a dialog box with the error (if running interactively) or it will
  print the error to stderr and exit (if running non-interactively).

  If successful, it will return true.

proc gdbtk_test_file {filename}
  This function is used to load the file given by FILENAME into
  Insight. It will automatically append ".exe" to any FILENAME
  on Cygwin-hosted systems.

  If successful, it will load the given file into Insight and
  return the output of gdb's file command. It will call "error"
  if it was succesful, therefore all calls to "gdbtk_test_file"
  should be called using "catch".

  Test authors should not use "gdb_cmd {file FILENAME}" to load
  files into gdb unless they are testing interface code between
  gdb and Insight.

proc gdbtk_test_run {{prog_args {}}}
  gdbtk_test_run runs the previoiusly loaded executable, passing
  the given arguments to the inferior. Like Insight's Run button,
  it will do whatever is necessary to get the executable running,
  including any target initialization (setting baud rate and remote
  protocol options), downloading the executable to the target, and
  finally starting execution.

  Test authors should NEVER use "gdb_cmd {run ARGUMENTS}" to run an
  executable. Doing so will insure that your tests will only run on
  a native debugger.

  It returns true if successful or false otherwise. It will report
  the error in a dialog box (if running interactively) or it will
  print the error to stderr.

proc gdbtk_test {name description script answer}
  This is Tcl testsuite equivalent of "expect". "name" is a canonical
  name of the test, usually of the form "shortname-major.minor". This is
  the name that is used when running selected tests from a given file.
  If "name" starts with an asterisk (*), it designates that the test
  is expected to fail.

  "description" is a short textual description of the test to help
  humans understand what it does.

  "script" is the actual test script to run. The result of this script
  will be compared against "answer" to determine if the test passed
  or failed.

  It calls gdbtk_print_verbose to print out the results to the terminal
  (if running interactively) or to the log file.

proc gdbtk_test_done {}
  gdbtk_test_done is called at the very end of all tcl tests. It is used
  to exit Insight and return control back to the dejagnu driver which
  started the tests.
  
proc gdbtk_dotests {file args}
  Obsolete.

proc do_test {{file {}} {verbose {}} {tests {}}}
  This procedure is used to invoke the Insight test(s) given in FILE
  which match the regular expression(s) in TESTS. This is invoked
  from Insight's console window to run tests interactively.

  VERBOSE sets the verbosity of the test run. When set to one,
  the testsuite will report all results in human readable form.
  When set  greater than one, it will print out results as a list,
  i.e., for passing to gdbtk_analyze_results. If zero, it will only
  print errors and failures in human readable form.

Public global variables:
objdir   - The objdir from dejagnu. See gdbtk_start for more information.
srcdir   - The srcdir from dejagnu. See gdbtk_start for more information.
test_ran - Indicates whether the last test ran or not. See example below.

Private functions:
proc gdbtk_test_error {desc}
  An internal function used to report a framework error in the testsuite.
  "desc" is a description of the error. It calls gdbtk_test_done.

proc gdbtk_print_verbose {status name description script code answer}
  A helper procedure to gdbtk_test which prints out results to the terminal
  or the logfile (or both or none).

Private global variables:
_test - An array used by the testsuite internals.


Tcl Test Basics

Armed with the basic interface described above, it is possible to test Insight's
GUI. Please do not write tests which attempt to imitate a user (moving the
mouse and clicking buttons), unless there is no other way to test the functionality.

The basic test file (with one test) looks like this (nonsensical one):
File: mytest.exp
1  if {![gdbtk_read_defs]} {
2    break
3  }
4  
5  global objdir test_ran
6  set program [file join $objdir mytest]
7  if {[catch {gdbtk_test_file $program} t]} {
8    gdbtk_test_error "loading \"$program\": $t"
9  }
10 if {![gdbtk_test_run]} { exit 1 }
11
12 global foo
13 set foo 1
14
15 # Test:  mytest-1.1
16 # Desc:  check if a source window was created
17 gdbtk_test mytest-1.1 {source window created} {
18   set window [ManagedWin::find SrcWin]
19   llength $window
20   set foo 13
21 } {1}
22
23 if {$test_ran} {
24   set foo 1
25 }
26
27 # Done
28 gdbtk_test_done

Line 1 calls the Inisght function gdbtk_read_defs to read in the testsuite
definitions file.

Line 6 then specifies the name of a file (mytest) in the object directory
which is loaded into gdb on Line 7. If loading the file into Insight
failed, gdbtk_test_error is called to publish the error (and terminate the
test run for this file).

Line 10 runs the executable on the target, and exits if it was unable
to do so.

Line 13 simply sets a global variable foo to illustrate the purpose
of the global "test_ran". Before the test "mytest-1.1" runs, foo is set to
one. In order to support running specific tests, the state of the debugger
cannot be altered INSIDE gdbtk_test scripts, since the contents of the
script may not be run if the user requested only a specific test to run.

Line 20 in the middle of the test modifies the global foo. If subsequent
test relied on foo being one, we would have a state violation, since
mytest-1.1 may have (or may have not) run.

Therefore, we can check if a test ran and reset foo by checking the
global "test_ran". If set, we know that the previous test (mytest-1.1)
was run, and that foo is now thirteen. We reset the result back to one.

(Aside: Some tests do not follow this rule explicitly: they can assume
that all tests run sequentially. In these cases, running a specific
test in the file will probably fail, since the debugger is not brought
to a known state along the way.)

Lines 17-21 contain the actual test. The test's name is "mytest-1.1". It
is this name that may be referred to when asking the testsuite to run
a specific test. The description of this test is "source window created",
indicating that mytest-1.1's purpose is to check whether a source window
was created.

If gdbtk_test determines that this test is to run, it will execute the
next part, lines 18-20, and compare the output of that script (llength
$window) with "1". If the result is not "1", a failure is recorded. If
it is "1", a pass is recorded.

Finally, the test file is done and exits on line 28.