URL
https://opencores.org/ocsvn/or1k/or1k/trunk
Subversion Repositories or1k
[/] [or1k/] [trunk/] [insight/] [gdb/] [testsuite/] [gdb.gdbtk/] [README] - Rev 1767
Go to most recent revision | Compare with Previous | Blame | View Log
The Insight Testsuite
---------------------
Keith Seitz (keiths@cygnus.com)
May 1, 2001
RUNNING THE TESTSUITE
The Insight testsuite is run in much the same way that gdb's testsuites
are run. The one big difference is the environment variable GDB_DISPLAY,
which governs what display should be used for the tests.
When GDB_DISPLAY is not set in the user's environment, the Insight testsuite
will attempt to run Xvfb, an X server with a virtual frame buffer. Using
Xvfb, the testsuite can run without interuppting the user.
When Xvfb is not available, the testsuite will mark the Insight tests
"untested" and print out some appropriate warning to the testsuite log
file.
If GDB_DISPLAY is set in the user's environment, the testsuite will attempt
to use this display for the tests. If this display is a desktop display,
it is very likely that any interaction between the user and his desktop
will interfere with the tests. Some tests warp the cursor, i.e., they
force the mouse to move on the screen. If you don't want this to happen to
you, put Xvfb in your path.
On Cygwin systems, Xvfb is not supported. Only two choices are available in
this environment: run the testsuites using the desktop or do not run the
testsuite. To run the testsuite on Cygwin, just define the environment
variable GDB_DISPLAY to anything.
The examples below summarize the usage of the environment variable GDB_DISPLAY
on unix/X-Windows hosts and Cygwin hosts. In all examples, assume that DISPLAY
set to the local workstation's main display (:0).
To run the testsuite using Xvfb -- unix only (Xvfb must be in PATH):
$ make check
To run the testsuite using a given display (either the desktop or a peviously
started Xvfb):
$ GDB_DISPLAY=$DISPLAY make check
To run the testsuite on Cygwin:
$ GDB_DISPLAY=foo make check
TESTSUITE INFRASTRUCTURE
The rest of this document deals with writing tests for Insight. This reading
is only noteworthy for developers and contributors.
The Insight testsuite consists of two large portions of code: code which is
run in dejagnu and code which runs in Insight's built-in Tcl interpreter. Files
containing dejagnu code (those files ending in ".exp" in the testsuite directory)
are "glue code" between gdb's dejagnu testsuite and Insight's Tcl testsuite.
Dejagnu testsuite files are considered "drivers" for any particular set of
tests, since they allow dejagnu to control Insight's Tcl testsuite.
Dejagnu Testsuite Infrastructure
The dejagnu code is responsible for doing several things. Some of the more
important responsibilities include:
o Initializing the display
o Determining if tests should be run
o Results accounting
o Compiling testcases in various languages
o Repoting results to gdb's testsuite
There are various functions defined to facilitate the writing of tests. These
functions currently reside in gdb's gdb.exp (src/gdb/testsuite/lib/gdb.exp) and
include:
Pulic functions:
proc gdbtk_initialize_display {}
gdbtk_initialize_display should be the first function called from the
(dejagnu) test file. It initializes the DISPLAY variable on unix systems
and determines (for all host systems) whether or not the testsuite should
run. It returns 1 if the test should run. If tests should not run, it
marks the test as "untested" and leaves a suitable message about why
the test should not run. If gdbtk_initialize_display returns zero, a test
should simply exit.
proc gdbtk_start {test}
This function marks the start of a test and will execute Insight for
testing. The TEST parameter should be the file name of the Tcl test
file to load into Insight's Tcl interpreter. It returns a list of
test results suitable for passing to gdbtk_done or gdbtk_analyze_results.
See gdbtk_analyze_results for more information on the format of results.
gdbtk_start is responsible for communicating target startup information
to Insight, so that Insight's testsuite may be run on any target supported
by gdb. It does this by setting several environment variables just before
it starts Insight. These environment variables are:
OBJDIR
The object directory of the dejagnu testsuite (i.e.,
objdir/gdb/testsuite).
SRCDIR
The dejagnu source directory in which the tests are located (i.e,
src/gdb/testsuite)
SUBDIR
The dejagnu testsuite subdirectory for the test (i.e., gdb.gdbtk)
GDBTK_LIBRARY
The location of the Insight tcl library (i.e., src/gdb/gdbtk/library)
TCL_LIBRARY
The location of the Tcl library (i.e., src/tcl/library)
TK_LIBRARY
The location of the Tk library (i.e., src/tk/library)
TIX_LIBRARY
The location of the Tix library (i.e., src/tix/library)
ITCL_LIBRARY
The location fo the Itcl librarry (i.e., src/itcl/itcl/library)
CYGNUS_GUI_LIBRARY
The location of libgui (i.e., src/libgui/library)
DEFS
The location of the testsuite definitions file (i.e.,
src/gdb/testsuite/gdb.gdbtk/defs)
Note that *_LIBRARY and DEFS are converted to absolute tcl-style
paths. On unix, this means that GDBTK_LIBRARY would point to, for example,
/home/keiths/insight/src/gdb/gdbtk/library. On Cygwin it would point to
C:/cygwin/home/keiths/insight/src/gdb/gdbtk/library. This is because of
a descrepency between Cygwin's posix paths and Tcl's not-quite-posix
paths.
proc gdbtk_analyze_results {results}
This function translates the list of results in RESULTS into dejagnu
results, reporting the number of failures, errors, passes, and expected
failures and passes. It currently does not deal with "untested" and other
test statuses from dejagnu since Insight's tcl testsuite does not
issue such results.
The format of the results expected by gdbtk_analyze_results is simple:
it is a list of {status name description msg}. "status" is the execution
status of one of the tcl tests run. This can be "PASS", "FAIL", "ERROR",
"XFAIL", or "XPASS".
"name" is the name of the test, and it is reported in all testsuite
results in dejagnu. This speeds location of the failing test. This
"name" may also be given to Insight's testsuite, telling it to
only run this test. See "do_tests" in Tcl Testsuite Infrastructure
for more information.
"description" is a textual description of the test given by "name".
"msg" is currently not used.
proc gdbtk_done {{results {}}}
gdbtk_done takes any RESULTS and passes it gdbtk_analyze_results for
outputting to the dejagnu part of the testsuite. It may be called
without an argument, in which case it will only terminate Xvfb if it
was started. Tests must call gdbtk_done _once_ at the end of their
test drivers.
Private functions:
proc _gdbtk_export_target_info
This functin exports information about the target into the environment
so that Insight's testsuite may run programs on any supported gdb
target. This target information is passed into the Tcl testsuite
via an environment variable, TARGET_INFO, which is really a Tcl array,
i.e., the array is constructed in tcl and exported into the environment
with Tcl's "array get" command).
There are four elements to the array:
TARGET_INFO(init) - (optional) A list of commands to execute in gdb
to initialize the session. This usually includes
setting baud rates and remote protocol options.
TARGET_INFO(target) - (required) The complete "target" command to connect
to the given target.
TARGET_INFO(load) - (optional) The complete "load" command to load an
executable into a target.
TARGET_INFO(run) - (required) The complete "run" command, sans arguments,
to start a process on the target. For remote targets,
this is usually just "continue".
proc _gdbtk_xvfb_init
This procedure actually determines whether the an Insight test should
run and what DISPLAY it should use for that test. It is called by
gdbtk_initialize_display to do most of the dirty work.
It has a simple heuristic: If GDB_DISPLAY is not set and Xvfb is available
(on unix), it starts Xvfb, using the current process id as the screen number.
If Xvfb is not available and GDB_DISPLAY was not set, it skips the tests.
proc _gdbtk_xvfb_exit
_gdbtk_xvfb_exit will kill any previously started Xvfb.
Private globals:
global _xvfb_spawn_id
This variable holds the spawn_id of any Xvfb process started
by the testsuite (or it is left undefined).
global _using_windows
A global variable which indicates whether the testsuite is running
on cygwin. Unfortunately, as of writing, the global variable
tcl_platform(platform) is "unix" on Cygwin, so it is not possible
to rely on this for platform-dependent operations.
Instead, this variable is set by gdbtk_initialize_display. The test
it uses to determine if Cygwin is being used: it looks for the program
cygpath in the PATH. Therefore, cygpath is REQUIRED to run the testsuite
on Cygwin. (gdbtk_start also uses cygpath to determine Windows
pathnames for Cygwin.)
Testsuite Driver Basics
Given the above interfaces for connecting Insight's Tcl testsuite and
gdb's dejagnu testsuite, the basic testsuite driver file should look
(minimally) like this:
File: mytest.exp
1 if {[gdbtk_initialize_display]} {
2 # We found a display to use
3 gdb_exit; # Make sure any previous gdb is gone
4 set results [gdbtk_start mytest.test]
5
6 # Done!
7 gdbtk_done [split $results \n]
8 }
Line 1 calls gdbtk_initialize_display to ascertain whether there is a display
to use for the test. This could use an existing display (if GDB_DISPLAY is
set in the environment) or gdbk_initialize_display could startup an Xvfb
for use by the testsuite.
Line 3 forces any previously executing gdb to terminate.
Line 4 signals the start of the test run. "mytest.test" is the name of the
Tcl test file to execute in Insight's built-in Tcl interpreter. The output
of gdbtk_start_test is all of the results of the Tcl test from Insight, which
is subsequently passed to gdbk_analyze_results via gdbtk_done on Line 7.
Note how nothing happens if gdbtk_initialize_display returns false.
Tcl Testsuite Infrastructure
The heart of Insight's testsuite is its Tcl testsuite. It is these tests
which run directly in Insight's Tcl interpreter and allow test writers
access to Insight's internals. Tcl testsuite files have the filename suffix
".test" to distinguish them from their driver files, which end in ".exp".
The design of the Insight Tcl testsuite parallels Tcl's testsuite. It has
many powerful features, including the ability to run ANY test in a given
Tcl test file. See the description of utility routines below for more
information about running any set of tests from a file.
The bulk of the code implementing the Tcl testsuite infrastructure in
Insight is contained in the testsuite definitions file, "defs", located
in src/gdb/testsuite/gdb.gdbtk. This file contains routines necessary
to write tests for Insight.
Public functions:
proc gdbtk_read_defs {}
This function, located in Insight's core Tcl library, attempts to load
the testsuite definitions file. If it fails, it will either pop up
a dialog box with the error (if running interactively) or it will
print the error to stderr and exit (if running non-interactively).
If successful, it will return true.
proc gdbtk_test_file {filename}
This function is used to load the file given by FILENAME into
Insight. It will automatically append ".exe" to any FILENAME
on Cygwin-hosted systems.
If successful, it will load the given file into Insight and
return the output of gdb's file command. It will call "error"
if it was succesful, therefore all calls to "gdbtk_test_file"
should be called using "catch".
Test authors should not use "gdb_cmd {file FILENAME}" to load
files into gdb unless they are testing interface code between
gdb and Insight.
proc gdbtk_test_run {{prog_args {}}}
gdbtk_test_run runs the previoiusly loaded executable, passing
the given arguments to the inferior. Like Insight's Run button,
it will do whatever is necessary to get the executable running,
including any target initialization (setting baud rate and remote
protocol options), downloading the executable to the target, and
finally starting execution.
Test authors should NEVER use "gdb_cmd {run ARGUMENTS}" to run an
executable. Doing so will insure that your tests will only run on
a native debugger.
It returns true if successful or false otherwise. It will report
the error in a dialog box (if running interactively) or it will
print the error to stderr.
proc gdbtk_test {name description script answer}
This is Tcl testsuite equivalent of "expect". "name" is a canonical
name of the test, usually of the form "shortname-major.minor". This is
the name that is used when running selected tests from a given file.
If "name" starts with an asterisk (*), it designates that the test
is expected to fail.
"description" is a short textual description of the test to help
humans understand what it does.
"script" is the actual test script to run. The result of this script
will be compared against "answer" to determine if the test passed
or failed.
It calls gdbtk_print_verbose to print out the results to the terminal
(if running interactively) or to the log file.
proc gdbtk_test_done {}
gdbtk_test_done is called at the very end of all tcl tests. It is used
to exit Insight and return control back to the dejagnu driver which
started the tests.
proc gdbtk_dotests {file args}
Obsolete.
proc do_test {{file {}} {verbose {}} {tests {}}}
This procedure is used to invoke the Insight test(s) given in FILE
which match the regular expression(s) in TESTS. This is invoked
from Insight's console window to run tests interactively.
VERBOSE sets the verbosity of the test run. When set to one,
the testsuite will report all results in human readable form.
When set greater than one, it will print out results as a list,
i.e., for passing to gdbtk_analyze_results. If zero, it will only
print errors and failures in human readable form.
Public global variables:
objdir - The objdir from dejagnu. See gdbtk_start for more information.
srcdir - The srcdir from dejagnu. See gdbtk_start for more information.
test_ran - Indicates whether the last test ran or not. See example below.
Private functions:
proc gdbtk_test_error {desc}
An internal function used to report a framework error in the testsuite.
"desc" is a description of the error. It calls gdbtk_test_done.
proc gdbtk_print_verbose {status name description script code answer}
A helper procedure to gdbtk_test which prints out results to the terminal
or the logfile (or both or none).
Private global variables:
_test - An array used by the testsuite internals.
Tcl Test Basics
Armed with the basic interface described above, it is possible to test Insight's
GUI. Please do not write tests which attempt to imitate a user (moving the
mouse and clicking buttons), unless there is no other way to test the functionality.
The basic test file (with one test) looks like this (nonsensical one):
File: mytest.exp
1 if {![gdbtk_read_defs]} {
2 break
3 }
4
5 global objdir test_ran
6 set program [file join $objdir mytest]
7 if {[catch {gdbtk_test_file $program} t]} {
8 gdbtk_test_error "loading \"$program\": $t"
9 }
10 if {![gdbtk_test_run]} { exit 1 }
11
12 global foo
13 set foo 1
14
15 # Test: mytest-1.1
16 # Desc: check if a source window was created
17 gdbtk_test mytest-1.1 {source window created} {
18 set window [ManagedWin::find SrcWin]
19 llength $window
20 set foo 13
21 } {1}
22
23 if {$test_ran} {
24 set foo 1
25 }
26
27 # Done
28 gdbtk_test_done
Line 1 calls the Inisght function gdbtk_read_defs to read in the testsuite
definitions file.
Line 6 then specifies the name of a file (mytest) in the object directory
which is loaded into gdb on Line 7. If loading the file into Insight
failed, gdbtk_test_error is called to publish the error (and terminate the
test run for this file).
Line 10 runs the executable on the target, and exits if it was unable
to do so.
Line 13 simply sets a global variable foo to illustrate the purpose
of the global "test_ran". Before the test "mytest-1.1" runs, foo is set to
one. In order to support running specific tests, the state of the debugger
cannot be altered INSIDE gdbtk_test scripts, since the contents of the
script may not be run if the user requested only a specific test to run.
Line 20 in the middle of the test modifies the global foo. If subsequent
test relied on foo being one, we would have a state violation, since
mytest-1.1 may have (or may have not) run.
Therefore, we can check if a test ran and reset foo by checking the
global "test_ran". If set, we know that the previous test (mytest-1.1)
was run, and that foo is now thirteen. We reset the result back to one.
(Aside: Some tests do not follow this rule explicitly: they can assume
that all tests run sequentially. In these cases, running a specific
test in the file will probably fail, since the debugger is not brought
to a known state along the way.)
Lines 17-21 contain the actual test. The test's name is "mytest-1.1". It
is this name that may be referred to when asking the testsuite to run
a specific test. The description of this test is "source window created",
indicating that mytest-1.1's purpose is to check whether a source window
was created.
If gdbtk_test determines that this test is to run, it will execute the
next part, lines 18-20, and compare the output of that script (llength
$window) with "1". If the result is not "1", a failure is recorded. If
it is "1", a pass is recorded.
Finally, the test file is done and exits on line 28.
Go to most recent revision | Compare with Previous | Blame | View Log