Subversion Repositories openrisc_2011-10-31

                            OpenRISC Test Scripts

                            =====================

Rationale

=========

The standard GNU tool chain regression is quite capable of running on multiple

remote targets.

However it does not include a facility to restart individual threads of

testing if a target fails. It is thus unsuitable for remote targets which are

in some respect unreliable.

The script run-all-tests.sh fixes this. It uses the standard GNU tool chain

DejaGnu test framework. However it does its own partitioning of threads. As

each thread completes, it then tests that the remote target is still alive,

and if not it marks that target as unavailable and repeats the test.

Under simulation it is also appropriate to use finer grain parallelism than is

available under the standard GNU toolchain test framework.

Prerequisites

=============

The tests require DejaGnu to be installed, and assume that the standard

OpenRISC build hierarchy (as used by bld-all.sh) is adopted.

DejaGnu Test Framework

======================

Board specifications are provided in the boards directory.

Testing of the or32-linux tool chain uses multiple remote targets with telnet

and FTP. The target may be physical hardware or software simulations

implementing Ethernet and running FTP and telnet daemons. The standard DejaGnu

scripts are incomplete for telnet and have errors for FTP, so the board

definition includes missing and fixed expect/TCL code to fix this.

The test script expects the DEJAGNU environment variable to point to the

site.exp file in the main or32 directory (the one containing the boards

directory). It uses the get-ip.sh script in that directory to obtain the IP

address of the next available remote target for testing.

The list of target IP addresses is placed in the file ${DEJAGNU}/ip-avail.txt.

Linux Configuration

===================

The standard Linux configuration for OpenRISC does not start an FTP

daemon. Modify arch/openrisc/support/initramfs/etc/inetd.conf as follows:

  telnet stream tcp nowait root /usr/sbin/telnetd telnetd -i

  ftp stream tcp nowait root ftpd ftpd -w /

The test scripts expect a /tmp directory to be present, so create this as

arch/openrisc/support/initramfs/tmp.

The test scripts FTP programs into the root directory. However telnet logs in

as root, with a home directory in /root. Fix this by editing

arch/openrisc/support/initramfs/etc/passwd to read

  root::0:0::/:/bin/sh

Remember to rebuild Linux after these changes.

Board Configuration

===================

The or32-linux-sim.exp script in the boards directory sets various

parameters, which may need adjusting. In particular it specifies the absolute

location of the C and C++ compilers and gives timeouts for command execution

(1200 seconds) and FTP upload/download (both 120 seconds).

run-all-tests.sh

================

Check the function "set_globals" to ensure that the various files and

directories are correct.

The main function is "run_tool_tests". The first argument is the tool test

directory, the second the tool source directory, the third the tool name and

the remaining arguments a list of test thread specifications.

Each test thread specification takes the form:

  #

A runtest target is typically an expect script, qualified by the names of

specific source files and/or directories to be tested, for example

  executed.exp=execute/2001*%execute/2002*

The default script runs all the tests for gcc, g++ and libstdc++-v3 (using

three calls to "run-tool_tests". Edit the file to adjust this.

Having set up ip-avail.txt, the script is then run as:

  ./run-all-tests.sh

Diagnostics

===========

To increase the diagnostic output, find the "runtest" command in

run-all-tests.sh and add "-v" to its options (for even more output add "-v -v"

or even "-v -v -v").

Note this is useful for debugging the test scripts rather than actual tests

themselves.

Supplementary Scripts

=====================

run-all-tests.sh leaves each tests results in its own log and summary

files. The script check-results.sh collates all the results. For example

  ./check-results.sh bd-linux/or32-linux/libstdc++-v3/testsuite/libstdc++/*.log

It is sometimes useful to see how long each test thread took. This may lead to

adjustment in partitioning of the tests. This can be achieved with the

check-times.sh script. For example:

  ./check-times.sh bd-linux/or32-linux/libstdc++-v3/testsuite/libstdc++/*.log

Feedback

========

Suggestions for improvements always welcome.

Jeremy Bennett 

3 June 2011