Subversion Repositories openrisc_2011-10-31

                            OpenRISC Test Scripts
                            =====================

Rationale
=========

The standard GNU tool chain regression is quite capable of running on multiple
remote targets.

However it does not include a facility to restart individual threads of
testing if a target fails. It is thus unsuitable for remote targets which are
in some respect unreliable.

The script run-all-tests.sh fixes this. It uses the standard GNU tool chain
DejaGnu test framework. However it does its own partitioning of threads. As
each thread completes, it then tests that the remote target is still alive,
and if not it marks that target as unavailable and repeats the test.

Under simulation it is also appropriate to use finer grain parallelism than is
available under the standard GNU toolchain test framework.


Prerequisites
=============

The tests require DejaGnu to be installed, and assume that the standard
OpenRISC build hierarchy (as used by bld-all.sh) is adopted.


DejaGnu Test Framework
======================

Board specifications are provided in the boards directory.

Testing of the or32-linux tool chain uses multiple remote targets with telnet
and FTP. The target may be physical hardware or software simulations
implementing Ethernet and running FTP and telnet daemons. The standard DejaGnu
scripts are incomplete for telnet and have errors for FTP, so the board
definition includes missing and fixed expect/TCL code to fix this.

The test script expects the DEJAGNU environment variable to point to the
site.exp file in the main or32 directory (the one containing the boards
directory). It uses the get-ip.sh script in that directory to obtain the IP
address of the next available remote target for testing.

The list of target IP addresses is placed in the file ${DEJAGNU}/ip-avail.txt.


Linux Configuration
===================

The standard Linux configuration for OpenRISC does not start an FTP
daemon. Modify arch/openrisc/support/initramfs/etc/inetd.conf as follows:

  telnet stream tcp nowait root /usr/sbin/telnetd telnetd -i
  ftp stream tcp nowait root ftpd ftpd -w /

The test scripts expect a /tmp directory to be present, so create this as
arch/openrisc/support/initramfs/tmp.

The test scripts FTP programs into the root directory. However telnet logs in
as root, with a home directory in /root. Fix this by editing
arch/openrisc/support/initramfs/etc/passwd to read

  root::0:0::/:/bin/sh

Remember to rebuild Linux after these changes.


Board Configuration
===================

The or32-linux-sim.exp script in the boards directory sets various
parameters, which may need adjusting. In particular it specifies the absolute
location of the C and C++ compilers and gives timeouts for command execution
(1200 seconds) and FTP upload/download (both 120 seconds).


run-all-tests.sh
================

Check the function "set_globals" to ensure that the various files and
directories are correct.

The main function is "run_tool_tests". The first argument is the tool test
directory, the second the tool source directory, the third the tool name and
the remaining arguments a list of test thread specifications.

Each test thread specification takes the form:

  <thread name>#<runtest target>

A runtest target is typically an expect script, qualified by the names of
specific source files and/or directories to be tested, for example

  executed.exp=execute/2001*%execute/2002*

The default script runs all the tests for gcc, g++ and libstdc++-v3 (using
three calls to "run-tool_tests". Edit the file to adjust this.

Having set up ip-avail.txt, the script is then run as:

  ./run-all-tests.sh


Diagnostics
===========

To increase the diagnostic output, find the "runtest" command in
run-all-tests.sh and add "-v" to its options (for even more output add "-v -v"
or even "-v -v -v").

Note this is useful for debugging the test scripts rather than actual tests
themselves.


Supplementary Scripts
=====================

run-all-tests.sh leaves each tests results in its own log and summary
files. The script check-results.sh collates all the results. For example

  ./check-results.sh bd-linux/or32-linux/libstdc++-v3/testsuite/libstdc++/*.log

It is sometimes useful to see how long each test thread took. This may lead to
adjustment in partitioning of the tests. This can be achieved with the
check-times.sh script. For example:

  ./check-times.sh bd-linux/or32-linux/libstdc++-v3/testsuite/libstdc++/*.log


Feedback
========

Suggestions for improvements always welcome.


Jeremy Bennett <jeremy.bennett@embecosm.com>
3 June 2011