Testing with QMTest
|
Testing with QMTest
|
===================
|
===================
|
|
|
You can use QMTest to test G++. (In the future, it may be possible to
|
You can use QMTest to test G++. (In the future, it may be possible to
|
test other parts of GCC with QMTest as well, but it is not possible
|
test other parts of GCC with QMTest as well, but it is not possible
|
yet.)
|
yet.)
|
|
|
The use of QMTest to run the G++ tests has not been approved as an
|
The use of QMTest to run the G++ tests has not been approved as an
|
officially supported testing procedure. Therefore, you must run the
|
officially supported testing procedure. Therefore, you must run the
|
tests using DejaGNU (with "make check-g++") before committing changes
|
tests using DejaGNU (with "make check-g++") before committing changes
|
that affect G++.
|
that affect G++.
|
|
|
QMTest emulates DejaGNU behavior very closely when running the tests.
|
QMTest emulates DejaGNU behavior very closely when running the tests.
|
|
|
QMTest has two output modes: a DejaGNU emulation mode and a native
|
QMTest has two output modes: a DejaGNU emulation mode and a native
|
QMTest mode.
|
QMTest mode.
|
|
|
In the DejaGNU mode, you should receive output that is almost exactly
|
In the DejaGNU mode, you should receive output that is almost exactly
|
the same as the DejaGNU output; in particular, you should see the same
|
the same as the DejaGNU output; in particular, you should see the same
|
number of passes, failures, etc. When using the DejaGNU-style output,
|
number of passes, failures, etc. When using the DejaGNU-style output,
|
QMTest uses the "xfail" indications in the test cases to determine
|
QMTest uses the "xfail" indications in the test cases to determine
|
which tests are expected to pass and which are expected to fail, and
|
which tests are expected to pass and which are expected to fail, and
|
presents that information in the same way as DejaGNU.
|
presents that information in the same way as DejaGNU.
|
|
|
In the QMTest mode, the number of passes and failures will be
|
In the QMTest mode, the number of passes and failures will be
|
different from that obtained when using DejaGNU. The reason is that a
|
different from that obtained when using DejaGNU. The reason is that a
|
single source file may contain multiple DejaGNU tests. In DejaGNU,
|
single source file may contain multiple DejaGNU tests. In DejaGNU,
|
each line where a diagnostic is expected is considered a separate
|
each line where a diagnostic is expected is considered a separate
|
test. Testing for successful compilation and testing for successful
|
test. Testing for successful compilation and testing for successful
|
execution of the generated program are considered separate tests. So,
|
execution of the generated program are considered separate tests. So,
|
a single source file "test.C" could contain, say, seven tests; some of
|
a single source file "test.C" could contain, say, seven tests; some of
|
which might pass and some of which might fail.
|
which might pass and some of which might fail.
|
|
|
In the QMTest mode, each source file is considered a single test. If
|
In the QMTest mode, each source file is considered a single test. If
|
any of the seven sub-tests fail, the entire test is considered to
|
any of the seven sub-tests fail, the entire test is considered to
|
fail. However, QMTest does present information about *why* the test
|
fail. However, QMTest does present information about *why* the test
|
failed, so the same information is effectively available.
|
failed, so the same information is effectively available.
|
|
|
In the QMTest mode, whether or not a test is expected to fail is
|
In the QMTest mode, whether or not a test is expected to fail is
|
determined not by an indication in the test, but rather by comparing
|
determined not by an indication in the test, but rather by comparing
|
the new results to the results of a previous run. Testing for whether
|
the new results to the results of a previous run. Testing for whether
|
a change caused a regression is very simple: run the tests before
|
a change caused a regression is very simple: run the tests before
|
making the change, run them again after making the change, and let
|
making the change, run them again after making the change, and let
|
QMTest compare the results.
|
QMTest compare the results.
|
|
|
The mode chosen only affects the output from QMTest, not how it runs
|
The mode chosen only affects the output from QMTest, not how it runs
|
the tests or how it stores the data. Therefore, if you choose to run
|
the tests or how it stores the data. Therefore, if you choose to run
|
in the QMTest mode and later want to get the DejaGNU style output, or
|
in the QMTest mode and later want to get the DejaGNU style output, or
|
vice versa, you can do that as described below.
|
vice versa, you can do that as described below.
|
|
|
Setting Up
|
Setting Up
|
==========
|
==========
|
|
|
You must download and install the following software:
|
You must download and install the following software:
|
|
|
- Python 2.2 (or greater)
|
- Python 2.2 (or greater)
|
|
|
See http://www.python.org.
|
See http://www.python.org.
|
|
|
You may already have Python on your system; in particular, many
|
You may already have Python on your system; in particular, many
|
GNU/Linux systems ship with Python installed.
|
GNU/Linux systems ship with Python installed.
|
|
|
Installation instructions are available on the web-site.
|
Installation instructions are available on the web-site.
|
|
|
- A current version of QMTest. No released version provides all of
|
- A current version of QMTest. No released version provides all of
|
the functionality required, so you must obtain QMTest from CVS.
|
the functionality required, so you must obtain QMTest from CVS.
|
|
|
To do that, follow the instructions at:
|
To do that, follow the instructions at:
|
|
|
http://www.codesourcery.com/qmtest
|
http://www.codesourcery.com/qmtest
|
|
|
Installation instructions are available in the file called README
|
Installation instructions are available in the file called README
|
after you check out QMTest.
|
after you check out QMTest.
|
|
|
- The "qmtc" and "qmtest_gcc" QMTest support packages. These are
|
- The "qmtc" and "qmtest_gcc" QMTest support packages. These are
|
available from the same CVS repository as QMTest. For example, to
|
available from the same CVS repository as QMTest. For example, to
|
check out "qmtc", do:
|
check out "qmtc", do:
|
|
|
cvs -d :pserver:anoncvs@cvs.codesourcery.com:/home/qm/Repository \
|
cvs -d :pserver:anoncvs@cvs.codesourcery.com:/home/qm/Repository \
|
co qmtc
|
co qmtc
|
|
|
You do not have to install these packages; you need only check them
|
You do not have to install these packages; you need only check them
|
out.
|
out.
|
|
|
Running the Tests
|
Running the Tests
|
=================
|
=================
|
|
|
First, you must set QMTEST_CLASS_PATH so that it can find the qmtc and
|
First, you must set QMTEST_CLASS_PATH so that it can find the qmtc and
|
qmtest_gcc support packages:
|
qmtest_gcc support packages:
|
|
|
export QMTEST_CLASS_PATH=/path/to/qmtc:/path/to/qmtest_gcc
|
export QMTEST_CLASS_PATH=/path/to/qmtc:/path/to/qmtest_gcc
|
|
|
The, run "make qmtest-g++" in the gcc directory of your build tree.
|
The, run "make qmtest-g++" in the gcc directory of your build tree.
|
|
|
Here are some more advanced usage instructions:
|
Here are some more advanced usage instructions:
|
|
|
1. To run a particular set of tests (rather than all of the tests),
|
1. To run a particular set of tests (rather than all of the tests),
|
use the make variable "QMTEST_GPP_TESTS". For example,
|
use the make variable "QMTEST_GPP_TESTS". For example,
|
|
|
make QMTEST_GPP_TESTS="g++.dg" qmtest-g++
|
make QMTEST_GPP_TESTS="g++.dg" qmtest-g++
|
|
|
will run only the tests in the g++.dg subdirectory, and:
|
will run only the tests in the g++.dg subdirectory, and:
|
|
|
make QMTEST_GPP_TESTS="g++.dg/special/conpr-1.C \
|
make QMTEST_GPP_TESTS="g++.dg/special/conpr-1.C \
|
g++.old-deja/g++.other/access2.C"
|
g++.old-deja/g++.other/access2.C"
|
qmtest-g++
|
qmtest-g++
|
|
|
will run only the two tests indicated.
|
will run only the two tests indicated.
|
|
|
2. To run qmtest with particular flags, use the make variables
|
2. To run qmtest with particular flags, use the make variables
|
"QMTESTFLAGS" and "QMTESTRUNFLAGS". For example:
|
"QMTESTFLAGS" and "QMTESTRUNFLAGS". For example:
|
|
|
make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++
|
make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++
|
|
|
will run qmtest like this:
|
will run qmtest like this:
|
|
|
qmtest -v run -f full ...
|
qmtest -v run -f full ...
|
|
|
(The "-f full" mode will provide detailed information about each
|
(The "-f full" mode will provide detailed information about each
|
test as it runs.)
|
test as it runs.)
|
|
|
3. To run the compiler with particular flags, use QMTESTRUNFLAGS to
|
3. To run the compiler with particular flags, use QMTESTRUNFLAGS to
|
set the QMTest context variable "CompilerTable.cplusplus_options",
|
set the QMTest context variable "CompilerTable.cplusplus_options",
|
like this:
|
like this:
|
|
|
make \
|
make \
|
QMTESTRUNFLAGS='-c CompilerTable.cplusplus_options="-funroll-loops"' \
|
QMTESTRUNFLAGS='-c CompilerTable.cplusplus_options="-funroll-loops"' \
|
qmtest-g++
|
qmtest-g++
|
|
|
The compiler will then use the "-funroll-loops" switch when
|
The compiler will then use the "-funroll-loops" switch when
|
compiling.
|
compiling.
|
|
|
4. If qmtest is not in your path, you can indicate the full path to
|
4. If qmtest is not in your path, you can indicate the full path to
|
QMTest by using the make variable "QMTEST_PATH", like this:
|
QMTest by using the make variable "QMTEST_PATH", like this:
|
|
|
make QMTEST_PATH=/path/to/qmtest qmtest-g++
|
make QMTEST_PATH=/path/to/qmtest qmtest-g++
|
|
|
5. To start the QMTest GUI, use:
|
5. To start the QMTest GUI, use:
|
|
|
make qmtest-gui
|
make qmtest-gui
|
|
|
(Note that this will run the program called "mozilla" in your path.
|
(Note that this will run the program called "mozilla" in your path.
|
If you want to use another browser, you must configure qmtest as
|
If you want to use another browser, you must configure qmtest as
|
described in its manual.)
|
described in its manual.)
|
|
|
Bear in mind that the QMTest GUI is insecure; malicious users with
|
Bear in mind that the QMTest GUI is insecure; malicious users with
|
access to your machine may be able to run commands as if they were
|
access to your machine may be able to run commands as if they were
|
you. The QMTest GUI only binds to the loopback IP addresss, which
|
you. The QMTest GUI only binds to the loopback IP addresss, which
|
provides a measure of security, but not enough for use in untrusted
|
provides a measure of security, but not enough for use in untrusted
|
environments.
|
environments.
|
|
|
6. If you have a multiprocessor, you can run the tests in parallel by
|
6. If you have a multiprocessor, you can run the tests in parallel by
|
passing the "-j" option to qmtest:
|
passing the "-j" option to qmtest:
|
|
|
make QMTESTRUNFLAGS="-j 4" qmtest-g++
|
make QMTESTRUNFLAGS="-j 4" qmtest-g++
|
|
|
will run tests in four threads. (It is also possible to run tests
|
will run tests in four threads. (It is also possible to run tests
|
across multiple machines; for more information see the QMTest
|
across multiple machines; for more information see the QMTest
|
manual.)
|
manual.)
|
|
|
7. If a test (say "g++.dg/abi/bitfield1.C") fails, and you want to get
|
7. If a test (say "g++.dg/abi/bitfield1.C") fails, and you want to get
|
more detailed information, you can do:
|
more detailed information, you can do:
|
|
|
cd qmtestsuite
|
cd qmtestsuite
|
qmtest summarize g++.qmr g++.dg/abi/bitfield1.C
|
qmtest summarize g++.qmr g++.dg/abi/bitfield1.C
|
|
|
to get more information about the commands that were run and the
|
to get more information about the commands that were run and the
|
output produced.
|
output produced.
|
|
|
