1 |
742 |
jeremybenn |
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
2 |
|
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
|
3 |
|
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Test</title><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"/><meta name="keywords" content=" ISO C++ , test , testsuite , performance , conformance , ABI , exception safety "/><meta name="keywords" content=" ISO C++ , library "/><meta name="keywords" content=" ISO C++ , runtime , library "/><link rel="home" href="../index.html" title="The GNU C++ Library"/><link rel="up" href="appendix_porting.html" title="Appendix B. Porting and Maintenance"/><link rel="prev" href="internals.html" title="Porting to New Hardware or Operating Systems"/><link rel="next" href="abi.html" title="ABI Policy and Guidelines"/></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Test</th></tr><tr><td align="left"><a accesskey="p" href="internals.html">Prev</a> </td><th width="60%" align="center">Appendix B.
|
4 |
|
|
Porting and Maintenance
|
5 |
|
|
|
6 |
|
|
</th><td align="right"> <a accesskey="n" href="abi.html">Next</a></td></tr></table><hr/></div><div class="section" title="Test"><div class="titlepage"><div><div><h2 class="title"><a id="manual.intro.setup.test"/>Test</h2></div></div></div><p>
|
7 |
|
|
The libstdc++ testsuite includes testing for standard conformance,
|
8 |
|
|
regressions, ABI, and performance.
|
9 |
|
|
</p><div class="section" title="Organization"><div class="titlepage"><div><div><h3 class="title"><a id="test.organization"/>Organization</h3></div></div></div><div class="section" title="Directory Layout"><div class="titlepage"><div><div><h4 class="title"><a id="test.organization.layout"/>Directory Layout</h4></div></div></div><p>
|
10 |
|
|
The directory <span class="emphasis"><em>libsrcdir/testsuite</em></span> contains the
|
11 |
|
|
individual test cases organized in sub-directories corresponding to
|
12 |
|
|
chapters of the C++ standard (detailed below), the dejagnu test
|
13 |
|
|
harness support files, and sources to various testsuite utilities
|
14 |
|
|
that are packaged in a separate testing library.
|
15 |
|
|
</p><p>
|
16 |
|
|
All test cases for functionality required by the runtime components
|
17 |
|
|
of the C++ standard (ISO 14882) are files within the following
|
18 |
|
|
directories.
|
19 |
|
|
</p><pre class="programlisting">
|
20 |
|
|
17_intro
|
21 |
|
|
18_support
|
22 |
|
|
19_diagnostics
|
23 |
|
|
20_util
|
24 |
|
|
21_strings
|
25 |
|
|
22_locale
|
26 |
|
|
23_containers
|
27 |
|
|
25_algorithms
|
28 |
|
|
26_numerics
|
29 |
|
|
27_io
|
30 |
|
|
28_regex
|
31 |
|
|
29_atomics
|
32 |
|
|
30_threads
|
33 |
|
|
</pre><p>
|
34 |
|
|
In addition, the following directories include test files:
|
35 |
|
|
</p><pre class="programlisting">
|
36 |
|
|
tr1 Tests for components as described by the Technical Report on Standard Library Extensions (TR1).
|
37 |
|
|
backward Tests for backwards compatibility and deprecated features.
|
38 |
|
|
demangle Tests for __cxa_demangle, the IA 64 C++ ABI demangler
|
39 |
|
|
ext Tests for extensions.
|
40 |
|
|
performance Tests for performance analysis, and performance regressions.
|
41 |
|
|
</pre><p>
|
42 |
|
|
Some directories don't have test files, but instead contain
|
43 |
|
|
auxiliary information:
|
44 |
|
|
</p><pre class="programlisting">
|
45 |
|
|
config Files for the dejagnu test harness.
|
46 |
|
|
lib Files for the dejagnu test harness.
|
47 |
|
|
libstdc++* Files for the dejagnu test harness.
|
48 |
|
|
data Sample text files for testing input and output.
|
49 |
|
|
util Files for libtestc++, utilities and testing routines.
|
50 |
|
|
</pre><p>
|
51 |
|
|
Within a directory that includes test files, there may be
|
52 |
|
|
additional subdirectories, or files. Originally, test cases
|
53 |
|
|
were appended to one file that represented a particular section
|
54 |
|
|
of the chapter under test, and was named accordingly. For
|
55 |
|
|
instance, to test items related to <code class="code"> 21.3.6.1 -
|
56 |
|
|
basic_string::find [lib.string::find]</code> in the standard,
|
57 |
|
|
the following was used:
|
58 |
|
|
</p><pre class="programlisting">
|
59 |
|
|
21_strings/find.cc
|
60 |
|
|
</pre><p>
|
61 |
|
|
However, that practice soon became a liability as the test cases
|
62 |
|
|
became huge and unwieldy, and testing new or extended
|
63 |
|
|
functionality (like wide characters or named locales) became
|
64 |
|
|
frustrating, leading to aggressive pruning of test cases on some
|
65 |
|
|
platforms that covered up implementation errors. Now, the test
|
66 |
|
|
suite has a policy of one file, one test case, which solves the
|
67 |
|
|
above issues and gives finer grained results and more manageable
|
68 |
|
|
error debugging. As an example, the test case quoted above
|
69 |
|
|
becomes:
|
70 |
|
|
</p><pre class="programlisting">
|
71 |
|
|
21_strings/basic_string/find/char/1.cc
|
72 |
|
|
21_strings/basic_string/find/char/2.cc
|
73 |
|
|
21_strings/basic_string/find/char/3.cc
|
74 |
|
|
21_strings/basic_string/find/wchar_t/1.cc
|
75 |
|
|
21_strings/basic_string/find/wchar_t/2.cc
|
76 |
|
|
21_strings/basic_string/find/wchar_t/3.cc
|
77 |
|
|
</pre><p>
|
78 |
|
|
All new tests should be written with the policy of one test
|
79 |
|
|
case, one file in mind.
|
80 |
|
|
</p></div><div class="section" title="Naming Conventions"><div class="titlepage"><div><div><h4 class="title"><a id="test.organization.naming"/>Naming Conventions</h4></div></div></div><p>
|
81 |
|
|
In addition, there are some special names and suffixes that are
|
82 |
|
|
used within the testsuite to designate particular kinds of
|
83 |
|
|
tests.
|
84 |
|
|
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
|
85 |
|
|
<span class="emphasis"><em>_xin.cc</em></span>
|
86 |
|
|
</p><p>
|
87 |
|
|
This test case expects some kind of interactive input in order
|
88 |
|
|
to finish or pass. At the moment, the interactive tests are not
|
89 |
|
|
run by default. Instead, they are run by hand, like:
|
90 |
|
|
</p><pre class="programlisting">
|
91 |
|
|
g++ 27_io/objects/char/3_xin.cc
|
92 |
|
|
cat 27_io/objects/char/3_xin.in | a.out
|
93 |
|
|
</pre></li><li class="listitem"><p>
|
94 |
|
|
<span class="emphasis"><em>.in</em></span>
|
95 |
|
|
</p><p>
|
96 |
|
|
This file contains the expected input for the corresponding <span class="emphasis"><em>
|
97 |
|
|
_xin.cc</em></span> test case.
|
98 |
|
|
</p></li><li class="listitem"><p>
|
99 |
|
|
<span class="emphasis"><em>_neg.cc</em></span>
|
100 |
|
|
</p><p>
|
101 |
|
|
This test case is expected to fail: it's a negative test. At the
|
102 |
|
|
moment, these are almost always compile time errors.
|
103 |
|
|
</p></li><li class="listitem"><p>
|
104 |
|
|
<span class="emphasis"><em>char</em></span>
|
105 |
|
|
</p><p>
|
106 |
|
|
This can either be a directory name or part of a longer file
|
107 |
|
|
name, and indicates that this file, or the files within this
|
108 |
|
|
directory are testing the <code class="code">char</code> instantiation of a
|
109 |
|
|
template.
|
110 |
|
|
</p></li><li class="listitem"><p>
|
111 |
|
|
<span class="emphasis"><em>wchar_t</em></span>
|
112 |
|
|
</p><p>
|
113 |
|
|
This can either be a directory name or part of a longer file
|
114 |
|
|
name, and indicates that this file, or the files within this
|
115 |
|
|
directory are testing the <code class="code">wchar_t</code> instantiation of
|
116 |
|
|
a template. Some hosts do not support <code class="code">wchar_t</code>
|
117 |
|
|
functionality, so for these targets, all of these tests will not
|
118 |
|
|
be run.
|
119 |
|
|
</p></li><li class="listitem"><p>
|
120 |
|
|
<span class="emphasis"><em>thread</em></span>
|
121 |
|
|
</p><p>
|
122 |
|
|
This can either be a directory name or part of a longer file
|
123 |
|
|
name, and indicates that this file, or the files within this
|
124 |
|
|
directory are testing situations where multiple threads are
|
125 |
|
|
being used.
|
126 |
|
|
</p></li><li class="listitem"><p>
|
127 |
|
|
<span class="emphasis"><em>performance</em></span>
|
128 |
|
|
</p><p>
|
129 |
|
|
This can either be an enclosing directory name or part of a
|
130 |
|
|
specific file name. This indicates a test that is used to
|
131 |
|
|
analyze runtime performance, for performance regression testing,
|
132 |
|
|
or for other optimization related analysis. At the moment, these
|
133 |
|
|
test cases are not run by default.
|
134 |
|
|
</p></li></ul></div></div></div><div class="section" title="Running the Testsuite"><div class="titlepage"><div><div><h3 class="title"><a id="test.run"/>Running the Testsuite</h3></div></div></div><div class="section" title="Basic"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.basic"/>Basic</h4></div></div></div><p>
|
135 |
|
|
You can check the status of the build without installing it
|
136 |
|
|
using the dejagnu harness, much like the rest of the gcc
|
137 |
|
|
tools.</p><pre class="programlisting"> make check</pre><p>in the <span class="emphasis"><em>libbuilddir</em></span> directory.</p><p>or</p><pre class="programlisting"> make check-target-libstdc++-v3</pre><p>in the <span class="emphasis"><em>gccbuilddir</em></span> directory.
|
138 |
|
|
</p><p>
|
139 |
|
|
These commands are functionally equivalent and will create a
|
140 |
|
|
'testsuite' directory underneath
|
141 |
|
|
<span class="emphasis"><em>libbuilddir</em></span> containing the results of the
|
142 |
|
|
tests. Two results files will be generated: <span class="emphasis"><em>
|
143 |
|
|
libstdc++.sum</em></span>, which is a PASS/FAIL summary for each
|
144 |
|
|
test, and <span class="emphasis"><em>libstdc++.log</em></span> which is a log of
|
145 |
|
|
the exact command line passed to the compiler, the compiler
|
146 |
|
|
output, and the executable output (if any).
|
147 |
|
|
</p><p>
|
148 |
|
|
Archives of test results for various versions and platforms are
|
149 |
|
|
available on the GCC website in the <a class="link" href="http://gcc.gnu.org/gcc-4.3/buildstat.html">build
|
150 |
|
|
status</a> section of each individual release, and are also
|
151 |
|
|
archived on a daily basis on the <a class="link" href="http://gcc.gnu.org/ml/gcc-testresults/current">gcc-testresults</a>
|
152 |
|
|
mailing list. Please check either of these places for a similar
|
153 |
|
|
combination of source version, operating system, and host CPU.
|
154 |
|
|
</p></div><div class="section" title="Variations"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.variations"/>Variations</h4></div></div></div><p>
|
155 |
|
|
There are several options for running tests, including testing
|
156 |
|
|
the regression tests, testing a subset of the regression tests,
|
157 |
|
|
testing the performance tests, testing just compilation, testing
|
158 |
|
|
installed tools, etc. In addition, there is a special rule for
|
159 |
|
|
checking the exported symbols of the shared library.
|
160 |
|
|
</p><p>
|
161 |
|
|
To debug the dejagnu test harness during runs, try invoking with a
|
162 |
|
|
specific argument to the variable RUNTESTFLAGS, as below.
|
163 |
|
|
</p><pre class="programlisting">
|
164 |
|
|
make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
|
165 |
|
|
</pre><p>
|
166 |
|
|
or
|
167 |
|
|
</p><pre class="programlisting">
|
168 |
|
|
make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
|
169 |
|
|
</pre><p>
|
170 |
|
|
To run a subset of the library tests, you will need to generate
|
171 |
|
|
the <span class="emphasis"><em>testsuite_files</em></span> file by running
|
172 |
|
|
<span class="command"><strong>make testsuite_files</strong></span> in the
|
173 |
|
|
<span class="emphasis"><em>libbuilddir/testsuite</em></span> directory, described
|
174 |
|
|
below. Edit the file to remove the tests you don't want and
|
175 |
|
|
then run the testsuite as normal.
|
176 |
|
|
</p><p>
|
177 |
|
|
There are two ways to run on a simulator: set up DEJAGNU to point to a
|
178 |
|
|
specially crafted site.exp, or pass down --target_board flags.
|
179 |
|
|
</p><p>
|
180 |
|
|
Example flags to pass down for various embedded builds are as follows:
|
181 |
|
|
</p><pre class="programlisting">
|
182 |
|
|
--target=powerpc-eabism (libgloss/sim)
|
183 |
|
|
make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"
|
184 |
|
|
|
185 |
|
|
--target=calmrisc32 (libgloss/sid)
|
186 |
|
|
make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"
|
187 |
|
|
|
188 |
|
|
--target=xscale-elf (newlib/sim)
|
189 |
|
|
make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
|
190 |
|
|
</pre><p>
|
191 |
|
|
Also, here is an example of how to run the libstdc++ testsuite
|
192 |
|
|
for a multilibed build directory with different ABI settings:
|
193 |
|
|
</p><pre class="programlisting">
|
194 |
|
|
make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'
|
195 |
|
|
</pre><p>
|
196 |
|
|
You can run the tests with a compiler and library that have
|
197 |
|
|
already been installed. Make sure that the compiler (e.g.,
|
198 |
|
|
<code class="code">g++</code>) is in your <code class="code">PATH</code>. If you are
|
199 |
|
|
using shared libraries, then you must also ensure that the
|
200 |
|
|
directory containing the shared version of libstdc++ is in your
|
201 |
|
|
<code class="code">LD_LIBRARY_PATH</code>, or equivalent. If your GCC source
|
202 |
|
|
tree is at <code class="code">/path/to/gcc</code>, then you can run the tests
|
203 |
|
|
as follows:
|
204 |
|
|
</p><pre class="programlisting">
|
205 |
|
|
runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite
|
206 |
|
|
</pre><p>
|
207 |
|
|
The testsuite will create a number of files in the directory in
|
208 |
|
|
which you run this command,. Some of those files might use the
|
209 |
|
|
same name as files created by other testsuites (like the ones
|
210 |
|
|
for GCC and G++), so you should not try to run all the
|
211 |
|
|
testsuites in parallel from the same directory.
|
212 |
|
|
</p><p>
|
213 |
|
|
In addition, there are some testing options that are mostly of
|
214 |
|
|
interest to library maintainers and system integrators. As such,
|
215 |
|
|
these tests may not work on all cpu and host combinations, and
|
216 |
|
|
may need to be executed in the
|
217 |
|
|
<span class="emphasis"><em>libbuilddir/testsuite</em></span> directory. These
|
218 |
|
|
options include, but are not necessarily limited to, the
|
219 |
|
|
following:
|
220 |
|
|
</p><pre class="programlisting">
|
221 |
|
|
make testsuite_files
|
222 |
|
|
</pre><p>
|
223 |
|
|
Five files are generated that determine what test files
|
224 |
|
|
are run. These files are:
|
225 |
|
|
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
|
226 |
|
|
<span class="emphasis"><em>testsuite_files</em></span>
|
227 |
|
|
</p><p>
|
228 |
|
|
This is a list of all the test cases that will be run. Each
|
229 |
|
|
test case is on a separate line, given with an absolute path
|
230 |
|
|
from the <span class="emphasis"><em>libsrcdir/testsuite</em></span> directory.
|
231 |
|
|
</p></li><li class="listitem"><p>
|
232 |
|
|
<span class="emphasis"><em>testsuite_files_interactive</em></span>
|
233 |
|
|
</p><p>
|
234 |
|
|
This is a list of all the interactive test cases, using the
|
235 |
|
|
same format as the file list above. These tests are not run
|
236 |
|
|
by default.
|
237 |
|
|
</p></li><li class="listitem"><p>
|
238 |
|
|
<span class="emphasis"><em>testsuite_files_performance</em></span>
|
239 |
|
|
</p><p>
|
240 |
|
|
This is a list of all the performance test cases, using the
|
241 |
|
|
same format as the file list above. These tests are not run
|
242 |
|
|
by default.
|
243 |
|
|
</p></li><li class="listitem"><p>
|
244 |
|
|
<span class="emphasis"><em>testsuite_thread</em></span>
|
245 |
|
|
</p><p>
|
246 |
|
|
This file indicates that the host system can run tests which
|
247 |
|
|
involved multiple threads.
|
248 |
|
|
</p></li><li class="listitem"><p>
|
249 |
|
|
<span class="emphasis"><em>testsuite_wchar_t</em></span>
|
250 |
|
|
</p><p>
|
251 |
|
|
This file indicates that the host system can run the wchar_t
|
252 |
|
|
tests, and corresponds to the macro definition <code class="code">
|
253 |
|
|
_GLIBCXX_USE_WCHAR_T</code> in the file c++config.h.
|
254 |
|
|
</p></li></ul></div><pre class="programlisting">
|
255 |
|
|
make check-abi
|
256 |
|
|
</pre><p>
|
257 |
|
|
The library ABI can be tested. This involves testing the shared
|
258 |
|
|
library against an ABI-defining previous version of symbol
|
259 |
|
|
exports.
|
260 |
|
|
</p><pre class="programlisting">
|
261 |
|
|
make check-compile
|
262 |
|
|
</pre><p>
|
263 |
|
|
This rule compiles, but does not link or execute, the
|
264 |
|
|
<span class="emphasis"><em>testsuite_files</em></span> test cases and displays the
|
265 |
|
|
output on stdout.
|
266 |
|
|
</p><pre class="programlisting">
|
267 |
|
|
make check-performance
|
268 |
|
|
</pre><p>
|
269 |
|
|
This rule runs through the
|
270 |
|
|
<span class="emphasis"><em>testsuite_files_performance</em></span> test cases and
|
271 |
|
|
collects information for performance analysis and can be used to
|
272 |
|
|
spot performance regressions. Various timing information is
|
273 |
|
|
collected, as well as number of hard page faults, and memory
|
274 |
|
|
used. This is not run by default, and the implementation is in
|
275 |
|
|
flux.
|
276 |
|
|
</p><p>
|
277 |
|
|
We are interested in any strange failures of the testsuite;
|
278 |
|
|
please email the main libstdc++ mailing list if you see
|
279 |
|
|
something odd or have questions.
|
280 |
|
|
</p></div><div class="section" title="Permutations"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.permutations"/>Permutations</h4></div></div></div><p>
|
281 |
|
|
To run the libstdc++ test suite under the <a class="link" href="debug_mode.html" title="Chapter 17. Debug Mode">debug mode</a>, edit
|
282 |
|
|
<code class="filename">libstdc++-v3/scripts/testsuite_flags</code> to add the
|
283 |
|
|
compile-time flag <code class="constant">-D_GLIBCXX_DEBUG</code> to the
|
284 |
|
|
result printed by the <code class="literal">--build-cxx</code>
|
285 |
|
|
option. Additionally, add the
|
286 |
|
|
<code class="constant">-D_GLIBCXX_DEBUG_PEDANTIC</code> flag to turn on
|
287 |
|
|
pedantic checking. The libstdc++ test suite should produce
|
288 |
|
|
precisely the same results under debug mode that it does under
|
289 |
|
|
release mode: any deviation indicates an error in either the
|
290 |
|
|
library or the test suite.
|
291 |
|
|
</p><p>
|
292 |
|
|
The <a class="link" href="parallel_mode.html" title="Chapter 18. Parallel Mode">parallel
|
293 |
|
|
mode</a> can be tested in much the same manner, substituting
|
294 |
|
|
<code class="constant">-D_GLIBCXX_PARALLEL</code> for
|
295 |
|
|
<code class="constant">-D_GLIBCXX_DEBUG</code> in the previous paragraph.
|
296 |
|
|
</p><p>
|
297 |
|
|
Or, just run the testsuites with <code class="constant">CXXFLAGS</code>
|
298 |
|
|
set to <code class="constant">-D_GLIBCXX_DEBUG</code> or
|
299 |
|
|
<code class="constant">-D_GLIBCXX_PARALLEL</code>.
|
300 |
|
|
</p></div></div><div class="section" title="Writing a new test case"><div class="titlepage"><div><div><h3 class="title"><a id="test.new_tests"/>Writing a new test case</h3></div></div></div><p>
|
301 |
|
|
The first step in making a new test case is to choose the correct
|
302 |
|
|
directory and file name, given the organization as previously
|
303 |
|
|
described.
|
304 |
|
|
</p><p>
|
305 |
|
|
All files are copyright the FSF, and GPL'd: this is very
|
306 |
|
|
important. The first copyright year should correspond to the date
|
307 |
|
|
the file was checked in to SVN.
|
308 |
|
|
</p><p>
|
309 |
|
|
As per the dejagnu instructions, always return 0 from main to
|
310 |
|
|
indicate success.
|
311 |
|
|
</p><p>
|
312 |
|
|
A bunch of utility functions and classes have already been
|
313 |
|
|
abstracted out into the testsuite utility library, <code class="code">
|
314 |
|
|
libtestc++</code>. To use this functionality, just include the
|
315 |
|
|
appropriate header file: the library or specific object files will
|
316 |
|
|
automatically be linked in as part of the testsuite run.
|
317 |
|
|
</p><p>
|
318 |
|
|
For a test that needs to take advantage of the dejagnu test
|
319 |
|
|
harness, what follows below is a list of special keyword that
|
320 |
|
|
harness uses. Basically, a test case contains dg-keywords (see
|
321 |
|
|
dg.exp) indicating what to do and what kinds of behavior are to be
|
322 |
|
|
expected. New test cases should be written with the new style
|
323 |
|
|
DejaGnu framework in mind.
|
324 |
|
|
</p><p>
|
325 |
|
|
To ease transition, here is the list of dg-keyword documentation
|
326 |
|
|
lifted from dg.exp.
|
327 |
|
|
</p><pre class="programlisting">
|
328 |
|
|
# The currently supported options are:
|
329 |
|
|
#
|
330 |
|
|
# dg-prms-id N
|
331 |
|
|
# set prms_id to N
|
332 |
|
|
#
|
333 |
|
|
# dg-options "options ..." [{ target selector }]
|
334 |
|
|
# specify special options to pass to the tool (eg: compiler)
|
335 |
|
|
#
|
336 |
|
|
# dg-do do-what-keyword [{ target/xfail selector }]
|
337 |
|
|
# `do-what-keyword' is tool specific and is passed unchanged to
|
338 |
|
|
# ${tool}-dg-test. An example is gcc where `keyword' can be any of:
|
339 |
|
|
# preprocess|compile|assemble|link|run
|
340 |
|
|
# and will do one of: produce a .i, produce a .s, produce a .o,
|
341 |
|
|
# produce an a.out, or produce an a.out and run it (the default is
|
342 |
|
|
# compile).
|
343 |
|
|
#
|
344 |
|
|
# dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]
|
345 |
|
|
# indicate an error message <regexp> is expected on this line
|
346 |
|
|
# (the test fails if it doesn't occur)
|
347 |
|
|
# Linenum=0 for general tool messages (eg: -V arg missing).
|
348 |
|
|
# "." means the current line.
|
349 |
|
|
#
|
350 |
|
|
# dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]
|
351 |
|
|
# indicate a warning message <regexp> is expected on this line
|
352 |
|
|
# (the test fails if it doesn't occur)
|
353 |
|
|
#
|
354 |
|
|
# dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]
|
355 |
|
|
# indicate a bogus error message <regexp> use to occur here
|
356 |
|
|
# (the test fails if it does occur)
|
357 |
|
|
#
|
358 |
|
|
# dg-build regexp comment [{ target/xfail selector }]
|
359 |
|
|
# indicate the build use to fail for some reason
|
360 |
|
|
# (errors covered here include bad assembler generated, tool crashes,
|
361 |
|
|
# and link failures)
|
362 |
|
|
# (the test fails if it does occur)
|
363 |
|
|
#
|
364 |
|
|
# dg-excess-errors comment [{ target/xfail selector }]
|
365 |
|
|
# indicate excess errors are expected (any line)
|
366 |
|
|
# (this should only be used sparingly and temporarily)
|
367 |
|
|
#
|
368 |
|
|
# dg-output regexp [{ target selector }]
|
369 |
|
|
# indicate the expected output of the program is <regexp>
|
370 |
|
|
# (there may be multiple occurrences of this, they are concatenated)
|
371 |
|
|
#
|
372 |
|
|
# dg-final { tcl code }
|
373 |
|
|
# add some tcl code to be run at the end
|
374 |
|
|
# (there may be multiple occurrences of this, they are concatenated)
|
375 |
|
|
# (unbalanced braces must be \-escaped)
|
376 |
|
|
#
|
377 |
|
|
# "{ target selector }" is a list of expressions that determine whether the
|
378 |
|
|
# test succeeds or fails for a particular target, or in some cases whether the
|
379 |
|
|
# option applies for a particular target. If the case of `dg-do' it specifies
|
380 |
|
|
# whether the test case is even attempted on the specified target.
|
381 |
|
|
#
|
382 |
|
|
# The target selector is always optional. The format is one of:
|
383 |
|
|
#
|
384 |
|
|
# { xfail *-*-* ... } - the test is expected to fail for the given targets
|
385 |
|
|
# { target *-*-* ... } - the option only applies to the given targets
|
386 |
|
|
#
|
387 |
|
|
# At least one target must be specified, use *-*-* for "all targets".
|
388 |
|
|
# At present it is not possible to specify both `xfail' and `target'.
|
389 |
|
|
# "native" may be used in place of "*-*-*".
|
390 |
|
|
|
391 |
|
|
Example 1: Testing compilation only
|
392 |
|
|
// { dg-do compile }
|
393 |
|
|
|
394 |
|
|
Example 2: Testing for expected warnings on line 36, which all targets fail
|
395 |
|
|
// { dg-warning "string literals" "" { xfail *-*-* } 36 }
|
396 |
|
|
|
397 |
|
|
Example 3: Testing for expected warnings on line 36
|
398 |
|
|
// { dg-warning "string literals" "" { target *-*-* } 36 }
|
399 |
|
|
|
400 |
|
|
Example 4: Testing for compilation errors on line 41
|
401 |
|
|
// { dg-do compile }
|
402 |
|
|
// { dg-error "no match for" "" { target *-*-* } 41 }
|
403 |
|
|
|
404 |
|
|
Example 5: Testing with special command line settings, or without the
|
405 |
|
|
use of pre-compiled headers, in particular the stdc++.h.gch file. Any
|
406 |
|
|
options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set
|
407 |
|
|
up in the normal.exp file.
|
408 |
|
|
// { dg-options "-O0" { target *-*-* } }
|
409 |
|
|
</pre><p>
|
410 |
|
|
More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.
|
411 |
|
|
</p></div><div class="section" title="Test Harness and Utilities"><div class="titlepage"><div><div><h3 class="title"><a id="test.harness"/>Test Harness and Utilities</h3></div></div></div><div class="section" title="Dejagnu Harness Details"><div class="titlepage"><div><div><h4 class="title"><a id="test.harness.dejagnu"/>Dejagnu Harness Details</h4></div></div></div><p>
|
412 |
|
|
Underlying details of testing for conformance and regressions are
|
413 |
|
|
abstracted via the GNU Dejagnu package. This is similar to the
|
414 |
|
|
rest of GCC.
|
415 |
|
|
</p><p>This is information for those looking at making changes to the testsuite
|
416 |
|
|
structure, and/or needing to trace dejagnu's actions with --verbose. This
|
417 |
|
|
will not be useful to people who are "merely" adding new tests to the existing
|
418 |
|
|
structure.
|
419 |
|
|
</p><p>The first key point when working with dejagnu is the idea of a "tool".
|
420 |
|
|
Files, directories, and functions are all implicitly used when they are
|
421 |
|
|
named after the tool in use. Here, the tool will always be "libstdc++".
|
422 |
|
|
</p><p>The <code class="code">lib</code> subdir contains support routines. The
|
423 |
|
|
<code class="code">lib/libstdc++.exp</code> file ("support library") is loaded
|
424 |
|
|
automagically, and must explicitly load the others. For example, files can
|
425 |
|
|
be copied from the core compiler's support directory into <code class="code">lib</code>.
|
426 |
|
|
</p><p>Some routines in <code class="code">lib/libstdc++.exp</code> are callbacks, some are
|
427 |
|
|
our own. Callbacks must be prefixed with the name of the tool. To easily
|
428 |
|
|
distinguish the others, by convention our own routines are named "v3-*".
|
429 |
|
|
</p><p>The next key point when working with dejagnu is "test files". Any
|
430 |
|
|
directory whose name starts with the tool name will be searched for test files.
|
431 |
|
|
(We have only one.) In those directories, any <code class="code">.exp</code> file is
|
432 |
|
|
considered a test file, and will be run in turn. Our main test file is called
|
433 |
|
|
<code class="code">normal.exp</code>; it runs all the tests in testsuite_files using the
|
434 |
|
|
callbacks loaded from the support library.
|
435 |
|
|
</p><p>The <code class="code">config</code> directory is searched for any particular "target
|
436 |
|
|
board" information unique to this library. This is currently unused and sets
|
437 |
|
|
only default variables.
|
438 |
|
|
</p></div><div class="section" title="Utilities"><div class="titlepage"><div><div><h4 class="title"><a id="test.harness.utils"/>Utilities</h4></div></div></div><p>
|
439 |
|
|
</p><p>
|
440 |
|
|
The testsuite directory also contains some files that implement
|
441 |
|
|
functionality that is intended to make writing test cases easier,
|
442 |
|
|
or to avoid duplication, or to provide error checking in a way that
|
443 |
|
|
is consistent across platforms and test harnesses. A stand-alone
|
444 |
|
|
executable, called <span class="emphasis"><em>abi_check</em></span>, and a static
|
445 |
|
|
library called <span class="emphasis"><em>libtestc++</em></span> are
|
446 |
|
|
constructed. Both of these items are not installed, and only used
|
447 |
|
|
during testing.
|
448 |
|
|
</p><p>
|
449 |
|
|
These files include the following functionality:
|
450 |
|
|
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
|
451 |
|
|
<span class="emphasis"><em>testsuite_abi.h</em></span>,
|
452 |
|
|
<span class="emphasis"><em>testsuite_abi.cc</em></span>,
|
453 |
|
|
<span class="emphasis"><em>testsuite_abi_check.cc</em></span>
|
454 |
|
|
</p><p>
|
455 |
|
|
Creates the executable <span class="emphasis"><em>abi_check</em></span>.
|
456 |
|
|
Used to check correctness of symbol versioning, visibility of
|
457 |
|
|
exported symbols, and compatibility on symbols in the shared
|
458 |
|
|
library, for hosts that support this feature. More information
|
459 |
|
|
can be found in the ABI documentation <a class="link" href="abi.html" title="ABI Policy and Guidelines">here</a>
|
460 |
|
|
</p></li><li class="listitem"><p>
|
461 |
|
|
<span class="emphasis"><em>testsuite_allocator.h</em></span>,
|
462 |
|
|
<span class="emphasis"><em>testsuite_allocator.cc</em></span>
|
463 |
|
|
</p><p>
|
464 |
|
|
Contains specialized allocators that keep track of construction
|
465 |
|
|
and destruction. Also, support for overriding global new and
|
466 |
|
|
delete operators, including verification that new and delete
|
467 |
|
|
are called during execution, and that allocation over max_size
|
468 |
|
|
fails.
|
469 |
|
|
</p></li><li class="listitem"><p>
|
470 |
|
|
<span class="emphasis"><em>testsuite_character.h</em></span>
|
471 |
|
|
</p><p>
|
472 |
|
|
Contains <code class="code">std::char_traits</code> and
|
473 |
|
|
<code class="code">std::codecvt</code> specializations for a user-defined
|
474 |
|
|
POD.
|
475 |
|
|
</p></li><li class="listitem"><p>
|
476 |
|
|
<span class="emphasis"><em>testsuite_hooks.h</em></span>,
|
477 |
|
|
<span class="emphasis"><em>testsuite_hooks.cc</em></span>
|
478 |
|
|
</p><p>
|
479 |
|
|
A large number of utilities, including:
|
480 |
|
|
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>VERIFY</p></li><li class="listitem"><p>set_memory_limits</p></li><li class="listitem"><p>verify_demangle</p></li><li class="listitem"><p>run_tests_wrapped_locale</p></li><li class="listitem"><p>run_tests_wrapped_env</p></li><li class="listitem"><p>try_named_locale</p></li><li class="listitem"><p>try_mkfifo</p></li><li class="listitem"><p>func_callback</p></li><li class="listitem"><p>counter</p></li><li class="listitem"><p>copy_tracker</p></li><li class="listitem"><p>copy_constructor</p></li><li class="listitem"><p>assignment_operator</p></li><li class="listitem"><p>destructor</p></li><li class="listitem"><p>pod_char, pod_int and associated char_traits specializations</p></li></ul></div></li><li class="listitem"><p>
|
481 |
|
|
<span class="emphasis"><em>testsuite_io.h</em></span>
|
482 |
|
|
</p><p>
|
483 |
|
|
Error, exception, and constraint checking for
|
484 |
|
|
<code class="code">std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>.
|
485 |
|
|
</p></li><li class="listitem"><p>
|
486 |
|
|
<span class="emphasis"><em>testsuite_iterators.h</em></span>
|
487 |
|
|
</p><p>
|
488 |
|
|
Wrappers for various iterators.
|
489 |
|
|
</p></li><li class="listitem"><p>
|
490 |
|
|
<span class="emphasis"><em>testsuite_performance.h</em></span>
|
491 |
|
|
</p><p>
|
492 |
|
|
A number of class abstractions for performance counters, and
|
493 |
|
|
reporting functions including:
|
494 |
|
|
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>time_counter</p></li><li class="listitem"><p>resource_counter</p></li><li class="listitem"><p>report_performance</p></li></ul></div></li></ul></div></div></div><div class="section" title="Special Topics"><div class="titlepage"><div><div><h3 class="title"><a id="test.special"/>Special Topics</h3></div></div></div><div class="section" title="Qualifying Exception Safety Guarantees"><div class="titlepage"><div><div><h4 class="title"><a id="test.exception.safety"/>
|
495 |
|
|
Qualifying Exception Safety Guarantees
|
496 |
|
|
<a id="id557519" class="indexterm"/>
|
497 |
|
|
</h4></div></div></div><div class="section" title="Overview"><div class="titlepage"><div><div><h5 class="title"><a id="test.exception.safety.overview"/>Overview</h5></div></div></div><p>
|
498 |
|
|
Testing is composed of running a particular test sequence,
|
499 |
|
|
and looking at what happens to the surrounding code when
|
500 |
|
|
exceptions are thrown. Each test is composed of measuring
|
501 |
|
|
initial state, executing a particular sequence of code under
|
502 |
|
|
some instrumented conditions, measuring a final state, and
|
503 |
|
|
then examining the differences between the two states.
|
504 |
|
|
</p><p>
|
505 |
|
|
Test sequences are composed of constructed code sequences
|
506 |
|
|
that exercise a particular function or member function, and
|
507 |
|
|
either confirm no exceptions were generated, or confirm the
|
508 |
|
|
consistency/coherency of the test subject in the event of a
|
509 |
|
|
thrown exception.
|
510 |
|
|
</p><p>
|
511 |
|
|
Random code paths can be constructed using the basic test
|
512 |
|
|
sequences and instrumentation as above, only combined in a
|
513 |
|
|
random or pseudo-random way.
|
514 |
|
|
</p><p> To compute the code paths that throw, test instruments
|
515 |
|
|
are used that throw on allocation events
|
516 |
|
|
(<code class="classname">__gnu_cxx::throw_allocator_random</code>
|
517 |
|
|
and <code class="classname">__gnu_cxx::throw_allocator_limit</code>)
|
518 |
|
|
and copy, assignment, comparison, increment, swap, and
|
519 |
|
|
various operators
|
520 |
|
|
(<code class="classname">__gnu_cxx::throw_type_random</code>
|
521 |
|
|
and <code class="classname">__gnu_cxx::throw_type_limit</code>). Looping
|
522 |
|
|
through a given test sequence and conditionally throwing in
|
523 |
|
|
all instrumented places. Then, when the test sequence
|
524 |
|
|
completes without an exception being thrown, assume all
|
525 |
|
|
potential error paths have been exercised in a sequential
|
526 |
|
|
manner.
|
527 |
|
|
</p></div><div class="section" title="Existing tests"><div class="titlepage"><div><div><h5 class="title"><a id="test.exception.safety.status"/>
|
528 |
|
|
Existing tests
|
529 |
|
|
</h5></div></div></div><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
|
530 |
|
|
Ad Hoc
|
531 |
|
|
</p><p>
|
532 |
|
|
For example,
|
533 |
|
|
<code class="filename">testsuite/23_containers/list/modifiers/3.cc</code>.
|
534 |
|
|
</p></li><li class="listitem"><p>
|
535 |
|
|
Policy Based Data Structures
|
536 |
|
|
</p><p>
|
537 |
|
|
For example, take the test
|
538 |
|
|
functor <code class="classname">rand_reg_test</code> in
|
539 |
|
|
in <code class="filename">testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc</code>. This uses <code class="classname">container_rand_regression_test</code> in
|
540 |
|
|
<code class="filename">testsuite/util/regression/rand/assoc/container_rand_regression_test.h</code>.
|
541 |
|
|
|
542 |
|
|
</p><p>
|
543 |
|
|
Which has several tests for container member functions,
|
544 |
|
|
Includes control and test container objects. Configuration includes
|
545 |
|
|
random seed, iterations, number of distinct values, and the
|
546 |
|
|
probability that an exception will be thrown. Assumes instantiating
|
547 |
|
|
container uses an extension
|
548 |
|
|
allocator, <code class="classname">__gnu_cxx::throw_allocator_random</code>,
|
549 |
|
|
as the allocator type.
|
550 |
|
|
</p></li><li class="listitem"><p>
|
551 |
|
|
C++11 Container Requirements.
|
552 |
|
|
</p><p>
|
553 |
|
|
Coverage is currently limited to testing container
|
554 |
|
|
requirements for exception safety,
|
555 |
|
|
although <code class="classname">__gnu_cxx::throw_type</code> meets
|
556 |
|
|
the additional type requirements for testing numeric data
|
557 |
|
|
structures and instantiating algorithms.
|
558 |
|
|
</p><p>
|
559 |
|
|
Of particular interest is extending testing to algorithms and
|
560 |
|
|
then to parallel algorithms. Also io and locales.
|
561 |
|
|
</p><p>
|
562 |
|
|
The test instrumentation should also be extended to add
|
563 |
|
|
instrumentation to <code class="classname">iterator</code>
|
564 |
|
|
and <code class="classname">const_iterator</code> types that throw
|
565 |
|
|
conditionally on iterator operations.
|
566 |
|
|
</p></li></ul></div></div><div class="section" title="C++11 Requirements Test Sequence Descriptions"><div class="titlepage"><div><div><h5 class="title"><a id="test.exception.safety.containers"/>
|
567 |
|
|
C++11 Requirements Test Sequence Descriptions
|
568 |
|
|
</h5></div></div></div><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
|
569 |
|
|
Basic
|
570 |
|
|
</p><p>
|
571 |
|
|
Basic consistency on exception propagation tests. For
|
572 |
|
|
each container, an object of that container is constructed,
|
573 |
|
|
a specific member function is exercised in
|
574 |
|
|
a <code class="literal">try</code> block, and then any thrown
|
575 |
|
|
exceptions lead to error checking in the appropriate
|
576 |
|
|
<code class="literal">catch</code> block. The container's use of
|
577 |
|
|
resources is compared to the container's use prior to the
|
578 |
|
|
test block. Resource monitoring is limited to allocations
|
579 |
|
|
made through the container's <span class="type">allocator_type</span>,
|
580 |
|
|
which should be sufficient for container data
|
581 |
|
|
structures. Included in these tests are member functions
|
582 |
|
|
are <span class="type">iterator</span> and <span class="type">const_iterator</span>
|
583 |
|
|
operations, <code class="function">pop_front</code>, <code class="function">pop_back</code>, <code class="function">push_front</code>, <code class="function">push_back</code>, <code class="function">insert</code>, <code class="function">erase</code>, <code class="function">swap</code>, <code class="function">clear</code>,
|
584 |
|
|
and <code class="function">rehash</code>. The container in question is
|
585 |
|
|
instantiated with two instrumented template arguments,
|
586 |
|
|
with <code class="classname">__gnu_cxx::throw_allocator_limit</code>
|
587 |
|
|
as the allocator type, and
|
588 |
|
|
with <code class="classname">__gnu_cxx::throw_type_limit</code> as
|
589 |
|
|
the value type. This allows the test to loop through
|
590 |
|
|
conditional throw points.
|
591 |
|
|
</p><p>
|
592 |
|
|
The general form is demonstrated in
|
593 |
|
|
<code class="filename">testsuite/23_containers/list/requirements/exception/basic.cc
|
594 |
|
|
</code>. The instantiating test object is <code class="classname">__gnu_test::basic_safety</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>.
|
595 |
|
|
</p></li><li class="listitem"><p>
|
596 |
|
|
Generation Prohibited
|
597 |
|
|
</p><p>
|
598 |
|
|
Exception generation tests. For each container, an object of
|
599 |
|
|
that container is constructed and all member functions
|
600 |
|
|
required to not throw exceptions are exercised. Included in
|
601 |
|
|
these tests are member functions
|
602 |
|
|
are <span class="type">iterator</span> and <span class="type">const_iterator</span> operations, <code class="function">erase</code>, <code class="function">pop_front</code>, <code class="function">pop_back</code>, <code class="function">swap</code>,
|
603 |
|
|
and <code class="function">clear</code>. The container in question is
|
604 |
|
|
instantiated with two instrumented template arguments,
|
605 |
|
|
with <code class="classname">__gnu_cxx::throw_allocator_random</code>
|
606 |
|
|
as the allocator type, and
|
607 |
|
|
with <code class="classname">__gnu_cxx::throw_type_random</code> as
|
608 |
|
|
the value type. This test does not loop, an instead is sudden
|
609 |
|
|
death: first error fails.
|
610 |
|
|
</p><p>
|
611 |
|
|
The general form is demonstrated in
|
612 |
|
|
<code class="filename">testsuite/23_containers/list/requirements/exception/generation_prohibited.cc
|
613 |
|
|
</code>. The instantiating test object is <code class="classname">__gnu_test::generation_prohibited</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>.
|
614 |
|
|
</p></li><li class="listitem"><p>
|
615 |
|
|
Propagation Consistent
|
616 |
|
|
</p><p>
|
617 |
|
|
Container rollback on exception propagation tests. For
|
618 |
|
|
each container, an object of that container is constructed,
|
619 |
|
|
a specific member function that requires rollback to a previous
|
620 |
|
|
known good state is exercised in
|
621 |
|
|
a <code class="literal">try</code> block, and then any thrown
|
622 |
|
|
exceptions lead to error checking in the appropriate
|
623 |
|
|
<code class="literal">catch</code> block. The container is compared to
|
624 |
|
|
the container's last known good state using such parameters
|
625 |
|
|
as size, contents, and iterator references. Included in these
|
626 |
|
|
tests are member functions
|
627 |
|
|
are <code class="function">push_front</code>, <code class="function">push_back</code>, <code class="function">insert</code>,
|
628 |
|
|
and <code class="function">rehash</code>. The container in question is
|
629 |
|
|
instantiated with two instrumented template arguments,
|
630 |
|
|
with <code class="classname">__gnu_cxx::throw_allocator_limit</code>
|
631 |
|
|
as the allocator type, and
|
632 |
|
|
with <code class="classname">__gnu_cxx::throw_type_limit</code> as
|
633 |
|
|
the value type. This allows the test to loop through
|
634 |
|
|
conditional throw points.
|
635 |
|
|
</p><p>
|
636 |
|
|
The general form demonstrated in
|
637 |
|
|
<code class="filename">testsuite/23_containers/list/requirements/exception/propagation_coherent.cc
|
638 |
|
|
</code>. The instantiating test object is <code class="classname">__gnu_test::propagation_coherent</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>.
|
639 |
|
|
</p></li></ul></div></div></div></div></div><div class="navfooter"><hr/><table width="100%" summary="Navigation footer"><tr><td align="left"><a accesskey="p" href="internals.html">Prev</a> </td><td align="center"><a accesskey="u" href="appendix_porting.html">Up</a></td><td align="right"> <a accesskey="n" href="abi.html">Next</a></td></tr><tr><td align="left" valign="top">Porting to New Hardware or Operating Systems </td><td align="center"><a accesskey="h" href="../index.html">Home</a></td><td align="right" valign="top"> ABI Policy and Guidelines</td></tr></table></div></body></html>
|