| 1 |
424 |
jeremybenn |
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
| 2 |
|
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
| 3 |
|
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Tokenizing</title><meta name="generator" content="DocBook XSL Stylesheets V1.75.2" /><meta name="keywords" content=" ISO C++ , library " /><link rel="home" href="../spine.html" title="The GNU C++ Library Documentation" /><link rel="up" href="bk01pt05ch13.html" title="Chapter 13. String Classes" /><link rel="prev" href="bk01pt05ch13s03.html" title="Arbitrary Character Types" /><link rel="next" href="bk01pt05ch13s05.html" title="Shrink to Fit" /></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Tokenizing</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="bk01pt05ch13s03.html">Prev</a> </td><th width="60%" align="center">Chapter 13. String Classes</th><td width="20%" align="right"> <a accesskey="n" href="bk01pt05ch13s05.html">Next</a></td></tr></table><hr /></div><div class="sect1" title="Tokenizing"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="strings.string.token"></a>Tokenizing</h2></div></div></div><p>
|
| 4 |
|
|
</p><p>The Standard C (and C++) function <code class="code">strtok()</code> leaves a lot to
|
| 5 |
|
|
be desired in terms of user-friendliness. It's unintuitive, it
|
| 6 |
|
|
destroys the character string on which it operates, and it requires
|
| 7 |
|
|
you to handle all the memory problems. But it does let the client
|
| 8 |
|
|
code decide what to use to break the string into pieces; it allows
|
| 9 |
|
|
you to choose the "whitespace," so to speak.
|
| 10 |
|
|
</p><p>A C++ implementation lets us keep the good things and fix those
|
| 11 |
|
|
annoyances. The implementation here is more intuitive (you only
|
| 12 |
|
|
call it once, not in a loop with varying argument), it does not
|
| 13 |
|
|
affect the original string at all, and all the memory allocation
|
| 14 |
|
|
is handled for you.
|
| 15 |
|
|
</p><p>It's called stringtok, and it's a template function. Sources are
|
| 16 |
|
|
as below, in a less-portable form than it could be, to keep this
|
| 17 |
|
|
example simple (for example, see the comments on what kind of
|
| 18 |
|
|
string it will accept).
|
| 19 |
|
|
</p><pre class="programlisting">
|
| 20 |
|
|
#include <string>
|
| 21 |
|
|
template <typename Container>
|
| 22 |
|
|
void
|
| 23 |
|
|
stringtok(Container &container, string const &in,
|
| 24 |
|
|
const char * const delimiters = " \t\n")
|
| 25 |
|
|
{
|
| 26 |
|
|
const string::size_type len = in.length();
|
| 27 |
|
|
string::size_type i = 0;
|
| 28 |
|
|
|
| 29 |
|
|
while (i < len)
|
| 30 |
|
|
{
|
| 31 |
|
|
// Eat leading whitespace
|
| 32 |
|
|
i = in.find_first_not_of(delimiters, i);
|
| 33 |
|
|
if (i == string::npos)
|
| 34 |
|
|
return; // Nothing left but white space
|
| 35 |
|
|
|
| 36 |
|
|
// Find the end of the token
|
| 37 |
|
|
string::size_type j = in.find_first_of(delimiters, i);
|
| 38 |
|
|
|
| 39 |
|
|
// Push token
|
| 40 |
|
|
if (j == string::npos)
|
| 41 |
|
|
{
|
| 42 |
|
|
container.push_back(in.substr(i));
|
| 43 |
|
|
return;
|
| 44 |
|
|
}
|
| 45 |
|
|
else
|
| 46 |
|
|
container.push_back(in.substr(i, j-i));
|
| 47 |
|
|
|
| 48 |
|
|
// Set up for next loop
|
| 49 |
|
|
i = j + 1;
|
| 50 |
|
|
}
|
| 51 |
|
|
}
|
| 52 |
|
|
</pre><p>
|
| 53 |
|
|
The author uses a more general (but less readable) form of it for
|
| 54 |
|
|
parsing command strings and the like. If you compiled and ran this
|
| 55 |
|
|
code using it:
|
| 56 |
|
|
</p><pre class="programlisting">
|
| 57 |
|
|
std::list<string> ls;
|
| 58 |
|
|
stringtok (ls, " this \t is\t\n a test ");
|
| 59 |
|
|
for (std::list<string>const_iterator i = ls.begin();
|
| 60 |
|
|
i != ls.end(); ++i)
|
| 61 |
|
|
{
|
| 62 |
|
|
std::cerr << ':' << (*i) << ":\n";
|
| 63 |
|
|
} </pre><p>You would see this as output:
|
| 64 |
|
|
</p><pre class="programlisting">
|
| 65 |
|
|
:this:
|
| 66 |
|
|
:is:
|
| 67 |
|
|
:a:
|
| 68 |
|
|
:test: </pre><p>with all the whitespace removed. The original <code class="code">s</code> is still
|
| 69 |
|
|
available for use, <code class="code">ls</code> will clean up after itself, and
|
| 70 |
|
|
<code class="code">ls.size()</code> will return how many tokens there were.
|
| 71 |
|
|
</p><p>As always, there is a price paid here, in that stringtok is not
|
| 72 |
|
|
as fast as strtok. The other benefits usually outweigh that, however.
|
| 73 |
|
|
</p><p><span class="emphasis"><em>Added February 2001:</em></span> Mark Wilden pointed out that the
|
| 74 |
|
|
standard <code class="code">std::getline()</code> function can be used with standard
|
| 75 |
|
|
<code class="code">istringstreams</code> to perform
|
| 76 |
|
|
tokenizing as well. Build an istringstream from the input text,
|
| 77 |
|
|
and then use std::getline with varying delimiters (the three-argument
|
| 78 |
|
|
signature) to extract tokens into a string.
|
| 79 |
|
|
</p></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="bk01pt05ch13s03.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="bk01pt05ch13.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="bk01pt05ch13s05.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Arbitrary Character Types </td><td width="20%" align="center"><a accesskey="h" href="../spine.html">Home</a></td><td width="40%" align="right" valign="top"> Shrink to Fit</td></tr></table></div></body></html>
|
