StringTokenizer Class Reference

#include <StringTokenizer.h>

---

Detailed Description

StringTokenizer A class similar to the StringTokenizer from Java. It splits a string at the given string or character (or one of the special cases NEWLINE or WHITECHAR) and allows to iterate over the so generated substrings.

The normal usage is like this:

 StringTokenizer st(CString("This is a line"), ' ');
 while(st.hasNext())
    cout << st.next() << endl;
 

This would generate the output:

 This
 is
 a
 line
 

There is something to know about the behaviour: When using WHITECHAR, a list of whitechars occuring in the string to split is regarded as a single divider. All other parameter will use multiple occurences of operators as a list of single divider and the string between them will have a length of zero.

Definition at line 67 of file StringTokenizer.h.

Public Member Functions
std::string	front ()
std::string	get (size_t pos) const
std::vector< std::string >	getVector ()
bool	hasNext ()
std::string	next ()
void	reinit ()
size_t	size () const
	StringTokenizer (std::string tosplit, int special)
	constructor When StringTokenizer.NEWLINE is used as second parameter, the string will be split at all occurences of a newline character (0x0d / 0x0a) When StringTokenizer.WHITECHARS is used as second parameter, the string will be split at all characters below 0x20 All other ints specified as second parameter are casted int o a char at which the string will be splitted.
	StringTokenizer (std::string tosplit, std::string token, bool splitAtAllChars=false)
	constructor the first string will be split at the second string's occurences. If the optional third parameter is true, the string will be split whenever a char from the second string occurs. If the string between two split positions is empty, it will nevertheless be returned.
	StringTokenizer (std::string tosplit)
	constructor same as StringTokenizer(tosplit, StringTokenizer.WHITECHARS) tosplit is the string to split into substrings. If the string between two split positions is empty, it will not be returned.
	StringTokenizer ()
	~StringTokenizer ()
Static Public Attributes
static const int	NEWLINE = -256
static const int	WHITECHARS = -257
Private Types
typedef std::vector< size_t >	SizeVector
Private Member Functions
void	prepare (const std::string &tosplit, const std::string &token, bool splitAtAllChars)
void	prepareWhitechar (const std::string &tosplit)
Private Attributes
SizeVector	myLengths
size_t	myPos
SizeVector	myStarts
std::string	myTosplit

---

Member Typedef Documentation

typedef std::vector<size_t> StringTokenizer::SizeVector [private]

a list of positions/lengths

Definition at line 137 of file StringTokenizer.h.

---

Constructor & Destructor Documentation

StringTokenizer::StringTokenizer

(

)

[inline]

default constructor

Definition at line 78 of file StringTokenizer.h.

{ }

StringTokenizer::StringTokenizer

(

std::string

tosplit

)

constructor same as StringTokenizer(tosplit, StringTokenizer.WHITECHARS) tosplit is the string to split into substrings. If the string between two split positions is empty, it will not be returned.

Definition at line 51 of file StringTokenizer.cpp.

References prepareWhitechar().

        : myTosplit(tosplit), myPos(0) {
    prepareWhitechar(tosplit);
}

StringTokenizer::StringTokenizer	(	std::string	tosplit,
		std::string	token,
		bool	splitAtAllChars = false
	)

constructor the first string will be split at the second string's occurences. If the optional third parameter is true, the string will be split whenever a char from the second string occurs. If the string between two split positions is empty, it will nevertheless be returned.

Definition at line 57 of file StringTokenizer.cpp.

References prepare().

        : myTosplit(tosplit), myPos(0) {
    prepare(tosplit, token, splitAtAllChars);
}

StringTokenizer::StringTokenizer	(	std::string	tosplit,
		int	special
	)

constructor When StringTokenizer.NEWLINE is used as second parameter, the string will be split at all occurences of a newline character (0x0d / 0x0a) When StringTokenizer.WHITECHARS is used as second parameter, the string will be split at all characters below 0x20 All other ints specified as second parameter are casted int o a char at which the string will be splitted.

Definition at line 63 of file StringTokenizer.cpp.

References NEWLINE, prepare(), prepareWhitechar(), and WHITECHARS.

        : myTosplit(tosplit), myPos(0) {
    switch (special) {
    case NEWLINE:
        prepare(tosplit, "\r\n", true);
        break;
    case WHITECHARS:
        prepareWhitechar(tosplit);
        break;
    default:
        char *buf = new char[2];
        buf[0] = (char) special;
        buf[1] = 0;
        prepare(tosplit, buf, false);
        delete[] buf;
        break;
    }
}

StringTokenizer::~StringTokenizer

(

)

destructor

Definition at line 83 of file StringTokenizer.cpp.

{}

---

Member Function Documentation

std::string StringTokenizer::front

(

)

returns the first substring without moving the iterator

Definition at line 106 of file StringTokenizer.cpp.

References myLengths, myStarts, and myTosplit.

Referenced by TEST().

                                 {
    if (myStarts.size()==0) {
        throw OutOfBoundsException();
    }
    if (myLengths[0]==0) {
        return "";
    }
    return myTosplit.substr(myStarts[0],myLengths[0]);
}

std::string StringTokenizer::get

(

size_t

pos

)

const

returns the item at the given position

Definition at line 116 of file StringTokenizer.cpp.

References myLengths, myStarts, and myTosplit.

Referenced by NamedColumnsParser::get(), and TEST().

                                               {
    if (pos>=myStarts.size()) {
        throw OutOfBoundsException();
    }
    if (myLengths[pos]==0) {
        return "";
    }
    size_t start = myStarts[pos];
    size_t length = myLengths[pos];
    return myTosplit.substr(start, length);
}

std::vector< std::string > StringTokenizer::getVector

(

)

Definition at line 180 of file StringTokenizer.cpp.

References hasNext(), next(), reinit(), and size().

Referenced by OptionsCont::getStringVector(), PCLoaderDlrNavteq::loadPolyFile(), MSRouteHandler::myStartElement(), MSEdge::parseEdgesList(), NIImporter_DlrNavteq::TrafficlightsHandler::report(), NIImporter_DlrNavteq::EdgesHandler::report(), and TEST().

                           {
    std::vector<std::string> ret;
    ret.reserve(size());
    while (hasNext()) {
        ret.push_back(next());
    }
    reinit();
    return ret;
}

bool StringTokenizer::hasNext

(

)

returns the information whether further substrings exist

Definition at line 89 of file StringTokenizer.cpp.

References myPos, and myStarts.

Referenced by NLHandler::addIncomingLanes(), NLDetectorBuilder::createEdgeLaneMeanData(), getVector(), RORDLoader_SUMOBase::myCharacters(), ROJTRTurnDefLoader::myStartElement(), MSRouteHandler::openRouteDistribution(), MSRouteHandler::openVehicleTypeDistribution(), GeomConvHelper::parseShapeReporting(), SUMOSAXAttributes::parseStringVector(), parseVehicleClasses(), readV(), NamedColumnsParser::reinitMap(), Option_IntVector::set(), and TEST().

                              {
    return myPos!=myStarts.size();
}

std::string StringTokenizer::next

(

)

returns the next substring when it exists. Otherwise the behaviour is undefined

Definition at line 93 of file StringTokenizer.cpp.

References myLengths, myPos, myStarts, and myTosplit.

Referenced by NLHandler::addIncomingLanes(), NLDetectorBuilder::createEdgeLaneMeanData(), getVector(), PCLoaderVisum::load(), RORDLoader_SUMOBase::myCharacters(), ROJTRTurnDefLoader::myStartElement(), MSRouteHandler::openRouteDistribution(), MSRouteHandler::openVehicleTypeDistribution(), GeomConvHelper::parseBoundaryReporting(), RGBColor::parseColor(), NIXMLConnectionsHandler::parseLaneBound(), GeomConvHelper::parseShapeReporting(), SUMOSAXAttributes::parseStringVector(), parseTimeLine(), parseVehicleClasses(), readO(), readTime(), readV(), NamedColumnsParser::reinitMap(), NIImporter_DlrNavteq::EdgesHandler::report(), Option_IntVector::set(), and TEST().

                                {
    if (myPos>=myStarts.size()) {
        throw OutOfBoundsException();
    }
    if (myLengths[myPos]==0) {
        myPos++;
        return "";
    }
    size_t start = myStarts[myPos];
    size_t length = myLengths[myPos++];
    return myTosplit.substr(start,length);
}

void StringTokenizer::prepare	(	const std::string &	tosplit,
		const std::string &	token,
		bool	splitAtAllChars
	)			[private]

splits the first string at all occurences of the second. If the third parameter is true split at all chars given in the second

Definition at line 133 of file StringTokenizer.cpp.

References myLengths, and myStarts.

Referenced by StringTokenizer().

                                                                                                    {
    size_t beg = 0;
    size_t len = token.length();
    if (splitAtAllChars) {
        len = 1;
    }
    while (beg<tosplit.length()) {
        size_t end;
        if (splitAtAllChars) {
            end = tosplit.find_first_of(token, beg);
        } else {
            end = tosplit.find(token, beg);
        }
        if (end == std::string::npos) {
            end = tosplit.length();
        }
        myStarts.push_back(beg);
        myLengths.push_back(end-beg);
        beg = end + len;
        if (beg==tosplit.length()) {
            myStarts.push_back(beg-1);
            myLengths.push_back(0);
        }
    }
}

void StringTokenizer::prepareWhitechar

(

const std::string &

tosplit

)

[private]

splits the first string at all occurences of whitechars

Definition at line 159 of file StringTokenizer.cpp.

References myLengths, and myStarts.

Referenced by StringTokenizer().

                                                               {
    size_t len = tosplit.length();
    size_t beg = 0;
    while (beg<len&&tosplit[beg]<=32) {
        beg++;
    }
    while (beg!=std::string::npos&&beg<len) {
        size_t end = beg;
        while (end<len&&tosplit[end]>32) {
            end++;
        }
        myStarts.push_back(beg);
        myLengths.push_back(end-beg);
        beg = end;
        while (beg<len&&tosplit[beg]<=32) {
            beg++;
        }
    }
}

void StringTokenizer::reinit

(

)

reinitialises the internal iterator

Definition at line 85 of file StringTokenizer.cpp.

References myPos.

Referenced by getVector(), and TEST().

                             {
    myPos = 0;
}

size_t StringTokenizer::size

(

)

const

returns the number of existing substrings

Definition at line 129 of file StringTokenizer.cpp.

References myStarts.

Referenced by NamedColumnsParser::get(), getVector(), NamedColumnsParser::know(), GeomConvHelper::parseBoundaryReporting(), RGBColor::parseColor(), NIXMLConnectionsHandler::parseLaneBound(), parseTimeLine(), readO(), and TEST().

                                   {
    return myStarts.size();
}

---

Field Documentation

SizeVector StringTokenizer::myLengths [private]

the list of substring lengths

Definition at line 149 of file StringTokenizer.h.

Referenced by front(), get(), next(), prepare(), and prepareWhitechar().

size_t StringTokenizer::myPos [private]

the current position in the list of substrings

Definition at line 143 of file StringTokenizer.h.

Referenced by hasNext(), next(), and reinit().

SizeVector StringTokenizer::myStarts [private]

the list of substring starts

Definition at line 146 of file StringTokenizer.h.

Referenced by front(), get(), hasNext(), next(), prepare(), prepareWhitechar(), and size().

std::string StringTokenizer::myTosplit [private]

the string to split

Definition at line 140 of file StringTokenizer.h.

Referenced by front(), get(), and next().

const int StringTokenizer::NEWLINE = -256 [static]

identifier for splitting the given string at all newline characters

Definition at line 70 of file StringTokenizer.h.

Referenced by StringTokenizer(), and TEST().

const int StringTokenizer::WHITECHARS = -257 [static]

identifier for splitting the given string at all whitespace characters

Definition at line 74 of file StringTokenizer.h.

Referenced by ROJTRTurnDefLoader::myStartElement(), readO(), readTime(), readV(), NIImporter_DlrNavteq::TrafficlightsHandler::report(), NIImporter_DlrNavteq::EdgesHandler::report(), StringTokenizer(), and TEST().

---

The documentation for this class was generated from the following files:

• StringTokenizer.h
• StringTokenizer.cpp