crp::StringTokenizer Class Reference

Class StringTokenizer implements a decomposition of a string into tokens separated by a set of delimiters. More...

#include <stringtokenizer.h>

Public Types
enum	WhitespaceMode { IGNORE_WHITESPACES, DELIMITERS, DELIMITERS_IGNORE }
	Whitespace treatement enumeration type. More...
Public Member Functions
	StringTokenizer (const std::string &str="", const std::string &delim="", const std::string &ls="\"", const std::string &le="\"", char esc= '\\', const std::string &comment="//")
	Starts a tokenizer on a string.
	~StringTokenizer ()
	Terminates the tokenizer.
bool	has_more_tokens ()
	Checks if there are mor tokens ahead.
std::string	next_token ()
	Returns the next token.
std::string	look_ahead (int n=1)
	Returns the token, which is n-tokens ahead.
std::string	get_rest_string ()
	Returns the rest of the parsed string.
const std::string &	get_input_string ()
	Returns the whole string.
void	append (const std::string &string)
	Appends a string to the input.
std::string	till_symbol (const std::string &symbol)
	Skips parsing till a symbol is reached.
void	reset (const std::string &string="", int position=0)
	Resets the parser.
int	get_row ()
	Returns the row position.
int	get_column ()
	Returns the column position.
int	get_position ()
	Returns the current position.
void	set_position (int npos)
	Sets the current position.
int	move (int number)
	Moves the tokenizer by the given number of characters.
bool	is_at_line_start ()
	Returns true if the tokenizer at the start of a new line.
void	set_whitespace_mode (WhitespaceMode status)
	Determines if white spaces are treated as delimiters.
void	set_ignore_delimiters (bool ignore)
	Determines the delimiters policy.
void	set_ignore_comments (bool status)
	Ignores comments.
void	set_ignore_literals (bool status)
	Sets the literal return property.
void	save ()
	Saves the current tokeinzer state.
void	restore ()
	Restores previously saved state.
void	ignore ()
	Ignores the previously saved state.
void	insert (const std::string &text)
	Inserts a string before the current position.

---

Detailed Description

Class StringTokenizer implements a decomposition of a string into tokens separated by a set of delimiters.

---

Member Enumeration Documentation

enum crp::StringTokenizer::WhitespaceMode

Whitespace treatement enumeration type.

Enumerator:

IGNORE_WHITESPACES
DELIMITERS
DELIMITERS_IGNORE

---

Constructor & Destructor Documentation

crp::StringTokenizer::StringTokenizer	(	const std::string &	str = "",
		const std::string &	delim = "",
		const std::string &	ls = "\"",
		const std::string &	le = "\"",
		char	esc = '\\',
		const std::string &	comment = "//"
	)

Starts a tokenizer on a string.

Parameters:

str	String to tokenize
delim	String of delimiters separated by spaces (white spaces are delimiters in any case if not ignored)
ls	Symbol which initiates a literal copy
le	Symbol which stops a literal copy
esc	Escape character
comment	Line comment string

crp::StringTokenizer::~StringTokenizer

(

)

[inline]

Terminates the tokenizer.

---

Member Function Documentation

void crp::StringTokenizer::append

(

const std::string &

string

)

Appends a string to the input.

Parameters:

string

String to append

int crp::StringTokenizer::get_column

(

)

[inline]

Returns the column position.

Returns:

Returns the position in the line of the current text

const std::string& crp::StringTokenizer::get_input_string

(

)

Returns the whole string.

Returns:

Parsed string

int crp::StringTokenizer::get_position

(

)

Returns the current position.

Returns:

Current position within the complete string

std::string crp::StringTokenizer::get_rest_string

(

)

Returns the rest of the parsed string.

Returns:

Unparsed part of the string

int crp::StringTokenizer::get_row

(

)

[inline]

Returns the row position.

Returns:

The line number of the current position in the text

bool crp::StringTokenizer::has_more_tokens

(

)

Checks if there are mor tokens ahead.

Returns:

true If there is at least one token more

void crp::StringTokenizer::ignore

(

)

Ignores the previously saved state.

void crp::StringTokenizer::insert

(

const std::string &

text

)

Inserts a string before the current position.

This effectively changes the input string.

Parameters:

text	Text to insert.

bool crp::StringTokenizer::is_at_line_start

(

)

[inline]

Returns true if the tokenizer at the start of a new line.

Returns:

True if there are now symbols before the tokenizer position or if the symbol preceding the tokenizer is a new line.

std::string crp::StringTokenizer::look_ahead

(

int

n = 1

)

Returns the token, which is n-tokens ahead.

Parameters:

n	How far to look

int crp::StringTokenizer::move

(

int

number

)

Moves the tokenizer by the given number of characters.

Parameters:

number

Number of characters

Returns:

Number of really moved characters

std::string crp::StringTokenizer::next_token

(

)

Returns the next token.

Returns:

Next token

void crp::StringTokenizer::reset	(	const std::string &	string = "",
		int	position = 0
	)

Resets the parser.

Parameters:

string	New string
position	New position

void crp::StringTokenizer::restore

(

)

Restores previously saved state.

void crp::StringTokenizer::save

(

)

Saves the current tokeinzer state.

You can use nested calls to save and restore.

void crp::StringTokenizer::set_ignore_comments

(

bool

status

)

[inline]

Ignores comments.

Parameters:

status

True if comments should be ignored (false by default)

void crp::StringTokenizer::set_ignore_delimiters

(

bool

ignore

)

[inline]

Determines the delimiters policy.

Parameters:

ignore

True if delimiters should be ignored (false by default)

void crp::StringTokenizer::set_ignore_literals

(

bool

status

)

[inline]

Sets the literal return property.

If set to false the tokenizer returns literal constants with literal delimiters and as pure strings else. In the second case it is not possible to decide if the token is a literal or not. The property is by default false.

Parameters:

status

True if literals should be ignored (false by default)

void crp::StringTokenizer::set_position

(

int

npos

)

Sets the current position.

Parameters:

npos	New position

void crp::StringTokenizer::set_whitespace_mode

(

WhitespaceMode

status

)

[inline]

Determines if white spaces are treated as delimiters.

If white spaces are not treated as delimiters only the delimiters from the delimiter string are used. If the property is DELIMITERS_IGNORE the white spaces are used as delimiters but never returned by the tokenizer.

Parameters:

ignore

Status of white space treatement

std::string crp::StringTokenizer::till_symbol

(

const std::string &

symbol

)

Skips parsing till a symbol is reached.

Returns:

Substring of the parsed string till symbol

---

The documentation for this class was generated from the following file:

• stringtokenizer.h