|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.knime.core.util.tokenizer.Tokenizer
public class Tokenizer
This class reads tokens from a stream and returns them as strings.
You can specify token delimiters, comments and quotes.
The tokenizer can be configured to include delimiters or to return them as
separate tokens, to discard, include, or return comments, to allow line
continuations, and to combine consecutive delimiters.
It always returns tokens as strings.
It returns null
if it read EOF before any other character.
(EOF is always a token delimiter.)
It will always ignore a '\r' if it immediately is followed by a '\n'.
You can set multiple delimiter patterns.
You can specify multiple block comment begin/end pair patterns.
You can specify multiple line comment begin patterns.
And you can specify multiple quote begin/end pair patterns - and an escape
character with each pair.
A pattern is a (multi or single character) string.
The discard/return/include option can be specified for each delimiter and
comment pattern separately.
You can specify a line continuation character. This character immediately followed by a newline and any space or tab character will be ignored then inside a token or quoted string.
You can push back one (the last) token.
Field Summary | |
---|---|
(package private) static String |
LF_STR
String containing only the LF char. |
static int |
MAX_CHAR
The maximum ASCII code for the first character of patterns (like delimiter, comment, and quote patterns. |
Constructor Summary | |
---|---|
Tokenizer(Reader source)
Creates a new tokenizer with the default behaviour. |
Method Summary | |
---|---|
void |
closeSourceStream()
Closes the stream the tokenizer reads from. |
String |
getLastQuoteBeginPattern()
Returns the left quote of the last token. |
String |
getLastQuoteEndPattern()
Returns the right quote of the last token. |
int |
getLineNumber()
|
long |
getReadBytes()
Returns the number of bytes returned so far. |
TokenizerSettings |
getSettings()
|
boolean |
isWhiteSpace(char c)
|
boolean |
lastTokenWasDelimiter()
|
boolean |
lastTokenWasQuoted()
Call this to distinguish between missing and empty tokens. |
String |
nextToken()
Reads the next token from the stream and returns it as string. |
void |
pushBack()
After a call to this function the token returned with the last call to the nextToken() function will be returned once again with
the next call the the nextToken() function. |
void |
resetToDefault()
Resets the tokenizer to its default behavior, which is: No comments are supported, No quoted strings are supported, No line continuation is supported, and No token delimiter is set (only EOF - i.e. |
void |
setSettings(TokenizerSettings ftSettings)
Set new user settings in this tokenizer. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final int MAX_CHAR
static final String LF_STR
Constructor Detail |
---|
public Tokenizer(Reader source)
source
- A reader the tokens are read from.for what's the default behaviour.
Method Detail |
---|
public void resetToDefault()
public boolean lastTokenWasDelimiter()
public String nextToken() throws TokenizerException
null
if no more token can be read.
TokenizerException
- if something goes wrong during tokenizing.public boolean isWhiteSpace(char c)
c
- the character to test.
public void pushBack()
nextToken()
function will be returned once again with
the next call the the nextToken()
function. Pushing back a
token does not decrease the line number accordingly.
nextToken()
public boolean lastTokenWasQuoted()
true
if the last token had quotes which were
removed by the tokenizer.public String getLastQuoteBeginPattern()
public String getLastQuoteEndPattern()
public int getLineNumber()
public long getReadBytes()
public void closeSourceStream()
nextToken()
after a call to this token will return
null
(indicating the end of the file).
public void setSettings(TokenizerSettings ftSettings)
FileTokenizerSettings
, add all parameters there and pass
the settings object through this method.
ftSettings
- the settings object containing new settings.public TokenizerSettings getSettings()
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |