Package org.apache.hc.core5.util

Class Tokenizer

java.lang.Object

org.apache.hc.core5.util.Tokenizer

Direct Known Subclasses:

TokenParser

@Contract(threading=IMMUTABLE)
public class Tokenizer
extends Object

Tokenizer that can be used as a foundation for more complex parsing routines. Methods of this class are designed to produce near zero intermediate garbage and make no intermediate copies of input data.

This class is immutable and thread safe.

Since:

5.1

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Tokenizer.Cursor
static interface	Tokenizer.Delimiter	Represents a predicate whether the given character is a delimiter.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	CR
static final char	DQUOTE	Double quote
static final char	ESCAPE	Backward slash / escape character
static final int	HT
static final Tokenizer	INSTANCE
static final int	LF
static final int	SP

Constructor Summary

Constructors

Constructor	Description
Tokenizer()

Method Summary

Modifier and Type	Method	Description
void	copyContent(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet, StringBuilder dst)	Deprecated. Use copyContent(CharSequence, Cursor, Delimiter, StringBuilder)
void	copyContent(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate, StringBuilder dst)	Transfers content into the destination buffer until a whitespace character or any of the given delimiters is encountered.
void	copyQuotedContent(CharSequence buf, Tokenizer.Cursor cursor, StringBuilder dst)	Transfers content enclosed with quote marks into the destination buffer.
void	copyUnquotedContent(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet, StringBuilder dst)	Deprecated. Use copyUnquotedContent(CharSequence, Cursor, Delimiter, StringBuilder)
void	copyUnquotedContent(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate, StringBuilder dst)	Transfers content into the destination buffer until a whitespace character, a quote, or any of the given delimiters is encountered.
static Tokenizer.Delimiter	delimiters(char delimiter)
static Tokenizer.Delimiter	delimiters(char... delimiters)
static Tokenizer.Delimiter	delimiters(char delimiter1, char delimiter2)
static Tokenizer.Delimiter	delimiters(char delimiter1, char delimiter2, char delimiter3)
static Tokenizer.Delimiter	delimiters(BitSet delimiters)
static BitSet	INIT_BITSET(int... b)	Deprecated. Do not use.
static boolean	isWhitespace(char ch)
String	parseContent(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet)	Deprecated. use parseContent(CharSequence, Cursor, Delimiter)
String	parseContent(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate)	Extracts from the sequence of chars a token terminated with any of the given delimiters or a whitespace characters.
String	parseToken(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet)	Deprecated. use parseToken(CharSequence, Cursor, Delimiter)
String	parseToken(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate)	Extracts from the sequence of chars a token terminated with any of the given delimiters discarding semantically insignificant whitespace characters.
String	parseValue(CharSequence buf, Tokenizer.Cursor cursor, BitSet bitSet)	Deprecated. use parseValue(CharSequence, Cursor, Delimiter)
String	parseValue(CharSequence buf, Tokenizer.Cursor cursor, Tokenizer.Delimiter delimiterPredicate)	Extracts from the sequence of chars a value which can be enclosed in quote marks and terminated with any of the given delimiters discarding semantically insignificant whitespace characters.
void	skipWhiteSpace(CharSequence buf, Tokenizer.Cursor cursor)	Skips semantically insignificant whitespace characters and moves the cursor to the closest non-whitespace character.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DQUOTE

public static final char DQUOTE

Double quote

See Also:

• Constant Field Values

ESCAPE

public static final char ESCAPE

Backward slash / escape character

See Also:

• Constant Field Values

CR

public static final int CR

See Also:

• Constant Field Values

LF

public static final int LF

See Also:

• Constant Field Values

SP

public static final int SP

See Also:

• Constant Field Values

HT

public static final int HT

See Also:

• Constant Field Values

INSTANCE

public static final Tokenizer INSTANCE

Constructor Details

Tokenizer

public Tokenizer()

Method Details

INIT_BITSET

@Deprecated
public static BitSet INIT_BITSET(int... b)

Deprecated.

Do not use.

isWhitespace

public static boolean isWhitespace(char ch)

delimiters

public static Tokenizer.Delimiter delimiters(BitSet delimiters)

Since:

5.3

delimiters

public static Tokenizer.Delimiter delimiters(char... delimiters)

Since:

5.3

delimiters

public static Tokenizer.Delimiter delimiters(char delimiter)

Since:

5.3

delimiters

public static Tokenizer.Delimiter delimiters(char delimiter1,
 char delimiter2)

Since:

5.3

delimiters

public static Tokenizer.Delimiter delimiters(char delimiter1,
 char delimiter2,
 char delimiter3)

Since:

5.3

parseContent

public String parseContent(CharSequence buf,
 Tokenizer.Cursor cursor,
 Tokenizer.Delimiter delimiterPredicate)

Extracts from the sequence of chars a token terminated with any of the given delimiters or a whitespace characters.

Parameters:

buf - buffer with the sequence of chars to be parsed

cursor - defines the bounds and current position of the buffer

delimiterPredicate - delimiter predicate. Can be null if the token is not delimited by any character.

parseContent

@Deprecated
public String parseContent(CharSequence buf,
 Tokenizer.Cursor cursor,
 BitSet bitSet)

Deprecated.

use parseContent(CharSequence, Cursor, Delimiter)

parseToken

public String parseToken(CharSequence buf,
 Tokenizer.Cursor cursor,
 Tokenizer.Delimiter delimiterPredicate)

Extracts from the sequence of chars a token terminated with any of the given delimiters discarding semantically insignificant whitespace characters.

Parameters:

buf - buffer with the sequence of chars to be parsed

cursor - defines the bounds and current position of the buffer

delimiterPredicate - delimiter predicate. Can be null if the token is not delimited by any character.

parseToken

@Deprecated
public String parseToken(CharSequence buf,
 Tokenizer.Cursor cursor,
 BitSet bitSet)

Deprecated.

use parseToken(CharSequence, Cursor, Delimiter)

parseValue

public String parseValue(CharSequence buf,
 Tokenizer.Cursor cursor,
 Tokenizer.Delimiter delimiterPredicate)

Extracts from the sequence of chars a value which can be enclosed in quote marks and terminated with any of the given delimiters discarding semantically insignificant whitespace characters.

Parameters:

buf - buffer with the sequence of chars to be parsed

cursor - defines the bounds and current position of the buffer

delimiterPredicate - delimiter predicate. Can be null if the token is not delimited by any character.

parseValue

@Deprecated
public String parseValue(CharSequence buf,
 Tokenizer.Cursor cursor,
 BitSet bitSet)

Deprecated.

use parseValue(CharSequence, Cursor, Delimiter)

skipWhiteSpace

public void skipWhiteSpace(CharSequence buf,
 Tokenizer.Cursor cursor)

Skips semantically insignificant whitespace characters and moves the cursor to the closest non-whitespace character.

Parameters:

buf - buffer with the sequence of chars to be parsed

cursor - defines the bounds and current position of the buffer

copyContent

public void copyContent(CharSequence buf,
 Tokenizer.Cursor cursor,
 Tokenizer.Delimiter delimiterPredicate,
 StringBuilder dst)

Transfers content into the destination buffer until a whitespace character or any of the given delimiters is encountered.

Parameters:

buf - buffer with the sequence of chars to be parsed

cursor - defines the bounds and current position of the buffer

delimiterPredicate - delimiter predicate. Can be null if the token is delimited by a whitespace only.

dst - destination buffer

copyContent

@Deprecated
public void copyContent(CharSequence buf,
 Tokenizer.Cursor cursor,
 BitSet bitSet,
 StringBuilder dst)

Deprecated.

Use copyContent(CharSequence, Cursor, Delimiter, StringBuilder)

copyUnquotedContent

public void copyUnquotedContent(CharSequence buf,
 Tokenizer.Cursor cursor,
 Tokenizer.Delimiter delimiterPredicate,
 StringBuilder dst)

Transfers content into the destination buffer until a whitespace character, a quote, or any of the given delimiters is encountered.

Parameters:

buf - buffer with the sequence of chars to be parsed

cursor - defines the bounds and current position of the buffer

delimiterPredicate - delimiter predicate. Can be null if the token is delimited by a whitespace or a quote only.

dst - destination buffer

copyUnquotedContent

@Deprecated
public void copyUnquotedContent(CharSequence buf,
 Tokenizer.Cursor cursor,
 BitSet bitSet,
 StringBuilder dst)

Deprecated.

Use copyUnquotedContent(CharSequence, Cursor, Delimiter, StringBuilder)

copyQuotedContent

public void copyQuotedContent(CharSequence buf,
 Tokenizer.Cursor cursor,
 StringBuilder dst)

Transfers content enclosed with quote marks into the destination buffer.

Parameters:

buf - buffer with the sequence of chars to be parsed

cursor - defines the bounds and current position of the buffer

dst - destination buffer