totalcross.io

Class TokenReader

java.lang.Object

totalcross.io.LineReader

totalcross.io.TokenReader

public class TokenReader
extends LineReader

Used to read an array of tokens in a line ending with \r\n (enter/linefeed) from a stream. Consecutive newlines are skipped. This class does not work well with multi-byte characters when the second byte contains the delimiter or enter/linefeed.

The usual way to parse a CSV file is:

 do
   line = readline
   Convert.tokenizeString(line)
 

Using this class takes less memory, because the line is read in tokens. For example, suppose a line contains 200 chars, and splitting them contains 10 tokens of 20 chars each. Using the first approach (readline/tokenizeString), the readline will create a string with 200 chars, then that String will be tokenized into 10 smaller strings of 20 chars each.

Using this class, it will read the 10 tokens of 20 chars each directly, no longer having to create the temporary string of 200 chars.

The delimiter can be any character except for \r and \n. Note that two consecutive delimiters are considered a single token. So ;a;; is returned as {"","a","",""}.

Here's a sample that parses the input from a file:

 TokenReader reader = new TokenReader(new File("text.csv",File.READ_WRITE), ',');
 String[] tokens;
 while ((tokens = reader.readTokens()) != null)
 {
    ... do whatever you want with the tokens.
 }
 

And here's another sample that parses from a string:

 String lines = "a;;;a;\na;;;a; \nb;;b;b;b \nb;;b;b;b;\nb\nb;\n;b\n;\n b ;\n b \n  b \n b  \nb  \n  b";
 String ll[] = Convert.tokenizeString(lines,'\n');
 TokenReader tk = new TokenReader(new ByteArrayStream(lines.getBytes()),';');
 tk.doTrim = true;
 String []line;
 for (int j =0; ((line = tk.readTokens()) != null); j++)
 {
    Vm.debug('"'+ll[j]+'"');
    for (int i =0; i < line.length; i++)
       Vm.debug(i+": '"+line[i]+"'");
    Vm.debug("");
 }
 

The output is:

 "a;;;a;"
 0: 'a'
 1: ''
 2: ''
 3: 'a'
 4: ''
 
 "a;;;a; "
 0: 'a'
 1: ''
 2: ''
 3: 'a'
 4: ''
 
 "b;;b;b;b "
 0: 'b'
 1: ''
 2: 'b'
 3: 'b'
 4: 'b'
 
 "b;;b;b;b;"
 0: 'b'
 1: ''
 2: 'b'
 3: 'b'
 4: 'b'
 5: ''
 
 "b"
 0: 'b'
 
 "b;"
 0: 'b'
 1: ''
 
 ";b"
 0: ''
 1: 'b'
 
 ";"
 0: ''
 1: ''
 
 " b ;"
 0: 'b'
 1: ''
 
 " b "
 0: 'b'
 
 "  b "
 0: 'b'
 
 " b  "
 0: 'b'
 
 "b  "
 0: 'b'
 
 "  b"
 0: 'b'
 

Note that this class already uses a buffer for faster detection of the newline and delimiters. Don't use TokenReader with a BufferedStream, it's nonsense and it will throw a warning on the desktop.

Since:

TotalCross 1.23

Field Summary

Fields

Modifier and Type	Field and Description
protected char	delimiter

Fields inherited from class totalcross.io.LineReader

doTrim, f, maxTries, ofs, readBuf, returnEmptyLines

Constructor Summary

Constructors

Constructor and Description
TokenReader(Stream f, char delimiter) Constructs a new TokenReader and sets maxTries accordingly to the type of class: 10 if its a Socket or a PortConnector; 0, otherwise.
TokenReader(Stream f, char delimiter, byte[] buffer, int start, int len) Constructs a new TokenReader and sets maxTries accordingly to the type of class: 10 if its a Socket or a PortConnector, 0 otherwise.

Method Summary

Modifier and Type	Method and Description
java.lang.String	readLine() Cannot be used; throws a RuntimeException.
java.lang.String[]	readTokens() Returns the next tokens available in this stream or null if none.

Methods inherited from class totalcross.io.LineReader

getStream, readMore, reuse, setStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

delimiter

protected char delimiter

Constructor Detail

TokenReader

public TokenReader(Stream f,
                   char delimiter)
            throws IOException

Constructs a new TokenReader and sets maxTries accordingly to the type of class: 10 if its a Socket or a PortConnector; 0, otherwise.

Throws:

IOException

TokenReader

public TokenReader(Stream f,
                   char delimiter,
                   byte[] buffer,
                   int start,
                   int len)
            throws IOException

Constructs a new TokenReader and sets maxTries accordingly to the type of class: 10 if its a Socket or a PortConnector, 0 otherwise. The given buffer contents are added to the internal buffer to start reading from them.

Throws:

IOException

Since:

TotalCross 1.25

Method Detail

readLine

public java.lang.String readLine()

Cannot be used; throws a RuntimeException.

Overrides:

readLine in class LineReader

See Also:

readTokens()

readTokens

public java.lang.String[] readTokens()
                              throws IOException

Returns the next tokens available in this stream or null if none. Empty lines are skipped.

Throws:

IOException