Class TokenIterator

java.lang.Object
com.ibm.icu.impl.data.TokenIterator

public class TokenIterator extends Object
An iterator class that returns successive string tokens from some source. String tokens are, in general, separated by Pattern_White_Space in the source test. Furthermore, they may be delimited by either single or double quotes (opening and closing quotes must match). Escapes are processed using standard ICU unescaping.

2015-sep-03 TODO: Only used in com.ibm.icu.dev.test.format, move there.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    private StringBuffer
     
    private boolean
     
    private int
     
    private String
     
    private int
     
     
  • Constructor Summary

    Constructors
    Constructor
    Description
    Construct an iterator over the tokens returned by the given ResourceReader, ignoring blank lines and comment lines (first non-blank character is '#').
  • Method Summary

    Modifier and Type
    Method
    Description
    Return a string description of the position of the last line returned by readLine() or readLineSkippingComments().
    int
    Return the one-based line number of the line of the last token returned by next().
    Return the next token from this iterator, or null if the last token has been returned.
    private int
    nextToken(int position)
    Read the next token from 'this.line' and append it to 'this.buf'.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

    • reader

      private ResourceReader reader
    • line

      private String line
    • buf

      private StringBuffer buf
    • done

      private boolean done
    • pos

      private int pos
    • lastpos

      private int lastpos
  • Constructor Details

    • TokenIterator

      public TokenIterator(ResourceReader r)
      Construct an iterator over the tokens returned by the given ResourceReader, ignoring blank lines and comment lines (first non-blank character is '#'). Note that trailing comments on a line, beginning with the first unquoted '#', are recognized.
  • Method Details

    • next

      public String next() throws IOException
      Return the next token from this iterator, or null if the last token has been returned.
      Throws:
      IOException
    • getLineNumber

      public int getLineNumber()
      Return the one-based line number of the line of the last token returned by next(). Should only be called after a call to next(); otherwise the return value is undefined.
    • describePosition

      public String describePosition()
      Return a string description of the position of the last line returned by readLine() or readLineSkippingComments().
    • nextToken

      private int nextToken(int position)
      Read the next token from 'this.line' and append it to 'this.buf'. Tokens are separated by Pattern_White_Space. Tokens may also be delimited by double or single quotes. The closing quote must match the opening quote. If a '#' is encountered, the rest of the line is ignored, unless it is backslash-escaped or within quotes.
      Parameters:
      position - the offset into the string
      Returns:
      offset to the next character to read from line, or if the end of the line is reached without scanning a valid token, -1