Class ColumnReader

java.lang.Object
dev.hardwood.reader.ColumnReader
All Implemented Interfaces:
Closeable, AutoCloseable
@Experimental public class ColumnReader extends Object implements Closeable

Batch-oriented column reader for reading a single column across all row groups.

Exposes a column's batch as typed leaf values plus a layer-model view of the schema chain between root and leaf. Each node along the chain contributes zero or one LayerKind layer:

Layers are numbered 0..getLayerCount() - 1 outermost-to-innermost. A flat column (no enclosing nullable groups, no repetition) reports getLayerCount() == 0 and is queried solely through getLeafValidity() plus the typed value accessors.

Polarity: validity bitmaps carry set bit = present semantics. Validity.NO_NULLS is the sparse representation of "every item at that scope is present in the current batch."

Real items only. Layer offsets and the leaf array are sized to real-items-only counts. Phantom positions for null/empty parents are excluded; getLayerOffsets(k)[i+1] - getLayerOffsets(k)[i] == 0 distinguishes empty from null at REPEATED layers (validity carries the null bit).

Array ownership. Every array and Validity handed back by an accessor (getInts(), getLongs(), getLayerOffsets(int), getLeafValidity(), and the rest) belongs to the current batch and is freshly allocated by the call that produced it — nextBatch(), or ColumnReaders.nextBatch() for a reader of a group — except Validity.NO_NULLS, a shared immutable singleton. getBinaryDictionary() returns an object whose entries never change, the same for every batch drawn from one dictionary. A later advance never reuses or overwrites an array returned for an earlier batch — so a returned array may be kept and read after the reader has advanced, including handed off to another thread for processing. The reader itself is still a single-threaded cursor: only one consumer thread may advance it. (The note on getBinaryValues() that the array is not sized to the values is about its length, not reuse; that buffer is fresh per batch too.) Treat returned arrays as read-only: the accessors of one batch share arrays (the dictionary ids, for instance, back getStrings()'s reuse of Strings), so changing one can change what another accessor of the batch returns.

This API is Experimental: the shape of the batch accessors and layer representation may change in future releases without prior deprecation.

  • Method Details

    • nextBatch

      public boolean nextBatch() throws IOException

      Advance to the next batch.

      One reader advances only itself. Readers built separately — each from its own ParquetFileReader.buildColumnReader(String) — are independent cursors, and the batch a reader hands out is sized for the columns it reads. A batch of a BYTE_ARRAY column therefore covers fewer rows than a batch of an INT32 one over the same file, and a filtered reader is sized for its predicate's columns as well as its own. Two such readers reach different rows on their nth batch.

      So do not pair them: while (a.nextBatch() & b.nextBatch()) ends when whichever reader has the larger batches runs out first, and the values it took from each on any given turn are from different rows. Neither reader reports anything wrong, because neither is wrong — each is a complete, correct read of its own column.

      To read several columns of the same rows, use ColumnReaders, from ParquetFileReader.buildColumnReaders(dev.hardwood.schema.ColumnProjection). Its readers share one decode pipeline and one batch size, and ColumnReaders.nextBatch() advances all of them. A reader obtained from a ColumnReaders shows the group's current batch and does not advance on its own: this method throws IllegalStateException for it.

      Returns:
      true if a batch is available, false if exhausted
      Throws:
      IOException - if the bytes could not be read
      ParquetReadException - if the file's bytes are not what a Parquet file can say: a footer or a page index that will not parse, a dictionary page the metadata places outside its column chunk, a page whose checksum fails, values that do not decode under the encoding declared for them. In a multi-file read this covers a later file that is not Parquet at all, or whose schema cannot be reconciled with the first file's
      IllegalStateException - if this reader was closed, or was obtained from a ColumnReaders

    • getRecordCount

      public int getRecordCount()
      Number of top-level records in the current batch.

    • getValueCount

      public int getValueCount()
      Total number of leaf values in the current batch — sized to real items only (phantom slots from null/empty parents are excluded). For flat columns this equals getRecordCount(). For a repeated column it is not bounded by the configured batch size — the batch size caps records, not leaf values — so a batch may carry more leaf values than records.

    • getLayerCount

      public int getLayerCount()
      Number of layers in this column's schema chain. 0 for a flat column. Stable for the lifetime of this reader and safe to call before the first nextBatch() — useful for sizing consumer-side buffers.

    • getLayerKind

      public LayerKind getLayerKind(int layer)
      Layer kind at layer. Stable for the lifetime of this reader and safe to call before the first nextBatch().

    • getLayerValidity

      public Validity getLayerValidity(int layer)
      Validity at layer. Returns Validity.NO_NULLS when no item at that layer is null in the current batch (the sparse fast path); otherwise returns a wrapper over the per-item null bitmap.

    • getLayerOffsets

      public int[] getLayerOffsets(int layer)
      Offsets at layer. Length == count(layer) + 1, with offsets[count(layer)] equal to count(layer + 1) (or to getValueCount() for the innermost layer). offsets[i+1] - offsets[i] == 0 denotes an empty list/map.
      Throws:
      IllegalStateException - if layer is outside [0, getLayerCount()) or if the layer is not LayerKind.REPEATED

    • getLeafValidity

      public Validity getLeafValidity()
      Validity over the leaf-value array, indexed 0..getValueCount(). Returns Validity.NO_NULLS when no leaf in the current batch is null.

    • getInts

      public int[] getInts()

    • getLongs

      public long[] getLongs()

    • getFloats

      public float[] getFloats()

    • getDoubles

      public double[] getDoubles()

    • getBooleans

      public boolean[] getBooleans()

    • getBinaryValues

      public byte[] getBinaryValues()

      The bytes a varlength leaf's values are read from. Value i occupies [getBinaryStarts()[i], getBinaryEnds()[i]).

      Values need not be contiguous or in value order, and several may cover the same bytes, as values of a dictionary-encoded column naming the same dictionary entry may. The array is not sized to the values; bytes outside every value's range are unspecified.

      Throws:
      IllegalStateException - for non-byte-array leaves

    • getBinaryStarts

      public int[] getBinaryStarts()
      Start of each value in getBinaryValues(), inclusive. Length == getValueCount(). A null value's range is empty.
      Throws:
      IllegalStateException - for non-byte-array leaves

    • getBinaryEnds

      public int[] getBinaryEnds()
      End of each value in getBinaryValues(), exclusive. Length == getValueCount(); the byte length of value i is ends[i] - starts[i].
      Throws:
      IllegalStateException - for non-byte-array leaves

    • getDictionaryIds

      public int[] getDictionaryIds()

      Each value's entry in the batch's dictionary: value i is entry getDictionaryIds()[i] of getBinaryDictionary(), and -1 at a null value. Length == getValueCount().

      Returns null unless every non-null value of the batch was decoded from the column chunk's dictionary: for every batch of a column without a dictionary, and, where a writer switched to plain encoding after its dictionary filled up, for the batch holding the switch and the rest of that row group. Read such a batch through the value accessors. A batch never draws on two dictionaries: a read that includes a binary column ends its batches at row-group boundaries. Dictionary ids are exposed for BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY and INT96 columns; for any other column this returns null.

    • getBinaryDictionary

      public BinaryDictionary getBinaryDictionary()
      The dictionary getDictionaryIds() refers to, or null when the ids are null. Every batch drawn from the same dictionary returns the same instance, so comparing it with the previous batch's (!=) tells when the dictionary changed.
      Throws:
      IllegalStateException - for non-byte-array leaves

    • getBinaries

      public byte[][] getBinaries()

      Materialises one byte[] per leaf value, copying out of the binary buffer. Returns null at indexes where getLeafValidity() is unset. Allocates one byte array per leaf — hot loops should consult getBinaryValues() with getBinaryStarts() and getBinaryEnds() directly.

      The returned array has length getValueCount() — i.e. the real leaf count, not getRecordCount(). For a flat column the two coincide; for list<binary> and similar nested chains they differ, and lookups must go through the appropriate layer offsets rather than indexing by record.

    • getStrings

      public String[] getStrings()

      Convenience: materialises one String per leaf value by UTF-8 decoding the slice of getBinaryValues() for each entry. Returns null at indexes where getLeafValidity() is unset.

      The column has to hold text: a BYTE_ARRAY annotated STRING, ENUM or JSON, or one carrying no annotation. Every other binary column — a DECIMAL, a UUID, a BSON, an INT96 — stores bytes that stand for something other than characters, and reading them as text is refused with an IllegalArgumentException; use getBinaries() / getBinaryValues() for those.

      The returned array has length getValueCount() — i.e. the real leaf count, not getRecordCount(). For a flat column the two coincide; for list<string> and similar nested chains they differ, and lookups must go through the appropriate layer offsets rather than indexing by record.

      Throws:
      IllegalArgumentException - if the column does not hold text

    • getColumnSchema

      public ColumnSchema getColumnSchema()

    • close

      public void close() throws IOException
      Releases the resources held by this reader. Idempotent: calling it more than once has no further effect. A reader obtained from a ColumnReaders holds no resources of its own, and closing it has no effect; ColumnReaders.close() releases the group.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException