Class ColumnReader

java.lang.Object
dev.hardwood.reader.ColumnReader
All Implemented Interfaces:
AutoCloseable
@Experimental public class ColumnReader extends Object implements AutoCloseable

Batch-oriented column reader for reading a single column across all row groups.

Exposes a column's batch as typed leaf values plus a layer-model view of the schema chain between root and leaf. Each non-leaf node along the chain contributes zero or one LayerKind layer:

Layers are numbered 0..getLayerCount() - 1 outermost-to-innermost. A flat column (no enclosing nullable groups, no repetition) reports getLayerCount() == 0 and is queried solely through getLeafValidity() plus the typed value accessors.

Polarity: validity bitmaps carry set bit = present semantics. A null return is the sparse representation of "every item at that scope is present in the current batch."

Real items only. Layer offsets and the leaf array are sized to real-items-only counts. Phantom positions for null/empty parents are excluded; getLayerOffsets(k)[i+1] - getLayerOffsets(k)[i] == 0 distinguishes empty from null at REPEATED layers (validity carries the null bit).

Array ownership. Every array and Validity handed back by an accessor (getInts(), getLongs(), getLayerOffsets(int), getLeafValidity(), and the rest) belongs to the current batch and is freshly allocated by the nextBatch() call that produced it. A later nextBatch() never reuses or overwrites an array returned for an earlier batch — so a returned array may be kept and read after the reader has advanced, including handed off to another thread for processing. The reader itself is still a single-threaded cursor: only one consumer thread may call nextBatch(). (The capacity-sizing note on getBinaryValues() is about array length, not reuse; that buffer is fresh per batch too.)

This API is Experimental: the shape of the batch accessors and layer representation may change in future releases without prior deprecation.

  • Method Details

    • nextBatch

      public boolean nextBatch()

      Advance to the next batch.

      One reader advances only itself. Readers built separately — each from its own ParquetFileReader.buildColumnReader(String) — are independent cursors, and the batch a reader hands out is sized for the columns it reads. A batch of a BYTE_ARRAY column therefore covers fewer rows than a batch of an INT32 one over the same file, and a filtered reader is sized for its predicate's columns as well as its own. Two such readers reach different rows on their nth batch.

      So do not pair them: while (a.nextBatch() & b.nextBatch()) ends when whichever reader has the larger batches runs out first, and the values it took from each on any given turn are from different rows. Neither reader reports anything wrong, because neither is wrong — each is a complete, correct read of its own column.

      To read several columns of the same rows, use ColumnReaders, from ParquetFileReader.buildColumnReaders(dev.hardwood.schema.ColumnProjection). Its readers share one RowGroupIterator and one batch size, and ColumnReaders.nextBatch() advances them together and checks that their record counts agree. Advance them through that method rather than this one: this method moves a single reader, so calling it on one member of a ColumnReaders leaves the rest behind on the batch they already hold.

      Returns:
      true if a batch is available, false if exhausted

    • getRecordCount

      public int getRecordCount()
      Number of top-level records in the current batch.

    • getValueCount

      public int getValueCount()
      Total number of leaf values in the current batch — sized to real items only (phantom slots from null/empty parents are excluded). For flat columns this equals getRecordCount(). For a repeated column it is not bounded by the configured batch size — the batch size caps records, not leaf values — so a batch may carry more leaf values than records.

    • getLayerCount

      public int getLayerCount()
      Number of layers in this column's schema chain. 0 for a flat column. Stable for the lifetime of this reader and safe to call before the first nextBatch() — useful for sizing consumer-side buffers.

    • getLayerKind

      public LayerKind getLayerKind(int layer)
      Layer kind at layer. Stable for the lifetime of this reader and safe to call before the first nextBatch().

    • getLayerValidity

      public Validity getLayerValidity(int layer)
      Validity at layer. Returns Validity.NO_NULLS when no item at that layer is null in the current batch (the sparse fast path); otherwise returns a wrapper over the per-item null bitmap.

    • getLayerOffsets

      public int[] getLayerOffsets(int layer)
      Offsets at layer. Length == count(layer) + 1, with offsets[count(layer)] equal to count(layer + 1) (or to getValueCount() for the innermost layer). offsets[i+1] - offsets[i] == 0 denotes an empty list/map.
      Throws:
      IllegalStateException - if layer is outside [0, getLayerCount()) or if the layer is not LayerKind.REPEATED

    • getLeafValidity

      public Validity getLeafValidity()
      Validity over the leaf-value array, indexed 0..getValueCount(). Returns Validity.NO_NULLS when no leaf in the current batch is null.

    • getInts

      public int[] getInts()

    • getLongs

      public long[] getLongs()

    • getFloats

      public float[] getFloats()

    • getDoubles

      public double[] getDoubles()

    • getBooleans

      public boolean[] getBooleans()

    • getBinaryValues

      public byte[] getBinaryValues()
      Backing byte buffer for a varlength leaf. Capacity-sized: only bytes in the half-open range [0, getBinaryOffsets()[getValueCount()]) are valid; bytes beyond that position are unspecified.
      Throws:
      IllegalStateException - for non-byte-array leaves

    • getBinaryOffsets

      public int[] getBinaryOffsets()
      Sentinel-suffixed offsets into getBinaryValues(). Length == getValueCount() + 1; the byte length of value i is offsets[i+1] - offsets[i]. For FIXED_LEN_BYTE_ARRAY columns the offsets are trivially i * width.
      Throws:
      IllegalStateException - for non-byte-array leaves

    • getBinaries

      public byte[][] getBinaries()

      Materialises one byte[] per leaf value, copying out of the binary buffer. Returns null at indexes where getLeafValidity() is unset. Allocates one byte array per leaf — hot loops should consult getBinaryValues() + getBinaryOffsets() directly.

      The returned array has length getValueCount() — i.e. the real leaf count, not getRecordCount(). For a flat column the two coincide; for list<binary> and similar nested chains they differ, and lookups must go through the appropriate layer offsets rather than indexing by record.

    • getStrings

      public String[] getStrings()

      Convenience: materialises one String per leaf value by UTF-8 decoding the slice of getBinaryValues() for each entry. Returns null at indexes where getLeafValidity() is unset. BSON columns are not string-decoded; use getBinaries() / getBinaryValues() for those.

      The returned array has length getValueCount() — i.e. the real leaf count, not getRecordCount(). For a flat column the two coincide; for list<string> and similar nested chains they differ, and lookups must go through the appropriate layer offsets rather than indexing by record.

    • getColumnSchema

      public ColumnSchema getColumnSchema()

    • close

      public void close()
      Releases the resources held by this reader. Idempotent: calling it more than once has no further effect.
      Specified by:
      close in interface AutoCloseable