Class ParquetFileReader

java.lang.Object
dev.hardwood.reader.ParquetFileReader
All Implemented Interfaces:
Closeable, AutoCloseable
public class ParquetFileReader extends Object implements Closeable

Reader for one or more Parquet files.

A reader opened over a list of files exposes the schema of the first file and reads rows / column batches across all files in order, with cross-file prefetching handled by the underlying iterator.

// Single file
try (ParquetFileReader reader = ParquetFileReader.open(InputFile.of(path))) {
    RowReader rows = reader.rowReader();
    // ...
}

// Multiple files (use Hardwood for a shared thread pool)
try (Hardwood hardwood = Hardwood.create();
     ParquetFileReader reader = hardwood.openAll(files)) {
    try (ColumnReaders cols = reader.columnReaders(
                ColumnProjection.columns("a", "b"))) {
        // ...
    }
}

After close: the metadata accessors split on whether they can read from disk. getFileMetaData(int) may have to load a footer, so it throws IllegalStateException once the reader is closed. getFileCount(), getFileMetaData() and getFileSchema() are served from state already in memory and stay usable. Building a row or column reader throws IllegalStateException once the reader is closed, and so does a build that is under way on another thread when close() is called; close() waits for such a build and closes the reader it made.

Ownership: open and openAll take ownership of every input file. The files are closed when the reader is closed, or before the exception propagates when opening fails, including files the reader has not opened yet. A context the call creates is closed with them; a context passed in stays the caller's.

Limitation: A column chunk read without an OffsetIndex must be at most 2 GB (Integer.MAX_VALUE bytes) of compressed data. Files larger than 2 GB are read locally and from S3, except by the in-memory backend and by an S3 file with RangeBacking.SPARSE_TEMPFILE, which are limited to 2 GB per file.

  • Method Details

    • open

      public static ParquetFileReader open(InputFile inputFile) throws IOException

      Open a single Parquet file with a dedicated context.

      Calls InputFile.open() and takes ownership of the file (see Ownership above).

      Throws:
      IOException

    • open

      public static ParquetFileReader open(InputFile inputFile, HardwoodContext context) throws IOException

      Open a single Parquet file with a shared context.

      Calls InputFile.open() and takes ownership of the file (see Ownership above). The caller retains ownership of the context.

      Throws:
      IOException

    • open

      public static ParquetFileReader open(InputFile inputFile, HardwoodContext context, ReaderConfig readerConfig) throws IOException
      Open a single Parquet file with a shared context and an explicit ReaderConfig. The context (shared runtime resources) and the config (per-read behaviour) are independent, so one context can back reads with different configs. Takes ownership of the file (see Ownership above); the caller retains ownership of the context.
      Throws:
      IOException

    • openAll

      public static ParquetFileReader openAll(List<? extends InputFile> inputFiles) throws IOException

      Open multiple Parquet files with a dedicated context. The schema is read from the first file and is assumed to be common across all files. Files are opened on demand by the iterator; the first file is opened eagerly so any I/O or metadata error surfaces immediately.

      A later file is opened when a read arrives at it, so its I/O errors and any disagreement with the reference schema are raised from the reading loop rather than from the call that built the reader — always before any row of that file is returned.

      Takes ownership of all files (see Ownership above).

      Throws:
      IOException

    • openAll

      public static ParquetFileReader openAll(List<? extends InputFile> inputFiles, HardwoodContext context) throws IOException
      Open multiple Parquet files with a shared context. Takes ownership of all files (see Ownership above); the caller retains ownership of the context.
      Throws:
      IOException

    • openAll

      public static ParquetFileReader openAll(List<? extends InputFile> inputFiles, HardwoodContext context, ReaderConfig readerConfig) throws IOException
      Open multiple Parquet files with a shared context and an explicit ReaderConfig. Takes ownership of all files (see Ownership above); the caller retains ownership of the context.
      Throws:
      IOException

    • getFileMetaData

      public FileMetaData getFileMetaData()
      File metadata of the first input file.

    • getFileCount

      public int getFileCount()

      Number of physical input files represented by this reader.

      This method performs no I/O.

    • getFileMetaData

      public FileMetaData getFileMetaData(int fileIndex) throws IOException

      Returns the metadata of one physical input file.

      Files are indexed in the order supplied to openAll(List). Metadata for the first file is read when the reader is opened; metadata for later files is read on first access or data-reader prefetch. In-progress, successful, and failed loads are cached for this reader's lifetime, and this synchronous accessor joins a load already in progress. Close and reopen the reader to retry a failed load or inspect a changed file.

      This method returns the physical file's footer without performing projection- or filter-specific cross-file schema validation. That validation occurs when a row or column reader is planned.

      Input files must not be modified while this reader is open.

      Parameters:
      fileIndex - zero-based physical input file index
      Returns:
      metadata parsed from that file's footer
      Throws:
      IndexOutOfBoundsException - if fileIndex is outside [0, getFileCount())
      IllegalStateException - if this reader is closed
      IOException - if the file cannot be opened or its footer cannot be read

    • getFileSchema

      public FileSchema getFileSchema()

    • isMultiFile

      public boolean isMultiFile()
      true when this reader was opened over more than one input file.

    • rowReader

      public RowReader rowReader() throws IOException
      Shortcut for buildRowReader().build() — read every row of every column with no filter.
      Throws:
      IOException

    • buildRowReader

      public ParquetFileReader.RowReaderBuilder buildRowReader()
      Begin configuring a RowReader with optional projection, filter, and head/tail limit.

    • columnReader

      public ColumnReader columnReader(String columnName) throws IOException
      Shortcut for buildColumnReader(String).build() — read every row group of the named column with no filter. Single-file only.
      Throws:
      IOException

    • columnReader

      public ColumnReader columnReader(int columnIndex) throws IOException
      Shortcut for buildColumnReader(int).build() — read every row group of the column at the given index with no filter. Single-file only.
      Throws:
      IOException

    • buildColumnReader

      public ParquetFileReader.ColumnReaderBuilder buildColumnReader(String columnName)
      Begin configuring a single-column ColumnReader. Single-file only.

    • buildColumnReader

      public ParquetFileReader.ColumnReaderBuilder buildColumnReader(int columnIndex)
      Begin configuring a single-column ColumnReader by column index. Single-file only.

    • columnReaders

      public ColumnReaders columnReaders(ColumnProjection projection) throws IOException
      Shortcut for buildColumnReaders(ColumnProjection).build() — every row group, no filter. Works for single- and multi-file.
      Throws:
      IOException

    • buildColumnReaders

      public ParquetFileReader.ColumnReadersBuilder buildColumnReaders(ColumnProjection projection)
      Begin configuring a ColumnReaders collection for batch-oriented access to a column projection. Works for single- and multi-file.

    • close

      public void close() throws IOException
      Waits for the child builds under way, which fail and close what they built, then closes every child reader still open, which stops its workers and waits for the prefetches its read started, then this reader's metadata, an owned context and owned input files. A failure to close a child does not stop the rest from being closed; the first failure is thrown, later ones suppressed beneath it.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException