Class ParquetFileReader

java.lang.Object
dev.hardwood.reader.ParquetFileReader
All Implemented Interfaces:
AutoCloseable
public class ParquetFileReader extends Object implements AutoCloseable

Reader for one or more Parquet files.

A reader opened over a list of files exposes the schema of the first file and reads rows / column batches across all files in order, with cross-file prefetching handled by the underlying iterator.

// Single file
try (ParquetFileReader reader = ParquetFileReader.open(InputFile.of(path))) {
    RowReader rows = reader.rowReader();
    // ...
}

// Multiple files (use Hardwood for a shared thread pool)
try (Hardwood hardwood = Hardwood.create();
     ParquetFileReader reader = hardwood.openAll(files)) {
    try (ColumnReaders cols = reader.columnReaders(
                ColumnProjection.columns("a", "b"))) {
        // ...
    }
}

After close: the metadata accessors split on whether they can read from disk. getFileMetaData(int) may have to load a footer, so it throws IllegalStateException once the reader is closed. getFileCount(), getFileMetaData() and getFileSchema() are served from state already in memory and stay usable.

Limitation: When using the default memory-mapped InputFile, the file itself may be arbitrarily large, but each individual column chunk must be at most 2 GB (Integer.MAX_VALUE bytes) of compressed data. The in-memory and object-store backends have a 2 GB limit on the whole file.

  • Method Details

    • open

      public static ParquetFileReader open(InputFile inputFile) throws IOException

      Open a single Parquet file with a dedicated context.

      Calls InputFile.open() and takes ownership of the file; it is closed when this reader is closed.

      Throws:
      IOException

    • open

      public static ParquetFileReader open(InputFile inputFile, HardwoodContext context) throws IOException

      Open a single Parquet file with a shared context.

      Calls InputFile.open() and takes ownership of the file; it is closed when this reader is closed. The caller retains ownership of the context.

      Throws:
      IOException

    • open

      public static ParquetFileReader open(InputFile inputFile, HardwoodContext context, ReaderConfig readerConfig) throws IOException
      Open a single Parquet file with a shared context and an explicit ReaderConfig. The context (shared runtime resources) and the config (per-read behaviour) are independent, so one context can back reads with different configs.
      Throws:
      IOException

    • openAll

      public static ParquetFileReader openAll(List<? extends InputFile> inputFiles) throws IOException
      Open multiple Parquet files with a dedicated context. The schema is read from the first file and is assumed to be common across all files. Files are opened on demand by the iterator; the first file is opened eagerly so any I/O or metadata error surfaces immediately.
      Throws:
      IOException

    • openAll

      public static ParquetFileReader openAll(List<? extends InputFile> inputFiles, HardwoodContext context) throws IOException
      Open multiple Parquet files with a shared context.
      Throws:
      IOException

    • openAll

      public static ParquetFileReader openAll(List<? extends InputFile> inputFiles, HardwoodContext context, ReaderConfig readerConfig) throws IOException
      Open multiple Parquet files with a shared context and an explicit ReaderConfig.
      Throws:
      IOException

    • getFileMetaData

      public FileMetaData getFileMetaData()
      File metadata of the first input file.

    • getFileCount

      public int getFileCount()

      Number of physical input files represented by this reader.

      This method performs no I/O.

    • getFileMetaData

      public FileMetaData getFileMetaData(int fileIndex) throws IOException

      Returns the metadata of one physical input file.

      Files are indexed in the order supplied to openAll(List). Metadata for the first file is read when the reader is opened; metadata for later files is read on first access or data-reader prefetch. In-progress, successful, and failed loads are cached for this reader's lifetime, and this synchronous accessor joins a load already in progress. Close and reopen the reader to retry a failed load or inspect a changed file.

      This method returns the physical file's footer without performing projection- or filter-specific cross-file schema validation. That validation occurs when a row or column reader is planned.

      Input files must not be modified while this reader is open.

      Parameters:
      fileIndex - zero-based physical input file index
      Returns:
      metadata parsed from that file's footer
      Throws:
      IndexOutOfBoundsException - if fileIndex is outside [0, getFileCount())
      IllegalStateException - if this reader is closed
      IOException - if the file cannot be opened or its footer cannot be read

    • getFileSchema

      public FileSchema getFileSchema()

    • isMultiFile

      public boolean isMultiFile()
      true when this reader was opened over more than one input file.

    • rowReader

      public RowReader rowReader()
      Shortcut for buildRowReader().build() — read every row of every column with no filter.

    • buildRowReader

      public ParquetFileReader.RowReaderBuilder buildRowReader()
      Begin configuring a RowReader with optional projection, filter, and head/tail limit.

    • columnReader

      public ColumnReader columnReader(String columnName)
      Shortcut for buildColumnReader(String).build() — read every row group of the named column with no filter. Single-file only.

    • columnReader

      public ColumnReader columnReader(int columnIndex)
      Shortcut for buildColumnReader(int).build() — read every row group of the column at the given index with no filter. Single-file only.

    • buildColumnReader

      public ParquetFileReader.ColumnReaderBuilder buildColumnReader(String columnName)
      Begin configuring a single-column ColumnReader. Single-file only.

    • buildColumnReader

      public ParquetFileReader.ColumnReaderBuilder buildColumnReader(int columnIndex)
      Begin configuring a single-column ColumnReader by column index. Single-file only.

    • columnReaders

      public ColumnReaders columnReaders(ColumnProjection projection)
      Shortcut for buildColumnReaders(ColumnProjection).build() — every row group, no filter. Works for single- and multi-file.

    • buildColumnReaders

      public ParquetFileReader.ColumnReadersBuilder buildColumnReaders(ColumnProjection projection)
      Begin configuring a ColumnReaders collection for batch-oriented access to a column projection. Works for single- and multi-file.

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Throws:
      IOException