Interface InputFile

All Superinterfaces:
AutoCloseable, Closeable
All Known Implementing Classes:
S3InputFile
public interface InputFile extends Closeable

Abstraction for reading Parquet file data.

This interface decouples the read pipeline from memory-mapped local files, enabling alternative backends such as object stores or in-memory buffers.

An InputFile starts in an unopened state. The open() method must be called before readRange(long, int) or length() can be used. The framework (Hardwood, ParquetFileReader) calls open() automatically and takes ownership of the files passed to it: the reader closes them when it is closed or when opening it fails. Callers create instances via of(Path) and close only files they have not handed to a reader. Closeable.close() may be called on a file that was never opened.

Implementations must be safe for concurrent use from multiple threads once opened. The returned ByteBuffer instances are owned by the caller and may be slices of a larger mapping or freshly allocated buffers, depending on the implementation.

See Also:

  • Method Details

    • open

      void open() throws IOException
      Performs expensive resource acquisition (e.g. memory-mapping, network connect). Must be called before readRange(long, int) or length().
      Throws:
      IOException - if the resource cannot be acquired

    • readRange

      ByteBuffer readRange(long offset, int length) throws IOException
      Read a range of bytes from the file.
      Parameters:
      offset - the byte offset to start reading from
      length - the number of bytes to read
      Returns:
      a ByteBuffer containing the requested data
      Throws:
      IOException - if the read fails
      IllegalStateException - if open() has not been called
      IndexOutOfBoundsException - if offset or length is out of range

    • length

      long length() throws IOException
      Returns the total size of the file in bytes.
      Returns:
      the file size
      Throws:
      IOException - if the size cannot be determined
      IllegalStateException - if open() has not been called

    • name

      String name()
      Returns an identifier for this file, used in log messages and JFR events.
      Returns:
      a human-readable name or path

    • identity

      @Experimental default Optional<String> identity() throws IOException

      An identifier of the file's content, which differs for different content, if the backend can supply one without a request of its own; empty when it cannot.

      Resolved by open() and fixed for the file's lifetime: it names the content the file was opened against, not the content at its location at the time of the call. A ParsedFooter records the identity of the file it was read from, and a reader compares the two before using a footer a MetadataSource supplied.

      Backend Identity
      Local file (of(Path)) the file's size, modification time and file key (or absolute path where the file system has no file keys), read when it is opened; when a reader whose context has no MetadataSource opened it, read when first asked for
      S3 the object's ETag
      In-memory (of(ByteBuffer)) empty

      Implement it whenever the backend can supply an identity: a MetadataSource that keys its cache on it can only detect a replaced file through it, and an empty identity turns that detection off. An InputFile that wraps another returns the wrapped file's identity. The default implementation returns empty.

      Returns:
      the identity, or empty when the backend has none
      Throws:
      IOException - if the identity cannot be determined
      IllegalStateException - if the implementation resolves its identity in open() and open() has not been called

    • of

      static InputFile of(ByteBuffer buffer)

      Creates an InputFile backed by an in-memory ByteBuffer.

      The file is the buffer's remaining content, from its position to its limit, taken when this method is called. The content is not copied, and the buffer's position and limit are not changed; later changes to them do not affect the file. Since the data is already in memory, no resource acquisition is needed and open() is a no-op.

      Parameters:
      buffer - the buffer containing Parquet file data between its position and limit
      Returns:
      a new InputFile backed by the buffer

    • of

      static InputFile of(Path path)
      Creates an unopened InputFile for a local file path.
      Parameters:
      path - the file to read
      Returns:
      a new unopened InputFile

    • ofPaths

      static List<InputFile> ofPaths(List<Path> paths)
      Creates unopened InputFile instances for a list of local file paths.
      Parameters:
      paths - the files to read
      Returns:
      a list of new unopened InputFile instances

    • ofPaths

      static List<InputFile> ofPaths(Path first, Path... more)
      Creates unopened InputFile instances for the given local file paths.
      Parameters:
      first - the first file to read
      more - additional files to read
      Returns:
      a list of new unopened InputFile instances

    • ofBuffers

      static List<InputFile> ofBuffers(List<ByteBuffer> buffers)

      Creates InputFile instances for a list of in-memory ByteBuffers. Each file is its buffer's remaining content, as described in of(ByteBuffer).

      Since the data is already in memory, no resource acquisition is needed and open() is a no-op for each instance.

      Parameters:
      buffers - the buffers containing Parquet file data
      Returns:
      a list of new InputFile instances backed by the buffers

    • ofBuffers

      static List<InputFile> ofBuffers(ByteBuffer first, ByteBuffer... more)

      Creates InputFile instances for the given in-memory ByteBuffers. Each file is its buffer's remaining content, as described in of(ByteBuffer).

      Since the data is already in memory, no resource acquisition is needed and open() is a no-op for each instance.

      Parameters:
      first - the first buffer containing Parquet file data
      more - additional buffers containing Parquet file data
      Returns:
      a list of new InputFile instances backed by the buffers