Class InMemoryOutputFile

java.lang.Object
dev.hardwood.InMemoryOutputFile
All Implemented Interfaces:
OutputFile, Closeable, AutoCloseable
@Experimental public final class InMemoryOutputFile extends Object implements OutputFile

An OutputFile that keeps the file in memory and hands it back.

Use it to produce Parquet bytes without a filesystem: a file to upload to an object store, a payload to send over a network, or one written and read back in the same process. It is created with OutputFile.inMemory(), and the buffer buffer() returns can be passed straight to InputFile.of(ByteBuffer):

InMemoryOutputFile out = OutputFile.inMemory();
try (ParquetFileWriter writer = ParquetFileWriter.create(out, schema)) {
    // write the file
}
try (ParquetFileReader reader = ParquetFileReader.open(InputFile.of(out.buffer()))) {
    // read it back
}

The whole file is held on the heap, which costs the size of the finished file in addition to the memory the writer uses for the row group it has open. A file can be at most Integer.MAX_VALUE - 8 bytes; a write past that fails with an IOException.

This API is Experimental: the shape may change in future releases.

  • Method Details

    • create

      public void create()
      Description copied from interface: OutputFile
      Performs resource acquisition (e.g. opening a file channel). Must be called before OutputFile.write(ByteBuffer) or OutputFile.position().
      Specified by:
      create in interface OutputFile

    • write

      public void write(ByteBuffer data) throws IOException
      Description copied from interface: OutputFile

      Appends the remaining bytes of data to the file.

      The buffer is consumed (its position advances to its limit).

      Specified by:
      write in interface OutputFile
      Parameters:
      data - the bytes to append
      Throws:
      IOException - if the write fails

    • position

      public long position()
      Description copied from interface: OutputFile
      Returns the number of bytes written so far, which is the offset at which the next OutputFile.write(ByteBuffer) will begin.
      Specified by:
      position in interface OutputFile
      Returns:
      the running byte offset

    • close

      public void close()
      Description copied from interface: OutputFile

      Finalizes the file and publishes it at the destination.

      When publishing fails, close() releases the resources and discards what was written before it throws, leaving the destination as if nothing was written. A failure to discard is attached to the thrown exception as suppressed.

      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in interface OutputFile

    • discard

      public void discard()
      Description copied from interface: OutputFile
      Discards everything written and releases resources without publishing a file at the destination. This is the failure counterpart to OutputFile.close(): OutputFile.close() finalizes (commits) the file, discard() throws it away. The writer calls discard() when it cannot finish a valid file, so a partially written file is never presented as valid. After discard() the destination is left as if nothing was written; calling close() afterwards is a no-op.
      Specified by:
      discard in interface OutputFile

    • buffer

      public ByteBuffer buffer()

      The file that was written.

      The returned buffer contains the whole file: its position is 0 and its limit is the number of bytes written, which is the range InputFile.of(ByteBuffer) reads, so it can be passed there unchanged. Each call returns a buffer with its own position and limit, so reading one buffer does not affect another. Do not modify the buffer's contents: whether a change shows in the buffers other calls return is unspecified.

      Returns:
      the bytes of the finished file
      Throws:
      IllegalStateException - if the file has not been closed, or was discarded