Class ParquetFileWriter

java.lang.Object
dev.hardwood.writer.ParquetFileWriter
All Implemented Interfaces:
Closeable, AutoCloseable
public final class ParquetFileWriter extends Object implements Closeable

Writes a Parquet file through a columnar or a row-oriented API: columnWriter() takes an aligned slice of typed arrays, rowWriter() takes one record at a time, and one file is written through one of the two, not both.

Every primitive physical type is written — BOOLEAN, INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY, and FIXED_LEN_BYTE_ARRAY — flat REQUIRED / OPTIONAL, nested inside REQUIRED / OPTIONAL struct groups, and inside LISTs and MAPs (including lists of lists, lists of structs, and maps of any in-scope value). Data is supplied as ColumnBatch slices; the writer packs each column into size-bounded data pages — a levelled column's pages carrying an RLE definition-level stream ahead of the values — and flushes a row group once it reaches one of the configured targets, or where the caller ends it through endRowGroup(). Each row group is buffered, written and forgotten, so what the writer holds follows the target rather than the size of the file, apart from the page index it keeps for the end of the file (a location and two bounds per data page) and the Bloom filters of the columns WriterConfig names — and the target counts what is held, so it is that number and not a multiple of it, give or take the slack the value stores carry from growing geometrically. Each column chunk is encoded one way throughout: by default the writer weighs a dictionary against PLAIN once the row group is buffered and takes the smaller, or a ColumnEncoding set per column names the encoding outright. Each page body is compressed with the configured codec (ZSTD by default). All of these are configurable through WriterConfig. The row groups and footer are finalized on close().

The file is produced front to back and is valid only after close() returns. A writer whose write has thrown accepts no more data, and close() discards its output rather than publishing the rows written before the failure. abort() discards the output on a failure the writer does not see, such as one in the code producing the data.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    DEFAULT_CREATED_BY
    Default created_by identifier written into the file footer, in the <app> version <version> (build <hash>) convention Parquet readers parse — for example hardwood version 1.1.0 (build a093aab).

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    abort()
    Discards the output without publishing a file, leaving nothing at the destination, and closes the writer.
    void
    close()
    Finishes the file: writes the row group still buffered and the footer, and publishes the file at the destination.
    ColumnWriter
    columnWriter()
    Returns the column-oriented view over this file: a batch-shaped API for callers that hold columns rather than records.
    static ParquetFileWriter
    create(OutputFile out, FileSchema schema)
    Opens a writer with the default WriterConfig.
    static ParquetFileWriter
    create(OutputFile out, FileSchema schema, WriterConfig config)
    Opens a writer, writing the leading magic bytes.
    void
    createdBy(String createdBy)
    Replaces the footer's created_by identifier, which defaults to DEFAULT_CREATED_BY.
    void
    endRowGroup()
    Closes the open row group and starts a new one, so the records written next go into a row group of their own.
    void
    keyValueMetadata(String key, String value)
    Stamps one application-defined key-value pair onto the file footer, replacing any value already held for that key.
    void
    keyValueMetadata(Map<String,String> metadata)
    Stamps every entry of metadata onto the file footer, replacing any value already held for a key it names and leaving the rest in place.
    RowWriter
    rowWriter()
    Returns the row-oriented view over this file: a record-shaped API for callers that hold records rather than columns.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • DEFAULT_CREATED_BY

      public static final String DEFAULT_CREATED_BY

      Default created_by identifier written into the file footer, in the <app> version <version> (build <hash>) convention Parquet readers parse — for example hardwood version 1.1.0 (build a093aab). The hash carries a -dirty suffix when the working tree was not clean at build time, and a build that cannot identify itself reports unknown in place of the version or the hash.

      A reader that cannot parse this field cannot tell which writer produced the file, and applies its writer-specific correctness workarounds to it by default.

  • Method Details

    • create

      public static ParquetFileWriter create(OutputFile out, FileSchema schema) throws IOException
      Opens a writer with the default WriterConfig.
      Parameters:
      out - the destination
      schema - the schema to write
      Returns:
      an open writer
      Throws:
      IOException - if the destination cannot be opened
      IllegalArgumentException - if the schema declares no columns, or a column the writer cannot write: a FIXED_LEN_BYTE_ARRAY without a positive length, or an annotation its physical type cannot carry or no value can be written for
      UnsupportedOperationException - if the schema has a column of an unsupported physical type, or a shape the writer cannot produce

    • create

      public static ParquetFileWriter create(OutputFile out, FileSchema schema, WriterConfig config) throws IOException
      Opens a writer, writing the leading magic bytes.
      Parameters:
      out - the destination
      schema - the schema to write
      config - the writer configuration
      Returns:
      an open writer
      Throws:
      IOException - if the destination cannot be opened
      UnsupportedOperationException - if the schema has a column of an unsupported physical type, a shape the writer cannot produce, or the configured codec cannot be written
      IllegalArgumentException - if the schema declares no columns, or a column the writer cannot write: a FIXED_LEN_BYTE_ARRAY without a positive length, or an annotation its physical type cannot carry or no value can be written for; or an encoding policy names a column the schema does not have, or one its physical type cannot carry; or a Bloom filter names a column the schema does not have, or a BOOLEAN column

    • keyValueMetadata

      public void keyValueMetadata(String key, String value)

      Stamps one application-defined key-value pair onto the file footer, replacing any value already held for that key.

      The footer's key_value_metadata is where the ecosystem records what a schema alone does not carry — ARROW:schema, pandas, Spark's org.apache.spark.sql.parquet.row.metadata, and the table-format stamps. Parquet itself does not interpret these entries, and nothing here validates them beyond requiring a key.

      Callable until close(), so a value the caller knows only once the data is written — a row count, a digest over what was produced — can still be stated.

      Parameters:
      key - the entry's key
      value - the entry's value, or null to write a key carrying no value, which the format allows and which is how a key read from such a file is written back
      Throws:
      IllegalArgumentException - if key is null
      IllegalStateException - if the writer is closed

    • keyValueMetadata

      public void keyValueMetadata(Map<String,String> metadata)

      Stamps every entry of metadata onto the file footer, replacing any value already held for a key it names and leaving the rest in place.

      Passing the map a reader returns from FileMetaData.keyValueMetadata() reproduces that file's application metadata, entries carrying no value included.

      Parameters:
      metadata - the entries to add
      Throws:
      IllegalArgumentException - if metadata is null or holds a null key
      IllegalStateException - if the writer is closed
      See Also:

    • createdBy

      public void createdBy(String createdBy)

      Replaces the footer's created_by identifier, which defaults to DEFAULT_CREATED_BY.

      Readers that key compatibility workarounds off this field expect the <app> version <version> (build <hash>) shape; a bare application name is rejected by some of them.

      Parameters:
      createdBy - the identifier to write
      Throws:
      IllegalArgumentException - if createdBy is null
      IllegalStateException - if the writer is closed

    • columnWriter

      public ColumnWriter columnWriter()

      Returns the column-oriented view over this file: a batch-shaped API for callers that hold columns rather than records. It takes an aligned slice of typed arrays through ColumnBatch, shreds it, and pages it into the file.

      The file writer keeps ownership: the returned view is not closeable, and the row group it has buffered is written by close(). The same instance is returned on every call.

      Returns:
      the column-oriented view over this file
      Throws:
      IllegalStateException - if the writer is closed, or rowWriter() has already been used on this file

    • rowWriter

      @Experimental public RowWriter rowWriter()

      Returns the row-oriented view over this file: a record-shaped API for callers that hold records rather than columns. It stages records into batches and submits them through ColumnWriter.writeBatch(Consumer), so the file it produces is the one the columnar API produces for the same data.

      The file writer keeps ownership: the returned view is not closeable, and its pending records are written by close(). The same instance is returned on every call.

      Returns:
      the row-oriented view over this file
      Throws:
      IllegalStateException - if the writer is closed, or columnWriter() has already been used on this file
      UnsupportedOperationException - if the schema has a shape the writer cannot produce record by record

    • endRowGroup

      public void endRowGroup() throws IOException

      Closes the open row group and starts a new one, so the records written next go into a row group of their own.

      The row-group targets in WriterConfig still cut row groups on their own; this is an additional cut, for a boundary only the caller can see — a change of sort key or partition value, or the end of an upstream batch. A row group closed this way is written exactly as one a target closes after the same records.

      With nothing written since the last row group was closed, the call does nothing, so it can be made at every boundary without tracking whether a record fell between two of them. Every row group adds a column chunk per column to the footer, so a file of row groups much smaller than the targets carries a larger footer, and a reader opens more column chunks for the same records.

      Records staged by rowWriter() are submitted first, so they belong to the row group this call closes. The call is rejected from inside a batch or record filler, where the batch or record being filled is not yet written. Any exception fails the writer, as it does from a write.

      Throws:
      IOException - if writing the row group fails
      IllegalStateException - if the writer is closed, a previous write has failed, or a batch or record filler is running

    • close

      public void close() throws IOException

      Finishes the file: writes the row group still buffered and the footer, and publishes the file at the destination.

      If a write has thrown, the output is discarded instead, leaving nothing at the destination. A failure while finishing or publishing the file discards the output as well, and a failure to discard it is attached to the thrown exception as suppressed. Does nothing if the writer is already closed or aborted.

      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException - if the file cannot be finished or published, or a discarded output cannot be released
      IllegalStateException - if called from inside a batch or record filler

    • abort

      public void abort() throws IOException

      Discards the output without publishing a file, leaving nothing at the destination, and closes the writer.

      For a failure the writer does not see, such as one in the code producing the data:

      try (ParquetFileWriter writer = ParquetFileWriter.create(out, schema)) {
    try {
        produce(writer);
    }
    catch (Exception e) {
        try {
            writer.abort();
        }
        catch (IOException discardFailure) {
            e.addSuppressed(discardFailure);
        }
        throw e;
    }
}

      A failure of a write needs no call: close() discards the output of a writer whose write has thrown. Does nothing if the writer is already closed or aborted, so a file close() has published stays published.

      Throws:
      IOException - if the discarded output cannot be released, for example an upload that cannot be cancelled