Class ParquetFileWriter

java.lang.Object
dev.hardwood.writer.ParquetFileWriter
All Implemented Interfaces:
Closeable, AutoCloseable
public final class ParquetFileWriter extends Object implements Closeable

Writes a Parquet file through a columnar or a row-oriented API: columnWriter() takes an aligned slice of typed arrays, rowWriter() takes one record at a time, and one file is written through one of the two, not both.

Every primitive physical type is written — BOOLEAN, INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY, and FIXED_LEN_BYTE_ARRAY — flat REQUIRED / OPTIONAL, nested inside REQUIRED / OPTIONAL struct groups, and inside LISTs and MAPs (including lists of lists, lists of structs, and maps of any in-scope value). Data is supplied as ColumnBatch slices; the writer packs each column into size-bounded data pages — a levelled column's pages carrying an RLE definition-level stream ahead of the values — and flushes a row group once the bytes it holds for that group reach the configured target. Each row group is buffered, written and forgotten, so what the writer holds follows the target rather than the size of the file — and the target counts what is held, so it is that number and not a multiple of it, give or take the slack the value stores carry from growing geometrically. Each column chunk is encoded one way throughout: by default the writer weighs a dictionary against PLAIN once the row group is buffered and takes the smaller, or a ColumnEncoding set per column names the encoding outright. Each page body is compressed with the configured codec (ZSTD by default). All of these are configurable through WriterConfig. The row groups and footer are finalized on close().

The file is produced front to back and is valid only after close() returns.

  • Field Details

    • DEFAULT_CREATED_BY

      public static final String DEFAULT_CREATED_BY

      Default created_by identifier written into the file footer, in the <app> version <version> (build <hash>) convention Parquet readers parse — for example hardwood version 1.1.0 (build a093aab). The hash carries a -dirty suffix when the working tree was not clean at build time, and a build that cannot identify itself reports unknown in place of the version or the hash.

      A reader that cannot parse this field cannot tell which writer produced the file, and applies its writer-specific correctness workarounds to it by default.

  • Method Details

    • create

      public static ParquetFileWriter create(OutputFile out, FileSchema schema) throws IOException
      Opens a writer with the default WriterConfig.
      Parameters:
      out - the destination
      schema - the schema to write
      Returns:
      an open writer
      Throws:
      IOException - if the destination cannot be opened
      UnsupportedOperationException - if the schema has a column of an unsupported physical type, or a shape the writer cannot produce

    • create

      public static ParquetFileWriter create(OutputFile out, FileSchema schema, WriterConfig config) throws IOException
      Opens a writer, writing the leading magic bytes.
      Parameters:
      out - the destination
      schema - the schema to write
      config - the writer configuration
      Returns:
      an open writer
      Throws:
      IOException - if the destination cannot be opened
      UnsupportedOperationException - if the schema has a column of an unsupported physical type, a shape the writer cannot produce, or the configured codec cannot be written
      IllegalArgumentException - if an encoding policy names a column the schema does not have, or one its physical type cannot carry

    • keyValueMetadata

      public void keyValueMetadata(String key, String value)

      Stamps one application-defined key-value pair onto the file footer, replacing any value already held for that key.

      The footer's key_value_metadata is where the ecosystem records what a schema alone does not carry — ARROW:schema, pandas, Spark's org.apache.spark.sql.parquet.row.metadata, and the table-format stamps. Parquet itself does not interpret these entries, and nothing here validates them beyond requiring a key.

      Callable until close(), so a value the caller knows only once the data is written — a row count, a digest over what was produced — can still be stated.

      Parameters:
      key - the entry's key
      value - the entry's value, or null to write a key carrying no value, which the format allows and which is how a key read from such a file is written back
      Throws:
      IllegalArgumentException - if key is null
      IllegalStateException - if the writer is closed

    • keyValueMetadata

      public void keyValueMetadata(Map<String,String> metadata)

      Stamps every entry of metadata onto the file footer, replacing any value already held for a key it names and leaving the rest in place.

      Passing the map a reader returns from FileMetaData.keyValueMetadata() reproduces that file's application metadata, entries carrying no value included.

      Parameters:
      metadata - the entries to add
      Throws:
      IllegalArgumentException - if metadata is null or holds a null key
      IllegalStateException - if the writer is closed
      See Also:

    • createdBy

      public void createdBy(String createdBy)

      Replaces the footer's created_by identifier, which defaults to DEFAULT_CREATED_BY.

      Readers that key compatibility workarounds off this field expect the <app> version <version> (build <hash>) shape; a bare application name is rejected by some of them.

      Parameters:
      createdBy - the identifier to write
      Throws:
      IllegalArgumentException - if createdBy is null
      IllegalStateException - if the writer is closed

    • columnWriter

      public ColumnWriter columnWriter()

      Returns the column-oriented view over this file: a batch-shaped API for callers that hold columns rather than records. It takes an aligned slice of typed arrays through ColumnBatch, shreds it, and pages it into the file.

      The file writer keeps ownership: the returned view is not closeable, and the row group it has buffered is written by close(). The same instance is returned on every call.

      Returns:
      the column-oriented view over this file
      Throws:
      IllegalStateException - if the writer is closed, or rowWriter() has already been used on this file

    • rowWriter

      @Experimental public RowWriter rowWriter()

      Returns the row-oriented view over this file: a record-shaped API for callers that hold records rather than columns. It stages records into batches and submits them through ColumnWriter.writeBatch(Consumer), so the file it produces is the one the columnar API produces for the same data.

      The file writer keeps ownership: the returned view is not closeable, and its pending records are written by close(). The same instance is returned on every call.

      Returns:
      the row-oriented view over this file
      Throws:
      IllegalStateException - if the writer is closed, or columnWriter() has already been used on this file
      UnsupportedOperationException - if the schema has a shape the writer cannot produce record by record

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException