Class WriterConfig

java.lang.Object
dev.hardwood.writer.WriterConfig
public final class WriterConfig extends Object

Tuning knobs for ParquetFileWriter.

Four targets govern the writer's output granularity:

  • Page byte target — the writer cuts a data page once the records it holds would encode to this many bytes. The page is cut before the record that would cross it, so this is a ceiling rather than something a page overshoots; only a single record larger than the whole target can breach it, a page holding whole records.
  • Page row target — the writer also cuts a data page once it holds this many records, which sets how finely the page index lets a reader skip within a column chunk.
  • Row-group row target — the writer cuts a row group once it holds this many records. This is the control over how a file is banded, and it is exactly what it says: a row count needs no estimate and does not vary with the data. It binds for narrow records.
  • Row-group buffer target — the writer also cuts a row group once the bytes it holds for that group reach this many: its level streams, dictionary indices, value stores and dictionaries. This is the memory control, and it bounds a row group of records wider than expected. A row group passes it by at most one record.

A row group is cut at whichever of the two row-group targets is reached first.

Neither of them is the size of what reaches the file. A row group is encoded and compressed after it is buffered, and both steps shrink it by a factor of the data's own repetitiveness: the same buffer target produces a row group a twentieth of its size on dictionary-friendly data and most of its size on incompressible data. A caller who needs a particular on-disk size measures one file and scales the setting; a caller who needs a particular banding sets the row target. See ParquetFileWriter for what bounds memory.

How a column's values are stored is a ColumnEncoding, set file-wide or per leaf column; how the resulting page bodies are compressed is the CompressionCodec. Both are configured here rather than on the schema.

A column can carry a Bloom filter in every row group, which lets a reader skip a row group that does not hold a value an equality or IN predicate asks for, where the row group's min / max span it. No column has one unless named through WriterConfig.Builder.bloomFilter(String).

A file's created_by identifier and its key-value metadata are set on ParquetFileWriter.

Obtain the defaults with defaults() or override individual knobs through builder().

  • Field Details

    • DEFAULT_PAGE_TARGET_BYTES

      public static final int DEFAULT_PAGE_TARGET_BYTES
      Default page target: 1 MiB of encoded values per data page.
      See Also:

    • DEFAULT_PAGE_TARGET_ROWS

      public static final int DEFAULT_PAGE_TARGET_ROWS
      Default page row target: 20,000 records per data page.
      See Also:

    • DEFAULT_ROW_GROUP_BUFFER_TARGET_BYTES

      public static final long DEFAULT_ROW_GROUP_BUFFER_TARGET_BYTES
      Default row-group buffer target: 128 MiB of buffered values per row group.
      See Also:

    • DEFAULT_ROW_GROUP_TARGET_ROWS

      public static final long DEFAULT_ROW_GROUP_TARGET_ROWS

      Default row-group row target: 1,048,576 records.

      Both Arrow implementations cap a row group's records here — DuckDB caps lower, at 122,880 — and on a flat three-column fixture the cap costs a quarter of a percent in file size for four times the banding. It binds wherever a record is narrower than the buffer target's share of it — for anything under about 128 bytes a record — and the buffer target takes over above that, so a narrow schema is banded by its record count and a wide one by what it holds.

      See Also:

    • DEFAULT_STATISTICS_TRUNCATION_LENGTH

      public static final int DEFAULT_STATISTICS_TRUNCATION_LENGTH
      Default statistics truncation length: BYTE_ARRAY min / max bounds longer than 64 bytes are truncated and flagged inexact.
      See Also:

    • DEFAULT_CODEC

      public static final CompressionCodec DEFAULT_CODEC
      Default page compression codec: ZSTD when the zstd-jni library is on the classpath, otherwise UNCOMPRESSED. Choosing a codec explicitly through WriterConfig.Builder.codec still requires that codec's library and fails at writer creation when it is missing; this default only avoids imposing the ZSTD dependency on callers who did not ask to compress.

    • DEFAULT_PRECISION_LOSS_POLICY

      public static final PrecisionLossPolicy DEFAULT_PRECISION_LOSS_POLICY
      Default precision-loss policy: reject a value the column cannot hold exactly, rather than silently dropping the digits that do not fit.

    • DEFAULT_ENCODING

      public static final ColumnEncoding DEFAULT_ENCODING
      Default encoding policy: ColumnEncoding.AUTO, leaving each column chunk's encoding to the size comparison the writer makes once the row group is buffered.

    • DEFAULT_BLOOM_FILTER_FPP

      public static final double DEFAULT_BLOOM_FILTER_FPP
      Default false-positive probability of a column's Bloom filter: 1%.
      See Also:

  • Method Details

    • defaults

      public static WriterConfig defaults()
      The default configuration.

    • builder

      public static WriterConfig.Builder builder()
      A builder pre-populated with the defaults.

    • pageTargetBytes

      public int pageTargetBytes()
      Encoded-byte threshold at which a data page is cut.

    • pageTargetRows

      public int pageTargetRows()
      Record count at which a data page is cut.

    • rowGroupBufferTargetBytes

      public long rowGroupBufferTargetBytes()
      Byte threshold at which a row group is cut, counted as the bytes the writer holds for it.

    • rowGroupTargetRows

      public long rowGroupTargetRows()
      Record count at which a row group is cut.

    • defaultEncoding

      public ColumnEncoding defaultEncoding()
      The encoding policy for columns with no override of their own.

    • columnEncodings

      public Map<String, ColumnEncoding> columnEncodings()
      The per-column encoding policies, keyed by dotted leaf path. Unmodifiable, and empty where no column was named.

    • bloomFilters

      public Map<String,Double> bloomFilters()
      The columns that carry a Bloom filter, keyed by dotted leaf path, each with its filter's false-positive probability. Unmodifiable, and empty where no column was named.

    • statisticsTruncationLength

      public int statisticsTruncationLength()
      The maximum length of a BYTE_ARRAY min / max statistics bound before it is truncated (and flagged inexact).

    • codec

      public CompressionCodec codec()
      The codec each page body is compressed with.

    • precisionLossPolicy

      public PrecisionLossPolicy precisionLossPolicy()
      What RowWriter does with a value carrying more precision than its column can hold.