Class WriterConfig.Builder

java.lang.Object
dev.hardwood.writer.WriterConfig.Builder
Enclosing class:
WriterConfig
public static final class WriterConfig.Builder extends Object
Builder for WriterConfig.

  • Method Details

    • pageTargetBytes

      public WriterConfig.Builder pageTargetBytes(int pageTargetBytes)
      Sets the page target; must be at least one INT32 (4 bytes).

    • rowGroupBufferTargetBytes

      public WriterConfig.Builder rowGroupBufferTargetBytes(long rowGroupBufferTargetBytes)

      Sets the byte threshold at which a row group is cut; must be positive.

      The bytes counted are the bytes the writer holds for the open row group: the level streams, the dictionary indices, the value stores and the dictionaries. This is the writer's memory control, and peak heap follows it. Two things sit on top: the buffers hold more than they are charged for while they grow — the value stores by half again, the level streams and packed content and every dictionary's arrays by double — and a schema with enough columns that each one's share falls below the floor under a column's buffers opens at a multiple of the target. Neither grows with how much is written.

      It is not the size the row group takes on disk. That is smaller by whatever the encoding and the codec win, which is a property of the data rather than of this setting. A row group passes this threshold by at most one record, a record not being divisible across row groups.

    • rowGroupTargetRows

      public WriterConfig.Builder rowGroupTargetRows(long rowGroupTargetRows)

      Sets the record count at which a row group is cut; must be positive.

      This is the control over how a file is banded, and the row groups it produces hold exactly this many records apart from the last. A row group is cut at this count or at rowGroupBufferTargetBytes, whichever is reached first, so a row target set above what the buffer target allows has no effect beyond it.

      A structural ceiling sits under both: a chunk accumulates into int-indexed buffers, so a row group holds at most Integer.MAX_VALUE - 8 records however many are asked for. A target above that is that ceiling, which is what makes Long.MAX_VALUE the way to say "no row limit, cut on bytes alone".

    • encoding

      public WriterConfig.Builder encoding(ColumnEncoding encoding)

      Sets the encoding policy for every column without an override of its own; must be non-null. Defaults to ColumnEncoding.AUTO.

      A default that no column of the schema can carry is rejected when the writer is created, so a file-wide BYTE_STREAM_SPLIT over a schema holding a BYTE_ARRAY column fails rather than quietly resolving that column to something else.

    • encoding

      public WriterConfig.Builder encoding(String columnPath, ColumnEncoding encoding)

      Sets the encoding policy for one leaf column, overriding the file-wide default; both arguments must be non-null.

      The column is named by its dotted leaf path as the schema spells it, synthetic list.element and key_value.key segments included — readings.list.element, not readings. A path matching no leaf column of the schema, or a policy its physical type cannot carry, is rejected when the writer is created.

    • statisticsTruncationLength

      public WriterConfig.Builder statisticsTruncationLength(int statisticsTruncationLength)
      Sets the maximum BYTE_ARRAY min / max statistics bound length; must be positive. A bound longer than this is truncated and flagged inexact.

    • codec

      public WriterConfig.Builder codec(CompressionCodec codec)

      Sets the codec each page body is compressed with; must be non-null.

      UNCOMPRESSED, GZIP, SNAPPY, ZSTD, LZ4_RAW and BROTLI are written. Everything but the first two needs its library on the classpath, which is checked when the writer is created rather than here.

      The other two members of CompressionCodec are not produced, and neither is waiting on a later release. LZ4 names the Hadoop framing the format deprecated in favour of LZ4_RAW; files already written with it are still read, so the refusal is on this side only. LZO has no maintained JVM implementation and is refused in both directions. Asking for either fails when the writer is created.

    • precisionLossPolicy

      public WriterConfig.Builder precisionLossPolicy(PrecisionLossPolicy precisionLossPolicy)
      Sets what RowWriter does with a value carrying more precision than its column can hold — an Instant with microseconds written to a TIMESTAMP(MILLIS) column, say; must be non-null. Defaults to PrecisionLossPolicy.REJECT.

    • build

      public WriterConfig build()
      Builds the immutable configuration.