Row-Oriented Writing

RowWriter writes one record at a time, addressing fields by the names they carry in the schema and taking logical-type values as the Java types the reader returns for them. It is the write-side mirror of RowReader.

Experimental API

RowWriter, RowWriteResult, StructBuilder, ListBuilder and MapBuilder are annotated @Experimental: their shape may change in a future release.

Writing a File

Writing takes three objects: a FileSchema describing what the file holds, an OutputFile naming where it goes, and a ParquetFileWriter joining them.

import dev.hardwood.OutputFile;
import dev.hardwood.metadata.LogicalType;
import dev.hardwood.metadata.PhysicalType;
import dev.hardwood.metadata.RepetitionType;
import dev.hardwood.schema.FileSchema;
import dev.hardwood.writer.ParquetFileWriter;
import dev.hardwood.writer.RowWriter;
import java.nio.file.Path;
import java.time.LocalDate;

FileSchema schema = FileSchema.builder("person")
        .addColumn("id", PhysicalType.INT64, RepetitionType.REQUIRED)
        .addColumn("name", PhysicalType.BYTE_ARRAY, RepetitionType.REQUIRED, LogicalType.string())
        .addColumn("hired", PhysicalType.INT32, RepetitionType.OPTIONAL, LogicalType.date())
        .build();

try (ParquetFileWriter writer = ParquetFileWriter.create(OutputFile.of(Path.of("people.parquet")), schema)) {
    RowWriter rows = writer.rowWriter();

    rows.writeRow(row -> row
            .setLong("id", 1)
            .setString("name", "Ada")
            .setDate("hired", LocalDate.of(2019, 3, 1)));

    rows.writeRow(row -> row
            .setLong("id", 2)
            .setString("name", "Alan")
            .setDate("hired", LocalDate.of(2021, 11, 15)));
}

The writer creates the StructBuilder, hands it to the filler, and stages the record when the filler returns — there is no separate build or submit step. RowWriter is not closeable: the ParquetFileWriter owns the file, and closing it writes the staged records along with the footer. A file is written through one API: calling both rowWriter() and columnWriter() on the same ParquetFileWriter is rejected.

The footer is written last, so the file is valid only once close() returns. After a failure, see Handle Write Failures.

Typed Setters

Each setter names the type of the field it writes, and a field's declared type decides which setter fits. Logical-type setters such as setDate and setDecimal convert the value to the column's physical representation. The full setter map is under Logical Types and Row Setters.

Given a schema that also declares a DECIMAL salary, a TIMESTAMP created_at and a UUID external_id:

rows.writeRow(row -> row
        .setLong("id", 7)
        .setString("name", "Grace")
        .setDecimal("salary", new BigDecimal("81500.00"))
        .setTimestamp("created_at", Instant.now())
        .setUuid("external_id", UUID.randomUUID()));

A setter that does not fit the field's declared type is rejected, as is a value outside the range the field's annotation declares; see Value Ranges.

A value carrying more precision than its column can hold — an Instant with microseconds written to a TIMESTAMP(MILLIS) column — is rejected by default. Configure precisionLossPolicy(PrecisionLossPolicy.TRUNCATE) to drop the digits that do not fit instead.

Nulls and Unset Fields

An OPTIONAL field is null in three equivalent spellings: passing null to an object-typed setter, calling setNull, and leaving the field unset.

rows.writeRow(row -> row
        .setLong("id", 3)
        .setString("name", "Katherine")
        .setNull("hired"));

// Same record: "hired" is never mentioned.
rows.writeRow(row -> row
        .setLong("id", 3)
        .setString("name", "Katherine"));

A REQUIRED field left unset fails the record, and a failed record fails the writer; see Handle Write Failures.

Skipping a Rejected Record

tryWriteRow stages the record or returns a RowWriteResult.Rejected without failing the writer. Later records can still be written, and close() publishes the ones that staged.

import dev.hardwood.writer.RowWriteResult;

try (ParquetFileWriter writer = ParquetFileWriter.create(OutputFile.of(Path.of("people.parquet")), schema)) {
    RowWriter rows = writer.rowWriter();
    for (Person person : people) {
        if (rows.tryWriteRow(row -> row
                .setLong("id", person.id())
                .setString("name", person.name())) instanceof RowWriteResult.Rejected rejected) {
            System.err.println("Skipped " + person.id() + ": " + rejected.message());
        }
    }
}

With the schema above, a person whose name is null is skipped with the message Field name is REQUIRED; it must be set to a non-null value in every record, and close() publishes every other person.

Which failures tryWriteRow returns as Rejected, and which still fail the writer, is listed under RowWriteResult.

Structs, Lists, and Maps

Nesting is entered with a filler per level. Fields are addressed by their user-visible names throughout — the synthetic list.element and key_value path segments Parquet's nested layout introduces never appear.

FileSchema schema = FileSchema.builder("person")
        .addColumn("id", PhysicalType.INT64, RepetitionType.REQUIRED)
        .struct("address", RepetitionType.OPTIONAL, address -> address
                .addColumn("city", PhysicalType.BYTE_ARRAY, RepetitionType.REQUIRED, LogicalType.string())
                .addColumn("zip", PhysicalType.BYTE_ARRAY, RepetitionType.OPTIONAL, LogicalType.string()))
        .list("phones", RepetitionType.OPTIONAL,
                element -> element.primitive(PhysicalType.BYTE_ARRAY, RepetitionType.REQUIRED,
                        LogicalType.string()))
        .map("props", RepetitionType.OPTIONAL, PhysicalType.BYTE_ARRAY, LogicalType.string(),
                value -> value.primitive(PhysicalType.INT64, RepetitionType.OPTIONAL))
        .build();

rows.writeRow(row -> row
        .setLong("id", 1)
        .setStruct("address", address -> address
                .setString("city", "Berlin")
                .setString("zip", "10115"))
        .setList("phones", phones -> phones
                .addString("+49 30 1234")
                .addString("+49 30 5678"))
        .setMap("props", props -> props
                .addEntry(entry -> entry.setString("key", "reads").setLong("value", 12))
                .addEntry(entry -> entry.setString("key", "writes").setNull("value"))));

A map entry is a struct with the two fields key and value, so addEntry takes a StructBuilder like any other struct.

Lists distinguish three states, and each has its own spelling:

State	How to write it
A list with entries	setList("phones", phones -> phones.addString("…"))
An empty list	setList("phones", phones -> { })
An absent (null) list	setNull("phones"), or leave the field unset

addNull() appends a null entry to a list whose element is OPTIONAL; on a list of REQUIRED elements it is rejected. Lists nest through addList and addStruct:

rows.writeRow(row -> row
        .setList("people", people -> people
                .addStruct(person -> person.setString("name", "Ada"))
                .addStruct(person -> person.setString("name", "Alan")))
        .setList("grid", grid -> grid
                .addList(inner -> inner.addInt(1).addInt(2))
                .addList(inner -> inner.addInt(3))));

rowWriter() rejects the legacy two-level lists, naming the group; Column by Column writes them. See Schema Shapes.

A builder is valid only inside the filler it was handed to. Retaining one and using it after its scope has ended is rejected rather than writing into a later record.

Addressing Fields by Index

Every by-name setter has a by-index twin, addressing a field by its position in the struct being filled. getFieldCount() and getFieldName(int) report the positions, which lets code that walks a record's fields uniformly write them back without knowing their names.

// person is the record being written; its fields are matched to the schema's by name.
rows.writeRow(row -> {
    for (int field = 0; field < row.getFieldCount(); field++) {
        switch (row.getFieldName(field)) {
            case "id" -> row.setLong(field, person.id());
            case "name" -> row.setString(field, person.name());
            default -> row.setNull(field);
        }
    }
});

Every rule of the by-name form applies unchanged, and an index outside [0, getFieldCount()) takes the place of an unknown name.

The reader's index and the writer's index are not always the same position

On the reader, a field index is a position among an accessor's projected children. On the writer, it is a position in the struct as declared in the schema being written. The two agree when a whole file is read into its own schema — copying a record field by field then works — and diverge as soon as a projection drops or reorders fields. Resolve the name with getFieldName(int) on both sides when the two schemas are not identical.

Configuring the Writer

Pass a WriterConfig to create:

import dev.hardwood.writer.PrecisionLossPolicy;
import dev.hardwood.writer.WriterConfig;

WriterConfig config = WriterConfig.builder()
        .precisionLossPolicy(PrecisionLossPolicy.TRUNCATE)
        .build();

try (ParquetFileWriter writer = ParquetFileWriter.create(out, schema, config)) {
    // ...
}

Every option, its default and what it rejects is under Writer Options. ParquetFileWriter.endRowGroup() closes the open row group after the records written so far; see Ending a Row Group. The footer's key-value metadata and created_by identifier are set on the ParquetFileWriter until close(); see File Metadata.