S3 Reference

Options and behavior of the hardwood-s3 module; Reading from S3 and Write to S3 show how to use it.

S3Source builder options

S3Source is the connection to one S3-compatible service. The same instance creates S3InputFiles for reading and S3OutputFiles for writing; S3Source.builder() takes these options:

Option	Default	Effect
region(String)	—	AWS region. Required unless endpoint is set.
endpoint(String)	—	Custom endpoint for S3-compatible services (R2, GCS HMAC, MinIO). Region may be omitted when set.
pathStyle(boolean)	false	Path-style addressing, for services that require it (MinIO, SeaweedFS).
credentials(...)	—	Static S3Credentials, an S3CredentialsProvider, or the AWS chain via hardwood-aws-auth. Required unless anonymous() is set.
anonymous()	off	Sends requests unsigned, for buckets that allow public access. Mutually exclusive with credentials(...).
connectTimeout(Duration)	10s	Maximum time to establish a TCP connection. Ignored when a custom httpClient is supplied.
requestTimeout(Duration)	30s	Maximum time for an individual HTTP request to receive its response headers, including the upload of one part, and then for each gap between bytes of the response body. A body that stops sending bytes fails after one to two timeouts.
maxRetries(int)	3	Retries per request on transient failures (HTTP 500, 503, network errors) with exponential backoff and jitter. 0 disables retries.
httpClient(HttpClient)	internal	Caller-supplied client for full transport control. The caller is responsible for closing it; S3Source.close() will not.
rangeBacking(RangeBacking)	RangeBacking.NONE	Whether files opened from this source cache the byte ranges they fetch. See Range backing.
tempDir(Path)	java.io.tmpdir	Directory holding the sparse backing file under RangeBacking.SPARSE_TEMPFILE. Ignored under RangeBacking.NONE.
uploadPartSize(int)	8 MiB	Part size of uploads through outputFile(...), at least 5 MiB. See Writing.

When the retry budget is exhausted, the read or write fails with an IOException. For a read with an HTTP error status, the message has the form Failed to open s3://<bucket>/<key>: HTTP <status> <body> when opening the file and Failed to read range [<start>, <end>) from s3://<bucket>/<key>: HTTP <status> <body> for a range read. A range read whose connection fails, before or during its body, is retried for the bytes not yet received, and the bytes received stay; for an object without a strong ETag the retry reads the whole range again. When the retries run out, the message has the form Failed to read range [<start>, <end>) from s3://<bucket>/<key>: received <n> of <length> bytes in <attempts> attempts, and the cause carries the last failure. For other network errors the original IOException message is preserved. A response body that stops sending bytes fails like a lost connection; for a read the message has the form GET s3://<bucket>/<key> range <range> timed out: no body bytes for <timeout> ms, where <range> is the request's Range header value, such as bytes=0-99, and for a write it names the request, such as PUT s3://<bucket>/<key> timed out: no body bytes for <timeout> ms. When an anonymous source receives HTTP 401 or 403, the message ends with (the request was sent unsigned, without credentials).

A range read must be answered with HTTP 206. An endpoint that ignores the Range header and answers HTTP 200 with the whole object makes the read fail with an IOException whose message has the form Failed to read range [<start>, <end>) from s3://<bucket>/<key>: HTTP 200, expected 206; the endpoint ignored the Range header.

Footer pre-fetch

open() issues one GET for the last 256 KiB of the object, or the whole object if it is smaller. That single round-trip discovers the object size, so there is never a separate HEAD request. A read falling entirely inside that trailing window is answered without a further request, so reading an object of up to 256 KiB costs that one GET. The reader releases the window once it has read past the file. An HTTP 200 answer to this GET is accepted when the whole object fits in 256 KiB; the object then forms the window. A longer answer fails the open with an IOException whose message has the form Failed to open s3://<bucket>/<key>: HTTP <status> with a body longer than 262144 bytes requested; the endpoint ignored the Range header.

For most files the window also carries the Parquet footer and the page index, which is why opening a file usually costs one GET. It is a cost heuristic rather than a guarantee: footer size grows with row groups × columns and with the size of the per-column statistics, so a wide schema, a heavily split file, or long string min/max values can push the footer past 256 KiB. Reading it then costs one extra GET. When the context's MetadataSource supplies the footer, the open checks only the 8-byte trailer, which lies in the window, so that extra GET is not made.

The ETag header of this response is the object's InputFile.identity(), quotes included. It is empty when the endpoint sends no ETag. Every later range request carries a strong ETag as If-Match, so a read never mixes bytes of two versions of the object. When the object has been replaced since the open, the endpoint answers HTTP 412, and the read fails with a ParquetReadException whose message has the form Failed to read range [<start>, <end>) from s3://<bucket>/<key>: the file changed since it was opened (HTTP 412 for If-Match <etag>). Opening the file again reads the new version. Without an ETag, or with a weak one (W/"<tag>"), range requests are sent without If-Match, and a replaced object is not detected.

This applies to every S3InputFile, independent of the range backing below.

I/O behavior

Hardwood coalesces the column chunk reads within each row group into as few HTTP requests as possible. The page index of the columns a read uses is fetched for several row groups at a time, one request for the offset indexes and, on a filtered read, one for the column indexes and one for the bloom filters it checks; a filtered read may add one request per dictionary it checks. Column projection, page-level predicate pushdown, and head(n) on an unfiltered read narrow the byte ranges before coalescing.

Without a row limit, the reader fetches all projected (and filtered) column data for an entire row group when any column enters it, even if the consumer stops reading early. head(n) on the buildRowReader() builder truncates each column's fetch to the pages covering the needed rows. With a filter, head(n) counts matching rows, which are not known before the pages are decoded, so it does not narrow the fetch.

Range backing

RangeBacking sets whether an S3InputFile also keeps the bytes it fetches outside that window:

Value	Behaviour
NONE (default)	Nothing beyond the footer pre-fetch is kept. Every other non-empty readRange issues an HTTP GET, including a repeat read of a range already fetched.
SPARSE_TEMPFILE	Fetched ranges are written into a sparse temp file mmapped into the process. A read covered by already-fetched bytes is served from the mapping with no HTTP GET; a read spanning a gap fetches only the missing sub-ranges.

The value changes how many HTTP requests go out; both give the reader identical bytes. The cache is per S3InputFile and lives until that file is closed.

SPARSE_TEMPFILE applies to a whole S3Source, so it suits workloads that re-read the same byte ranges; hardwood dive opts in for every file it opens. Its properties:

• Disk and memory footprint scale with bytes fetched, not with file size, on filesystems that support sparse files (ext4, xfs, apfs, ntfs).
• Files larger than Integer.MAX_VALUE bytes (2 GB) cannot be range-backed; open() throws an UnsupportedOperationException naming the limit. Read those with RangeBacking.NONE.
• The backing file is created in tempDir, which must exist and be writeable with room for the worst-case bytes fetched per file. S3Source.builder().build() rejects a directory that is missing or not writeable.
• close() deletes the backing file, but the mapping behind it is released by the garbage collector, so the address space outlives the close() call. On Windows, where a mapped file cannot be deleted, the file itself also outlives it, until JVM exit.

Fetch-cost counters

Each S3InputFile tracks the network I/O it has performed since open():

• networkRequestCount() — HTTP GET requests issued: the footer pre-fetch plus every column-chunk range fetch. Reads answered from the pre-fetched footer window are not counted, nor are reads answered from the cache under RangeBacking.SPARSE_TEMPFILE.
• networkBytesFetched() — total bytes fetched across those requests, counted the same way.

Read the counters before the try-with-resources block exits, while the S3InputFile is still open.

Not supported

• Requester-pays buckets — Hardwood does not send the x-amz-request-payer header. Reads against requester-pays buckets will fail with 403 AccessDenied even with valid credentials.

Writing

S3Source.outputFile(String bucket, String key) and outputFile(String uri) (an s3://bucket/key URI) return an S3OutputFile, the OutputFile for an S3 object, for ParquetFileWriter. A source built with anonymous() cannot create one; the call throws IllegalStateException.

Aspect	Behavior
Buffering	Each open output holds up to five buffers of uploadPartSize bytes: one filling, four uploading
Requests	An object smaller than one part is sent in one PUT when the writer closes. A larger one is sent as a multipart upload: one UploadPart each time a buffer fills, with up to four uploading in the background while the writer continues, and CompleteMultipartUpload when the writer closes. With four uploads in flight, the writer waits for the oldest
Size limit	10,000 parts, so 10,000 × uploadPartSize bytes (about 78 GiB at the default). A write beyond it fails with an IOException
Visibility	Nothing appears at the key before the writer closes. A successful close replaces an object already at the key
Failure	A failed write, ParquetFileWriter.abort(), or a failed close aborts the multipart upload, releasing its parts. A failure to abort is attached to the thrown exception as suppressed
Retries	Every request, including the completion, is retried as maxRetries sets. A completion answered with 200 and an InternalError, SlowDown or ServiceUnavailable error is retried too
Timeout	Each part upload must complete within requestTimeout. Up to four parts upload at once, sharing the connection's bandwidth: at 30 s and 8 MiB parts, an upload bandwidth below about 1.1 MB/s times out
Bucket	Must exist

When close fails after the completing request reached S3 and its response was lost, the object may have been published although close throws.

The credentials need s3:PutObject and s3:AbortMultipartUpload on the key. An upload that cannot be aborted, for instance because the process ended, keeps its parts until a bucket lifecycle rule for incomplete multipart uploads removes them.

Errors have the form <request> s3://<bucket>/<key> failed: HTTP <status> <code>: <message>, for example PUT s3://my-bucket/events.parquet failed: HTTP 404 NoSuchBucket: The specified bucket does not exist.