Skip to content

S3 Reference

Look-it-up reference for the hardwood-s3 module. For worked examples — credentials, S3-compatible endpoints, and multi-file reads — see Reading from S3.

S3Source builder options

Option Default Effect
region(String) — AWS region. Required unless endpoint is set.
endpoint(String) — Custom endpoint for S3-compatible services (R2, GCS HMAC, MinIO). Region may be omitted when set.
pathStyle(boolean) false Path-style addressing, for services that require it (MinIO, SeaweedFS).
credentials(...) — Static S3Credentials, an S3CredentialsProvider, or the AWS chain via hardwood-aws-auth. Required.
connectTimeout(Duration) 10s Maximum time to establish a TCP connection. Ignored when a custom httpClient is supplied.
requestTimeout(Duration) 30s Maximum time for an individual HTTP request to complete.
maxRetries(int) 3 GET retries on transient failures (HTTP 500, 503, network errors) with exponential backoff and jitter. 0 disables retries.
httpClient(HttpClient) internal Caller-supplied client for full transport control. The caller is responsible for closing it — S3Source.close() will not.
rangeBacking(RangeBacking) RangeBacking.NONE Whether files opened from this source cache the byte ranges they fetch. See Range backing.
tempDir(Path) java.io.tmpdir Directory holding the sparse backing file under RangeBacking.SPARSE_TEMPFILE. Ignored under RangeBacking.NONE.

When the retry budget is exhausted, the last failure is re-thrown as an IOException. For HTTP errors the message has the form GET s3://<bucket>/<key> failed: HTTP <status>; for network errors the original IOException message is preserved.

open() issues one GET for the last 64 KB of the object — the whole object if it is smaller. That single round-trip discovers the object size, so there is never a separate HEAD request. Those bytes are kept for as long as the file is open, and any read falling entirely inside that trailing window is answered without a further request.

For most files the window also carries the Parquet footer, which is why opening a file usually costs one GET. It is a cost heuristic rather than a guarantee: footer size grows with row groups × columns and with the size of the per-column statistics, so a wide schema, a heavily split file, or long string min/max values can push the footer past 64 KB. Reading it then costs one extra GET and nothing more — the window bounds request count, never what is readable.

This applies to every S3InputFile, independent of the range backing below.

Range backing

RangeBacking sets whether an S3InputFile also keeps the bytes it fetches outside that window:

Value Behaviour
NONE (default) Nothing beyond the footer pre-fetch is kept. Every other readRange issues an HTTP GET, including a repeat read of a range already fetched.
SPARSE_TEMPFILE Fetched ranges are written into a sparse temp file mmapped into the process. A read covered by already-fetched bytes is served from the mapping with no HTTP GET; a read spanning a gap fetches only the missing sub-ranges.

The value changes how many HTTP requests go out, not what a read returns — both give the reader identical bytes. The cache is per S3InputFile and lives until that file is closed.

SPARSE_TEMPFILE applies to a whole S3Source, so it suits workloads that re-read the same byte ranges — hardwood dive opts in for every file it opens. Its properties:

  • Disk and memory footprint scale with bytes fetched, not with file size, on filesystems that support sparse files (ext4, xfs, apfs, ntfs).
  • Files larger than Integer.MAX_VALUE bytes (2 GB) cannot be range-backed; open() throws an IOException naming the limit. Read those with RangeBacking.NONE.
  • The backing file is created in tempDir, which must exist and be writeable with room for the worst-case bytes fetched per file. S3Source.builder().build() rejects a directory that is missing or not writeable.
  • close() deletes the backing file, but the mapping behind it is released by the garbage collector, so the address space — and on Windows, where a mapped file cannot be deleted, the file itself until JVM exit — outlives the close() call.

Fetch-cost counters

Each S3InputFile tracks the network I/O it has performed since open():

  • networkRequestCount() — HTTP GET requests issued: the footer pre-fetch plus every column-chunk range fetch. Reads answered from the pre-fetched footer window are not counted, nor are reads answered from the cache under RangeBacking.SPARSE_TEMPFILE.
  • networkBytesFetched() — total bytes fetched across those requests, counted the same way.

Read the counters before the try-with-resources block exits, while the S3InputFile is still open.

Not currently supported

  • Anonymous (unsigned) requests — S3Credentials requires an access key + secret key pair; there is no built-in mode for public buckets without credentials.
  • Requester-pays buckets — Hardwood does not send the x-amz-request-payer header. Reads against requester-pays buckets will fail with 403 AccessDenied even with valid credentials.