Reading from S3

The hardwood-s3 module reads Parquet files directly from Amazon S3 and S3-compatible services (Cloudflare R2, GCP Cloud Storage via HMAC keys, MinIO) without downloading them first. Column projection, row group filtering and all other reader features work with S3 files.

Try it yourself

Want to run it or explore the capabilities yourself? The Reading from S3 example streams a Parquet file straight from an S3-compatible endpoint, proves projection only fetches the byte ranges it needs via networkBytesFetched() / networkRequestCount(), and opens a whole bucket as one dataset with Hardwood.openAll(...).

<dependency>
    <groupId>dev.hardwood</groupId>
    <artifactId>hardwood-s3</artifactId>
</dependency>

Credentials

Read a file with static credentials:

import dev.hardwood.s3.S3Credentials;
import dev.hardwood.s3.S3Source;
import dev.hardwood.reader.ParquetFileReader;
import dev.hardwood.reader.RowReader;

S3Source source = S3Source.builder()
        .region("us-east-1")
        .credentials(S3Credentials.of("AKIA...", "secret"))
        .build();

try (ParquetFileReader reader = ParquetFileReader.open(
        source.inputFile("s3://my-bucket/data/trips.parquet"))) {
    try (RowReader rows = reader.rowReader()) {
        while (rows.hasNext()) {
            rows.next();
            long id = rows.getLong("id");
        }
    }
}

For dynamic or refreshable credentials, implement the S3CredentialsProvider functional interface:

S3Source source = S3Source.builder()
        .region("us-east-1")
        .credentials(() -> fetchCredentialsFromVault())
        .build();

To resolve credentials through the AWS credential chain (environment variables, profile files, container and instance metadata), add the optional hardwood-aws-auth module:

<dependency>
    <groupId>dev.hardwood</groupId>
    <artifactId>hardwood-aws-auth</artifactId>
</dependency>

import dev.hardwood.aws.auth.SdkCredentialsProviders;

S3Source source = S3Source.builder()
        .region("us-east-1")
        .credentials(SdkCredentialsProviders.defaultChain())
        .build();

defaultChain() delegates to the AWS SDK's DefaultCredentialsProvider, which decides the order in which the sources are tried. SdkCredentialsProviders.fromProfile("my-profile") binds one named profile from ~/.aws/credentials instead. SSO and web-identity profiles need the AWS SDK's sso, ssooidc and sts modules on the classpath; hardwood-aws-auth does not bring them.

Multiple Files

Read multiple files in the same bucket:

import dev.hardwood.Hardwood;

try (Hardwood hardwood = Hardwood.create();
     ParquetFileReader parquet = hardwood.openAll(
             source.inputFilesInBucket("my-bucket",
                     "data/part-001.parquet",
                     "data/part-002.parquet",
                     "data/part-003.parquet"));
     RowReader reader = parquet.rowReader()) {
    while (reader.hasNext()) {
        reader.next();
        // ...
    }
}

Read multiple files across buckets:

hardwood.openAll(source.inputFiles(
        "s3://bucket-a/events.parquet",
        "s3://bucket-b/events.parquet"));

S3-Compatible Services

Set a custom endpoint for S3-compatible services:

// Cloudflare R2
S3Source source = S3Source.builder()
        .endpoint("https://<account-id>.r2.cloudflarestorage.com")
        .credentials(S3Credentials.of(accessKeyId, secretKey))
        .build();

// GCP Cloud Storage (HMAC keys)
S3Source source = S3Source.builder()
        .endpoint("https://storage.googleapis.com")
        .credentials(S3Credentials.of(hmacAccessId, hmacSecret))
        .build();

// MinIO (path-style)
S3Source source = S3Source.builder()
        .endpoint("http://localhost:9000")
        .pathStyle(true)
        .credentials(S3Credentials.of(accessKeyId, secretKey))
        .build();

When a custom endpoint is set, region can be omitted. Use .pathStyle(true) for services that require path-style access (e.g. MinIO, SeaweedFS).

Configuration

The builder sets timeouts, retries and the HTTP client. The S3 Reference lists every option with its default.

S3Source source = S3Source.builder()
        .region("us-east-1")
        .credentials(S3Credentials.of("AKIA...", "secret"))
        .connectTimeout(Duration.ofSeconds(5))
        .requestTimeout(Duration.ofSeconds(60))
        .maxRetries(5)            // 0 disables retries
        .build();

To control connection pooling, proxy settings or other transport configuration, pass a pre-configured HttpClient with .httpClient(client). The caller closes that client, and connectTimeout does not apply to it.

Limiting Fetched Data

To fetch only the pages covering the rows a partial read needs, set head(n) on the buildRowReader() builder. I/O behavior describes what each read fetches.

Reusing Footers Across Opens

A process that opens the same objects repeatedly can take each object's footer from its own cache, through a MetadataSource on the HardwoodContext (see Reuse a parsed footer across readers for the recipe). An S3 object's InputFile.identity() is its ETag, read from the response to the GET every open makes, so a cache keyed on the object's URI and ETag misses when the object is replaced, at no extra request. On a hit, opening an object costs that one GET whatever the footer's size (Footer pre-fetch).

Measuring Fetch Cost

Each S3InputFile counts the GET requests it issued and the bytes it fetched:

import dev.hardwood.s3.S3InputFile;

try (S3InputFile in = source.inputFile("s3://my-bucket/data/trips.parquet");
     ParquetFileReader reader = ParquetFileReader.open(in);
     RowReader rows = reader.rowReader()) {
    while (rows.hasNext()) {
        rows.next();
        // ...
    }
    System.out.printf("fetched %d bytes over %d requests%n",
            in.networkBytesFetched(), in.networkRequestCount());
}

The S3 Reference defines what each counter includes and when to read it.

Anonymous (unsigned) requests and requester-pays buckets are not supported.