Add reproducible S3 storage benchmark

Benchmark S3-compatible providers through one Files SDK workload with network preflights, integrity checks, balanced campaigns, comparison reports, tests, documentation, and CI.

Author: leoisadev1

.env.example

@@ -0,0 +1,47 @@
+# Choose: local, neon, r2, or railway
+BENCH_PROVIDER=local
+
+# Shared benchmark settings
+BENCH_SIZES=1024,1048576,8388608
+BENCH_FILES_PER_SIZE=3
+BENCH_CONCURRENCY=2
+BENCH_WARMUP_RUNS=1
+BENCH_RUNS=3
+BENCH_TIMEOUT_MS=120000
+BENCH_PREFIX=s3-bench
+BENCH_KEEP_OBJECTS=false
+BENCH_OUTPUT_DIR=.results
+BENCH_NETWORK_PREFLIGHT=true
+BENCH_NETWORK_DOWNLOAD_BYTES=10000000
+BENCH_NETWORK_UPLOAD_BYTES=2000000
+
+# Local Files SDK filesystem adapter
+BENCH_LOCAL_ROOT=.bench-data
+
+# Neon S3-compatible storage
+NEON_BUCKET=
+AWS_ENDPOINT_URL_S3=https://your-branch.storage.your-region.aws.neon.tech
+AWS_ACCESS_KEY_ID=
+AWS_SECRET_ACCESS_KEY=
+AWS_REGION=us-east-2
+
+# Cloudflare R2 HTTP API
+R2_BUCKET=
+R2_ACCOUNT_ID=
+R2_ACCESS_KEY_ID=
+R2_SECRET_ACCESS_KEY=
+
+# Railway Storage Bucket
+RAILWAY_S3_ENDPOINT=https://storage.railway.app
+RAILWAY_S3_REGION=auto
+RAILWAY_S3_BUCKET=
+RAILWAY_S3_ACCESS_KEY_ID=
+RAILWAY_S3_SECRET_ACCESS_KEY=
+RAILWAY_S3_FORCE_PATH_STYLE=false
+
+# Pin AWS SDK v3 to one total attempt so retries cannot hide latency or failures
+AWS_MAX_ATTEMPTS=1
+
+# Railway/Bun HTTP service
+PORT=3000
+BENCH_API_TOKEN=

.github/workflows/ci.yml

@@ -0,0 +1,20 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: 1.3.14
+      - run: bun install --frozen-lockfile
+      - run: bun run typecheck
+      - run: bun test

.gitignore

@@ -0,0 +1,8 @@
+node_modules/
+.env
+.env.*
+!.env.example
+.results/
+.bench-data/
+coverage/
+.DS_Store

CONTRIBUTING.md

@@ -0,0 +1,27 @@
+# Contributing
+
+Thanks for helping improve S3 Bench.
+
+## Development
+
+1. Install [Bun](https://bun.sh/) 1.2 or newer.
+2. Run `bun install`.
+3. Copy `.env.example` to `.env` only if you need a remote provider.
+4. Make the smallest change that solves the problem.
+5. Run `bun run typecheck` and `bun test` before opening a pull request.
+
+Use the local filesystem adapter for routine development and tests:
+
+```bash
+bun run benchmark:local
+```
+
+## Benchmark changes
+
+Changes to timing, concurrency, retries, payload generation, integrity checks, aggregation, or provider ordering can invalidate comparisons. Include a test or a reproducible report showing the behavior, and explain any methodology change in the pull request.
+
+Never commit credentials, signed URLs, bucket identifiers, raw `.env` files, or generated `.results/` reports. Remote benchmarks may incur storage, request, and egress charges, so use isolated test buckets and keep cleanup enabled.
+
+## Pull requests
+
+Keep pull requests focused. Describe what changed, why it is fair across providers, and which validation commands you ran.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 S3 Bench contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,170 @@
+# S3 Bench
+
+A reproducible object-storage benchmark for S3-compatible providers, built on [Files SDK](https://files-sdk.dev/). It runs the same upload, HEAD, download, LIST, integrity, and cleanup workload against local storage, Neon, Cloudflare R2, and Railway Storage Buckets.
+
+S3 Bench records raw samples and produces both JSON and readable Markdown reports. It is designed for honest provider comparisons, so retries are disabled, downloads are SHA-256 verified, network conditions are recorded, and public campaigns rotate provider order.
+
+## Quick start
+
+Requires [Bun](https://bun.sh/) 1.2 or newer.
+
+```bash
+bun install
+cp .env.example .env
+chmod 600 .env
+bun run benchmark:local
+```
+
+The local benchmark needs no cloud credentials. Reports are written to `.results/`, and temporary objects are removed after each run.
+
+## What it measures
+
+The default workload uses 1 KiB, 1 MiB, and 8 MiB objects, with three objects per size, concurrency two, one warmup cycle, and three measured cycles.
+
+Each measured cycle:
+
+1. Uploads every object.
+2. Runs HEAD and verifies the stored byte size.
+3. Downloads every object and verifies its length and SHA-256 hash.
+4. Lists the run prefix and verifies every expected key exists.
+5. Deletes every uploaded object in a `finally` block.
+
+Within upload, HEAD, and download, each size runs as a separate concurrent phase. Measured cycles rotate the size order so one size does not permanently benefit from running first or last.
+
+Before each provider run, S3 Bench also records a Cloudflare HTTP preflight consisting of a small request, a 10 MB download, and a 2 MB upload. This is a connection snapshot, not a direct latency measurement to the storage provider.
+
+## Configure a provider
+
+Copy `.env.example` to `.env`, then fill in one provider section. Keep `.env` local and mode `600`.
+
+### Neon
+
+```dotenv
+NEON_BUCKET=your-bucket
+AWS_ENDPOINT_URL_S3=https://your-branch.storage.your-region.aws.neon.tech
+AWS_ACCESS_KEY_ID=generated-key-id
+AWS_SECRET_ACCESS_KEY=generated-secret
+AWS_REGION=us-east-2
+```
+
+The Neon adapter uses path-style addressing through Files SDK's AWS SDK v3 S3 client.
+
+### Cloudflare R2
+
+```dotenv
+R2_BUCKET=your-bucket
+R2_ACCOUNT_ID=your-account-id
+R2_ACCESS_KEY_ID=generated-key-id
+R2_SECRET_ACCESS_KEY=generated-secret
+```
+
+The benchmark explicitly selects Files SDK's AWS SDK client path for R2 instead of its optional fetch client.
+
+### Railway Storage Bucket
+
+```dotenv
+RAILWAY_S3_ENDPOINT=https://storage.railway.app
+RAILWAY_S3_REGION=auto
+RAILWAY_S3_BUCKET=your-bucket
+RAILWAY_S3_ACCESS_KEY_ID=generated-key-id
+RAILWAY_S3_SECRET_ACCESS_KEY=generated-secret
+RAILWAY_S3_FORCE_PATH_STYLE=false
+```
+
+Use the values and URL style shown in Railway's Bucket credentials tab. Older buckets can require path-style URLs. Railway documents that its Storage Buckets run on Tigris infrastructure, but S3 Bench treats Railway Bucket as the product being measured.
+
+All remote providers use the same Files SDK AWS SDK v3 S3Client path. Files SDK retries are disabled, and `AWS_MAX_ATTEMPTS=1` limits the underlying AWS SDK to one total attempt so retries cannot hide failures or inflate latency.
+
+## Run benchmarks
+
+Run one provider:
+
+```bash
+bun run benchmark -- --provider r2
+bun run benchmark -- --provider neon
+bun run benchmark -- --provider railway
+```
+
+Override the workload when needed:
+
+```bash
+bun run benchmark -- \
+  --provider r2 \
+  --sizes 16777216,67108864,104857600 \
+  --files 2 \
+  --concurrency 2 \
+  --warmups 1 \
+  --runs 3
+```
+
+Sizes are bytes. Use `--skip-network` only for tests or offline development. `--keep` leaves uploaded objects behind; cleanup is enabled by default.
+
+### Balanced public campaign
+
+A single sequential pass is vulnerable to provider-order and transient network effects. The campaign command runs three Latin-square rounds, so every provider appears once in each sequence position.
+
+```bash
+bun run campaign
+bun run publication
+```
+
+The publication generator validates that all nine reports used an identical workload, one AWS SDK attempt, balanced positions, and zero failures before producing aggregate and per-size tables.
+
+`bun run compare` remains available for a quick comparison of the latest individual reports, but the balanced campaign is the appropriate path for published results.
+
+### Large objects
+
+The shared `files.upload()` path is a single object upload. Provider limits can differ. Neon accepted 100 MiB in testing but rejected 128 MiB and 256 MiB single uploads; Neon recommends multipart upload through the AWS SDK for larger files. Multipart is outside this benchmark's shared Files SDK operation and should be tested as a separate workload rather than mixed into these results.
+
+## Reports
+
+Each run writes:
+
+- JSON containing every raw operation sample, workload configuration, runtime context, network preflight, integrity failures, cleanup errors, and run errors.
+- Markdown containing p50 and p95 request latency plus aggregate wall-clock throughput.
+
+Generated reports live in `.results/` and are intentionally ignored by Git because they may contain machine, bucket, endpoint, and timing metadata. Review any report before publishing it.
+
+## HTTP service
+
+S3 Bench can run as a small authenticated Bun service, including on Railway. Set provider credentials and a strong `BENCH_API_TOKEN`, then start it with:
+
+```bash
+bun run start
+```
+
+```bash
+curl -X POST https://YOUR-SERVICE.example/benchmark \
+  -H "Authorization: Bearer $BENCH_API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"provider":"neon"}'
+```
+
+Health check:
+
+```bash
+curl https://YOUR-SERVICE.example/health
+```
+
+Only one HTTP benchmark runs at a time. Attach persistent storage or retain the returned JSON if reports must survive redeploys.
+
+## Interpreting results
+
+Benchmark numbers are observations, not universal provider rankings. Region placement, Wi-Fi conditions, internet routing, object size, concurrency, hot versus cold reads, and provider limits all affect the outcome. Compare per-size tables, inspect round consistency, and publish the recorded limitations with the numbers.
+
+Running this project against paid storage can incur request, storage, and egress charges. Start with the local adapter, use isolated test buckets, and keep cleanup enabled.
+
+## Development
+
+```bash
+bun run typecheck
+bun test
+```
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). Security issues should follow [SECURITY.md](SECURITY.md).
+
+## License
+
+[MIT](LICENSE)

SECURITY.md

@@ -0,0 +1,15 @@
+# Security Policy
+
+## Reporting a vulnerability
+
+Please do not open a public issue for vulnerabilities involving credential exposure, authentication bypass, arbitrary object access, or destructive storage behavior.
+
+Report the issue privately through GitHub's security advisory feature for this repository. Include reproduction steps, affected versions or commits, and the potential impact. Do not include live cloud credentials or signed URLs.
+
+## Supported versions
+
+Until the project publishes versioned releases, security fixes apply to the latest commit on the default branch.
+
+## Credential safety
+
+S3 Bench reads cloud credentials from environment variables. `.env`, generated reports, local benchmark data, and dependency directories are ignored by Git, but contributors should still review changes and reports before sharing them publicly.