Skip to content

Commit d06814a

Browse files
scarmuegascarmuega
authored
docs: add per-example READMEs and slim the examples showcase (#943)
1 parent fb02bbd commit d06814a

29 files changed

Lines changed: 938 additions & 114 deletions

File tree

‎docs/v2/examples/index.mdx‎

Lines changed: 42 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -28,76 +28,63 @@ for the feature flags, or use a Docker image that bundles them.
2828

2929
## Companion files & setup
3030

31-
Some examples need more than the `daemon.toml`. Check the example's folder before running:
32-
33-
- **Backing service via `docker-compose`** — start it first from the example directory
34-
(`docker compose up -d`): `kafka`, `redis`, `rabbitmq`, `elasticsearch`, `postgresql`,
35-
`redis_cursor`, `n2c_source`.
36-
- **Build step** — `wasm_basic` loads `./extract_fee/plugin.wasm`; the Go plugin under
37-
`extract_fee/` must be built first (see its README).
38-
- **Relative output/state created in the working directory** — `into_json`, `parse_cbor` and
39-
`file_rotate` write to `./output/`; `mithril` downloads to `./snapshot/`; `file_cursor`
40-
writes `my_cursor.json`; `sqlite` writes `./mydatabase.db` and ships an `init.sql` and a
41-
`Makefile`.
42-
- **Has its own README with extra steps** — `metadata_regex_filter`, `postgresql`,
43-
`redis_cursor`, `wasm_basic`.
44-
45-
> The `n2c_source` example sets `socket_path = "examples/n2c_source/node/node.socket"`, a path
46-
> written relative to the **repository root** (unlike the others). Run it from the repo root or
47-
> edit the path to point at your node socket.
31+
Some examples need more than the `daemon.toml` — a backing service (`docker compose up -d`),
32+
a build step, a cargo feature flag, or they write files to the working directory. Each
33+
example's own `README.md` documents its pipeline, prerequisites, and run command. Check it
34+
before running.
4835

4936
## Sources
5037

51-
| Example | What it shows | Setup |
52-
| :------ | :------------ | :---- |
53-
| [`n2c_source`](https://github.com/txpipe/oura/tree/main/examples/n2c_source) | Node-to-Client source over a unix socket → Stdout | docker compose · run from repo root |
54-
| [`hydra`](https://github.com/txpipe/oura/tree/main/examples/hydra) | Hydra head source over a WebSocket → Stdout | needs a running Hydra node |
55-
| [`mithril`](https://github.com/txpipe/oura/tree/main/examples/mithril) | Mithril snapshot bootstrap with a `Breadcrumbs` intersect | downloads `./snapshot` |
56-
| [`dolos_source`](https://github.com/txpipe/oura/tree/main/examples/dolos_source) | UTxO RPC (U5C) source against a Dolos/Demeter endpoint | needs a running Dolos (or a hosted U5C endpoint) |
38+
| Example | What it shows |
39+
| :------ | :------------ |
40+
| [`n2c_source`](https://github.com/txpipe/oura/tree/main/examples/n2c_source) | Node-to-Client source over a unix socket → Stdout |
41+
| [`hydra`](https://github.com/txpipe/oura/tree/main/examples/hydra) | Hydra head source over a WebSocket → Stdout |
42+
| [`mithril`](https://github.com/txpipe/oura/tree/main/examples/mithril) | Mithril snapshot bootstrap with a `Breadcrumbs` intersect |
43+
| [`dolos_source`](https://github.com/txpipe/oura/tree/main/examples/dolos_source) | UTxO RPC (U5C) source against a Dolos/Demeter endpoint |
5744

5845
## Filters & pipelines
5946

60-
| Example | What it shows | Setup |
61-
| :------ | :------------ | :---- |
62-
| [`select`](https://github.com/txpipe/oura/tree/main/examples/select) | `SplitBlock` → `ParseCbor` → `Select` by address → Stdout | — |
63-
| [`metadata_regex_filter`](https://github.com/txpipe/oura/tree/main/examples/metadata_regex_filter) | `Select` matching a metadata label + regex (preprod) → Stdout | see README |
64-
| [`parse_cbor`](https://github.com/txpipe/oura/tree/main/examples/parse_cbor) | `SplitBlock` → `ParseCbor` → JSONL file output | writes `./output` |
65-
| [`into_json`](https://github.com/txpipe/oura/tree/main/examples/into_json) | `ParseCbor` → `IntoJson` → compressed JSONL file output | writes `./output` |
66-
| [`wasm_basic`](https://github.com/txpipe/oura/tree/main/examples/wasm_basic) | Custom `WasmPlugin` filter loaded from a `.wasm` file | build step · `wasm` feature · README |
47+
| Example | What it shows |
48+
| :------ | :------------ |
49+
| [`select`](https://github.com/txpipe/oura/tree/main/examples/select) | `SplitBlock` → `ParseCbor` → `Select` by address → Stdout |
50+
| [`metadata_regex_filter`](https://github.com/txpipe/oura/tree/main/examples/metadata_regex_filter) | `Select` matching a metadata label + regex (preprod) → Stdout |
51+
| [`parse_cbor`](https://github.com/txpipe/oura/tree/main/examples/parse_cbor) | `SplitBlock` → `ParseCbor` → JSONL file output |
52+
| [`into_json`](https://github.com/txpipe/oura/tree/main/examples/into_json) | `ParseCbor` → `IntoJson` → compressed JSONL file output |
53+
| [`wasm_basic`](https://github.com/txpipe/oura/tree/main/examples/wasm_basic) | Custom `WasmPlugin` filter loaded from a `.wasm` file |
6754

6855
## Sinks
6956

70-
| Example | What it shows | Setup |
71-
| :------ | :------------ | :---- |
72-
| [`stdout`](https://github.com/txpipe/oura/tree/main/examples/stdout) | Legacy v1 events printed to standard output | — |
73-
| [`webhook_basics`](https://github.com/txpipe/oura/tree/main/examples/webhook_basics) | `WebHook` sink posting events over HTTP | needs an HTTP endpoint |
74-
| [`kafka`](https://github.com/txpipe/oura/tree/main/examples/kafka) | `Kafka` sink with `ByBlock` partitioning | docker compose · `kafka` feature |
75-
| [`rabbitmq`](https://github.com/txpipe/oura/tree/main/examples/rabbitmq) | `Rabbitmq` sink | docker compose · `rabbitmq` feature |
76-
| [`zeromq`](https://github.com/txpipe/oura/tree/main/examples/zeromq) | `Zeromq` PUSH socket sink | `zeromq` feature |
77-
| [`elasticsearch`](https://github.com/txpipe/oura/tree/main/examples/elasticsearch) | `ElasticSearch` sink | docker compose · `elasticsearch` feature |
78-
| [`redis`](https://github.com/txpipe/oura/tree/main/examples/redis) | `Redis` streams sink | docker compose · `redis` feature |
79-
| [`sqlite`](https://github.com/txpipe/oura/tree/main/examples/sqlite) | `SqlDb` sink writing raw CBOR into SQLite | writes `./mydatabase.db` · `sql` feature |
80-
| [`postgresql`](https://github.com/txpipe/oura/tree/main/examples/postgresql) | `SqlDb` sink targeting PostgreSQL | docker compose · README · `sql` feature |
81-
| [`file_rotate`](https://github.com/txpipe/oura/tree/main/examples/file_rotate) | `FileRotate` sink writing rotated, compressed JSONL | writes `./output` |
82-
| [`aws_lambda`](https://github.com/txpipe/oura/tree/main/examples/aws_lambda) | `AwsLambda` sink invoking a function per event | AWS credentials · `aws` feature |
83-
| [`aws_s3`](https://github.com/txpipe/oura/tree/main/examples/aws_s3) | `AwsS3` sink writing objects | AWS credentials · `aws` feature |
84-
| [`aws_sqs`](https://github.com/txpipe/oura/tree/main/examples/aws_sqs) | `AwsSqs` sink enqueueing messages | AWS credentials · `aws` feature |
85-
| [`gcp_pubsub`](https://github.com/txpipe/oura/tree/main/examples/gcp_pubsub) | `GcpPubSub` sink publishing to a topic | GCP credentials · `gcp` feature |
86-
| [`gcp_cloudfunction`](https://github.com/txpipe/oura/tree/main/examples/gcp_cloudfunction) | `GcpCloudFunction` sink calling a function | GCP credentials · `gcp` feature |
87-
| [`assert`](https://github.com/txpipe/oura/tree/main/examples/assert) | `Assert` sink used for testing/validation | — |
57+
| Example | What it shows |
58+
| :------ | :------------ |
59+
| [`stdout`](https://github.com/txpipe/oura/tree/main/examples/stdout) | Legacy v1 events printed to standard output |
60+
| [`webhook_basics`](https://github.com/txpipe/oura/tree/main/examples/webhook_basics) | `WebHook` sink posting events over HTTP |
61+
| [`kafka`](https://github.com/txpipe/oura/tree/main/examples/kafka) | `Kafka` sink with `ByBlock` partitioning |
62+
| [`rabbitmq`](https://github.com/txpipe/oura/tree/main/examples/rabbitmq) | `Rabbitmq` sink |
63+
| [`zeromq`](https://github.com/txpipe/oura/tree/main/examples/zeromq) | `Zeromq` PUSH socket sink |
64+
| [`elasticsearch`](https://github.com/txpipe/oura/tree/main/examples/elasticsearch) | `ElasticSearch` sink |
65+
| [`redis`](https://github.com/txpipe/oura/tree/main/examples/redis) | `Redis` streams sink |
66+
| [`sqlite`](https://github.com/txpipe/oura/tree/main/examples/sqlite) | `SqlDb` sink writing raw CBOR into SQLite |
67+
| [`postgresql`](https://github.com/txpipe/oura/tree/main/examples/postgresql) | `SqlDb` sink targeting PostgreSQL |
68+
| [`file_rotate`](https://github.com/txpipe/oura/tree/main/examples/file_rotate) | `FileRotate` sink writing rotated, compressed JSONL |
69+
| [`aws_lambda`](https://github.com/txpipe/oura/tree/main/examples/aws_lambda) | `AwsLambda` sink invoking a function per event |
70+
| [`aws_s3`](https://github.com/txpipe/oura/tree/main/examples/aws_s3) | `AwsS3` sink writing objects |
71+
| [`aws_sqs`](https://github.com/txpipe/oura/tree/main/examples/aws_sqs) | `AwsSqs` sink enqueueing messages |
72+
| [`gcp_pubsub`](https://github.com/txpipe/oura/tree/main/examples/gcp_pubsub) | `GcpPubSub` sink publishing to a topic |
73+
| [`gcp_cloudfunction`](https://github.com/txpipe/oura/tree/main/examples/gcp_cloudfunction) | `GcpCloudFunction` sink calling a function |
74+
| [`assert`](https://github.com/txpipe/oura/tree/main/examples/assert) | `Assert` sink used for testing/validation |
8875

8976
## Cursors
9077

91-
| Example | What it shows | Setup |
92-
| :------ | :------------ | :---- |
93-
| [`file_cursor`](https://github.com/txpipe/oura/tree/main/examples/file_cursor) | Persisting pipeline progress to a `File` cursor | writes `my_cursor.json` |
94-
| [`redis_cursor`](https://github.com/txpipe/oura/tree/main/examples/redis_cursor) | Persisting pipeline progress to a `Redis` cursor | docker compose · README · `redis` feature |
78+
| Example | What it shows |
79+
| :------ | :------------ |
80+
| [`file_cursor`](https://github.com/txpipe/oura/tree/main/examples/file_cursor) | Persisting pipeline progress to a `File` cursor |
81+
| [`redis_cursor`](https://github.com/txpipe/oura/tree/main/examples/redis_cursor) | Persisting pipeline progress to a `Redis` cursor |
9582

9683
## Observability
9784

98-
| Example | What it shows | Setup |
99-
| :------ | :------------ | :---- |
100-
| [`metrics`](https://github.com/txpipe/oura/tree/main/examples/metrics) | Exposing a Prometheus metrics endpoint via the `[metrics]` block | — |
85+
| Example | What it shows |
86+
| :------ | :------------ |
87+
| [`metrics`](https://github.com/txpipe/oura/tree/main/examples/metrics) | Exposing a Prometheus metrics endpoint via the `[metrics]` block |
10188

10289
## As a library
10390

‎examples/assert/README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Assert sink
2+
3+
Run the chain through the `Assert` sink, which validates the event stream against internal
4+
invariants. Intended for testing and pipeline validation rather than production output.
5+
6+
## Pipeline
7+
8+
```mermaid
9+
flowchart LR
10+
src[N2N source] --> f1[LegacyV1] --> sink[Assert sink]
11+
```
12+
13+
- **Source** — `N2N`: mainnet relay, starting from the chain tip.
14+
- **Filters** — `LegacyV1`: maps records to the legacy v1 event model.
15+
- **Sink** — `Assert`: checks invariants over the event stream and reports violations.
16+
17+
## Run
18+
19+
```sh
20+
cd examples/assert
21+
oura daemon --config daemon.toml
22+
```

‎examples/aws_lambda/README.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# AWS Lambda sink
2+
3+
Decode transactions and invoke an AWS Lambda function once per event.
4+
5+
## Pipeline
6+
7+
```mermaid
8+
flowchart LR
9+
src[N2N source] --> f1[SplitBlock] --> f2[ParseCbor] --> sink[AwsLambda sink]
10+
```
11+
12+
- **Source** — `N2N`: mainnet relay, starting from the chain tip.
13+
- **Filters**
14+
- `SplitBlock`: breaks each block into individual transactions.
15+
- `ParseCbor`: decodes the raw transaction CBOR into structured records.
16+
- **Sink** — `AwsLambda`: invokes `function_name` in `region` with each event as the payload.
17+
18+
## Prerequisites
19+
20+
- Built with the `aws` feature.
21+
- AWS credentials available to the process (env vars, profile, or instance role) with
22+
permission to invoke the function.
23+
- Edit `region` and `function_name` in `daemon.toml` to match your function.
24+
25+
## Run
26+
27+
```sh
28+
cd examples/aws_lambda
29+
cargo run --features aws --bin oura -- daemon --config daemon.toml
30+
```
31+
32+
(or `oura daemon --config daemon.toml` with a binary built with the `aws` feature.)

‎examples/aws_s3/README.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# AWS S3 sink
2+
3+
Decode transactions and write each one as an object into an S3 bucket.
4+
5+
## Pipeline
6+
7+
```mermaid
8+
flowchart LR
9+
src[N2N source] --> f1[ParseCbor] --> sink[AwsS3 sink]
10+
```
11+
12+
- **Source** — `N2N`: mainnet relay, starting from the chain tip.
13+
- **Filters** — `ParseCbor`: decodes the raw transaction CBOR into structured records.
14+
- **Sink** — `AwsS3`: writes objects under `prefix` in `bucket` (`region`).
15+
16+
## Prerequisites
17+
18+
- Built with the `aws` feature.
19+
- AWS credentials available to the process (env vars, profile, or instance role) with
20+
permission to write to the bucket.
21+
- Edit `region`, `bucket`, and `prefix` in `daemon.toml` to match your bucket.
22+
23+
## Run
24+
25+
```sh
26+
cd examples/aws_s3
27+
cargo run --features aws --bin oura -- daemon --config daemon.toml
28+
```
29+
30+
(or `oura daemon --config daemon.toml` with a binary built with the `aws` feature.)

‎examples/aws_sqs/README.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# AWS SQS sink
2+
3+
Decode transactions and enqueue each one as a message on an Amazon SQS queue.
4+
5+
## Pipeline
6+
7+
```mermaid
8+
flowchart LR
9+
src[N2N source] --> f1[SplitBlock] --> f2[ParseCbor] --> sink[AwsSqs sink]
10+
```
11+
12+
- **Source** — `N2N`: mainnet relay, starting from the chain tip.
13+
- **Filters**
14+
- `SplitBlock`: breaks each block into individual transactions.
15+
- `ParseCbor`: decodes the raw transaction CBOR into structured records.
16+
- **Sink** — `AwsSqs`: sends messages to `queue_url` (`region`) with the configured
17+
`group_id` (FIFO queues).
18+
19+
## Prerequisites
20+
21+
- Built with the `aws` feature.
22+
- AWS credentials available to the process (env vars, profile, or instance role) with
23+
permission to send to the queue.
24+
- Edit `region`, `queue_url`, and `group_id` in `daemon.toml` to match your queue.
25+
26+
## Run
27+
28+
```sh
29+
cd examples/aws_sqs
30+
cargo run --features aws --bin oura -- daemon --config daemon.toml
31+
```
32+
33+
(or `oura daemon --config daemon.toml` with a binary built with the `aws` feature.)

‎examples/dolos_source/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# UTxO RPC (U5C) source
2+
3+
Read the chain from a [UTxO RPC](https://utxorpc.org) (U5C) endpoint — such as a local
4+
[Dolos](https://github.com/txpipe/dolos) node or a hosted [Demeter](https://demeter.run)
5+
endpoint — and print events to standard output.
6+
7+
## Pipeline
8+
9+
```mermaid
10+
flowchart LR
11+
src[U5C source] --> sink[Stdout sink]
12+
```
13+
14+
- **Source** — `U5C`: connects to the gRPC `url`, starting from the chain tip. The
15+
`metadata` table carries extra gRPC headers; leave it empty (`{}`) for a local
16+
unauthenticated Dolos, or pass an API key for a hosted endpoint
17+
(`metadata = { "dmtr-api-key" = "dmtr_..." }`).
18+
- **Sink** — `Stdout`: prints each event.
19+
20+
## Prerequisites
21+
22+
- A running U5C endpoint (a local Dolos on `localhost:50051`, or a hosted U5C URL).
23+
- Built with the `u5c` feature (it ships in the released binaries by default).
24+
25+
## Run
26+
27+
```sh
28+
cd examples/dolos_source
29+
oura daemon --config daemon.toml
30+
```
31+
32+
From source:
33+
34+
```sh
35+
cargo run --features u5c --bin oura -- daemon --config daemon.toml
36+
```

‎examples/elasticsearch/README.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Elasticsearch sink
2+
3+
Decode transactions and index them into Elasticsearch.
4+
5+
## Pipeline
6+
7+
```mermaid
8+
flowchart LR
9+
src[N2N source] --> f1[SplitBlock] --> f2[ParseCbor] --> sink[ElasticSearch sink]
10+
```
11+
12+
- **Source** — `N2N`: mainnet relay, starting from the chain tip.
13+
- **Filters**
14+
- `SplitBlock`: breaks each block into individual transactions.
15+
- `ParseCbor`: decodes the raw transaction CBOR into structured records.
16+
- **Sink** — `ElasticSearch`: indexes documents into `index` on the cluster at `url`
17+
(`idempotency = true` makes writes safe to replay).
18+
19+
## Prerequisites
20+
21+
- Built with the `elasticsearch` feature.
22+
- A running Elasticsearch cluster — a `docker-compose.yaml` is included.
23+
24+
```sh
25+
docker compose up -d
26+
```
27+
28+
## Run
29+
30+
```sh
31+
cd examples/elasticsearch
32+
cargo run --features elasticsearch --bin oura -- daemon --config daemon.toml
33+
```
34+
35+
(or `oura daemon --config daemon.toml` with a binary built with the `elasticsearch` feature.)

‎examples/file_cursor/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# File cursor
2+
3+
Persist pipeline progress to a local file so that, on restart, Oura resumes from the last
4+
processed point instead of the configured intersect.
5+
6+
## Pipeline
7+
8+
```mermaid
9+
flowchart LR
10+
src[N2N source] --> sink[Stdout sink]
11+
sink -. progress .-> cur[(File cursor)]
12+
```
13+
14+
- **Source** — `N2N`: mainnet relay, starting from the `Point` in `[intersect]` (used only
15+
on the first run, before a cursor exists).
16+
- **Cursor** — `File`: writes the latest position to `my_cursor.json` in the working
17+
directory; subsequent runs resume from it.
18+
- **Sink** — `Stdout`: prints each event.
19+
20+
## Run
21+
22+
```sh
23+
cd examples/file_cursor
24+
oura daemon --config daemon.toml
25+
```
26+
27+
Stop and restart the daemon to see it resume from `my_cursor.json` rather than the intersect
28+
point.

‎examples/file_rotate/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# File Rotate sink
2+
3+
Write legacy v1 events to rotated, compressed JSONL files on disk.
4+
5+
## Pipeline
6+
7+
```mermaid
8+
flowchart LR
9+
src[N2N source] --> f1[LegacyV1] --> sink[FileRotate sink]
10+
```
11+
12+
- **Source** — `N2N`: mainnet relay, starting from the `Point` in `[intersect]`.
13+
- **Filters** — `LegacyV1`: maps records to the legacy v1 event model
14+
(`include_transaction_details = true`).
15+
- **Sink** — `FileRotate`: writes JSONL to `./output/logs.jsonl`, rotating across up to 5
16+
compressed files of 5 MB each.
17+
18+
See the [FileRotate sink docs](../../docs/v2/sinks/file_rotate.mdx).
19+
20+
## Run
21+
22+
```sh
23+
cd examples/file_rotate
24+
oura daemon --config daemon.toml
25+
```
26+
27+
Output is written to `./output/` in the working directory.

0 commit comments

Comments
 (0)