Choose where your chain data lands

A Substreams package produces output. A sink is the integration that carries that output to a destination. The destinations named across this site are Postgres, ClickHouse, Parquet, a Pub/Sub topic, a webhook and plain JSON. The homepage states the same list under Sink anywhere.

The first division to make is not technical. It is who runs the process that does the writing. Postgres and ClickHouse can be reached by Hosted Sinks, a managed service on The Graph Market that runs your sink for you. The other four destinations are reached by a sink you deploy and keep alive yourself.

The second division is maintenance of the connector, and it is separate from the first. The sinks documentation keeps two tables, one for official support and one for community support, and states that some sinks are officially supported by StreamingFast while others are community-driven and support cannot be guaranteed. Parquet arrives through the files sink, which sits in the community table. That is worth knowing before a destination becomes a dependency.

Hosted Sinks is described in the documentation as a managed service on The Graph Market that runs your Substreams sink for you. Instead of provisioning servers, managing deployments or operating sink processes, you configure the sink in the portal and StreamingFast handles the infrastructure. It is one of two Hosted Services; the other is a Hosted Store. The documentation marks Hosted Sinks as currently in beta.

The boundary is stated plainly, and it is the part worth reading twice. A hosted sink continuously reads your package and writes the output into your Postgres or ClickHouse database. StreamingFast hosts the sink runner only. It does not create, host or manage the database.

What you still bring

• The database. One you already operate, or a managed one. The documentation provides walkthroughs for Supabase and Neon on Postgres, and ClickHouse Cloud on ClickHouse.
• Network reachability. The database must be reachable from StreamingFast's network, through a public hostname rather than localhost or a private container name.
• Credentials. The documentation asks for a dedicated database user whose write access is restricted to the target schema.
• A package that outputs for SQL. Database Changes Mode through a db_out module, or Relational Mappings Mode.
• The schema. Either already applied, or created by the package itself.

What you configure in the portal is a sink name, the package by URL or by its Substreams Registry ID, the connection details, an optional GraphQL layer, then the execution settings: start block, stop block, output module, and optional filters and parameters. Deployment moves through Deploying to Deployed, and a running pod reports its own execution state as Initiating, Catching up, Live or Failing. The detail page carries pod logs and a deployment event log, and the edit dialog changes the package, the execution settings or the connection details, after which the sink process restarts.

The optional GraphQL layer is a separate read-only deployment, powered by Hasura, that reads the same database the sink writes. It tracks the tables and views in the target schema, excludes the sink's own bookkeeping tables, generates no mutation fields, and can be added or removed without touching the pipeline. Both Postgres and ClickHouse sinks are supported.

Both engines are reached by the same sink, which is built into the substreams CLI as substreams sink postgres and substreams sink clickhouse, and both are offered by Hosted Sinks. The choice between them is a choice about the consumer, not about the connector.

Postgres suits a product that reads rows by key and keeps current state alongside the application's own tables. ClickHouse suits analytical queries written over wide ranges of history. Both accept the two documented mapping modes, so the mode decision is separable from the engine decision; Build a reliable blockchain-to-SQL pipeline works through that decision and the write path in detail.

Two documented differences bear on operations rather than on query style.

Plan capacity for those tables, and for the backup policy that now covers them. The reorganization documentation also notes that history tracking applies only in the reversible segment, so a historical backfill does not accumulate it.

The operating cost of either engine is the cost of running that engine. A hosted sink does not change it. What a hosted sink changes is whether anyone on your team is also responsible for the process writing into it.

Parquet arrives through substreams-sink-files, a tool that writes blockchain data into file formats: Parquet, line-based CSV, line-based JSONL, and arbitrary Protobuf turned into JSONL. Parquet is the default encoder when no --encoder flag is given. The sinks documentation lists this tool under community support.

The encoder derives a Parquet schema from the Protobuf message your module outputs. Table structures are inferred, or set explicitly with the (parquet.table_name) option; individual columns are controlled with (parquet.column) for type and compression, and a field can be left out with (parquet.ignored). The repository documents a tools parquet schema command that prints the derived schema without running the extraction, which is the cheapest way to find out whether your output models the tables you expect.

Output can be a local path or a cloud bucket, addressed as gs://, s3:// or az://. Files are written in bundles, and the number of blocks per bundle is set with --file-block-count.

What you operate

• The process. Nothing hosted writes Parquet for you.
• The cursor. It is saved to a file in the folder given by --state-store. The repository is explicit that this file must be persisted and must follow the deployment: if it is lost, the tool restarts from the beginning of the chain and redoes all previous processing.
• The reading side. The engine, catalogue or notebook that queries the files, and whatever registers new files as they appear.

One documented property decides most cases on its own. The repository states that the tool syncs up to the most recent final block of the chain, and that this means it is not real-time. For a warehouse load or a dataset handed to analysts, that is the correct behaviour, because no bundle on disk ever has to be unwound. For anything that reacts to the head of the chain, it is the wrong destination.

These two destinations deliver messages rather than rows. Both are documented as officially supported, and both are a process you run. They differ in how many consumers can read the same output and in what happens when delivery fails.

A Pub/Sub topic

substreams sink pubsub publishes the output of any module to a Google Cloud Pub/Sub topic. The topic is given as an ID with --project or as a full projects/<project>/topics/<topic> path, and credentials are application default credentials. Each block message carries the attribute type=batch and a body holding the manifest and the blocks, where each block's data is that module's output as JSON. Message ordering is enabled, and the ordering key is the output module's name, so a subscription created with message ordering enabled receives that module's messages in publish order.

Corrections are explicit. --pubsub-undo publishes a reorganization notification on the same topic, with attribute type=undo, naming the last valid block. Without that flag the cursor still moves back and the replacement blocks are published as usual, which means a subscriber that assumes append-only delivery will see the same block numbers twice and will have to reconcile them itself. A module whose output is sf.substreams.sink.pubsub.v1.Publish takes the other path: each of its messages is published on its own, and its reorganization message — attributes LastValidBlock, Step=Undo and the cursor, with no body — is published whether or not the flag is set.

Failure handling is also explicit. The cursor lives in --state-file. A payload that could not be published is kept in a .pending file beside it, the process exits with status 75 under the default --pubsub-on-failure=exit, and the next start publishes that payload before reconnecting. --pubsub-on-failure=skip drops a block after the last retry and continues, and an undo notification is never dropped.

A webhook

substreams sink webhook posts to an HTTP endpoint you supply. The payload is JSON carrying clock, manifest and data. Retries are configurable through --webhook-max-retries, --webhook-timeout and --webhook-max-retry-interval, with exponential backoff and jitter; -1 retries indefinitely. Network errors and server errors are retried. Client errors in the 4xx range are not, because the sink treats them as permanent.

Two behaviours decide whether this destination fits. The repository states that the sink continues processing even if webhook calls fail after all retries, and that the cursor is still saved for successful blocks, so block processing is never blocked by a failing endpoint. It also lists idempotent handlers that tolerate duplicate calls as a best practice. Taken together: your endpoint is the component that must notice it missed something, and your endpoint is the component that must not double-apply what it receives twice.

JSON is the destination with the widest range, because it covers three different situations that only look alike.

• Inspection. The CLI prints module output as JSON lines, which is how the chain pages on this site demonstrate a stream in one command. Nothing is stored and nothing is operated. This is the right first step on any new package.
• JSONL files. The ProtoJSON sink writes JSONL, and its current implementation lives in the files sink repository. The protojson:<expression> encoder extracts rows from a repeated field of the output message and writes one element per line, and the documented expression form today is .<repeated_field_name>[]. Operationally this is the files destination from the previous chapter, with a different encoder.
• A consumer you write. The Substreams:Stream service streams data directly into a JavaScript, Go, Rust or Python application through the SDKs.

The third case is the one to be deliberate about. Writing the consumer gives you every freedom and transfers every guarantee. The cursor, the resume position, the response to an undo signal, the idempotence of each write, the detection of a stalled stream and the restart procedure all become code your team maintains. The documented sinks already contain that work; an application reading the stream directly does not.

Choose it when the output's destination is your own process rather than a store, when the transformation that remains cannot be expressed in the package, or when no documented sink targets the system you need to reach. Choose a documented sink when the destination is one of the five above, because then the guarantees arrive with it.

Start from the query or the reaction that has to happen, then read backwards to the destination. Starting from the format produces a destination that works and a consumer that does not fit it.

Three answers to write down before deploying

• Who runs the consumer. A hosted sink, or a process with an owner, an alert and a restart procedure.
• What happens to a record the chain takes back. Reverted by the sink, delivered as an undo notification, avoided by writing only finalized blocks, or reconciled by hand. One of those four, named.
• Who deduplicates. The destination's keys, the receiving endpoint, or nobody. If the answer is nobody, that is the first thing to fix.

Two destinations can be correct at once. A hosted sink into ClickHouse for the queries a product runs, and Parquet into a bucket for the archive an analytics team loads, are the same package with two sinks. That is cheaper than it sounds, because the modelling work is in the package, not in the destination.

The destination is also the most reversible decision in the pipeline. The package keeps producing the same output, so changing where it lands is a new sink and a backfill, not a rewrite. The decision that is expensive to reverse is the one about the model, which is upstream of everything in this guide.

For a discussion with StreamingFast, bring the destination you have chosen, who will operate the consumer, your finality requirement, and whether the database is reachable from outside your network. Those four answers are what determine whether a hosted sink applies to your case.