Storage Operations

Looking for uploads?

This page covers data set management, retrieval, and lifecycle operations. For uploading data (from simple one-liner to manual store/pull/commit), see the Upload Pipeline.

Key Concepts

Data Set: A logical container of pieces stored with one provider. When a data set is created, a payment rail is established with that provider. All pieces in the data set share this single payment rail and are verified together via PDP proofs.

PieceCID: Content-addressed identifier for your data (format: bafkzcib...). Automatically calculated during upload and used to retrieve data from any provider.

Metadata: Optional key-value pairs for organization:

• Data Set Metadata: Max 10 keys (e.g., project, environment)
• Piece Metadata: Max 3 keys per piece (e.g., filename, contentType)

Copies and Durability: By default, upload() stores your data with 2 independent providers. Each provider maintains its own data set with separate PDP proofs and payment rails. If one provider goes down, your data is still available from the other.

Storage Manager: The main entry point for storage operations (synapse.storage). Handles provider selection, multi-copy orchestration, data set management, and provider-agnostic downloads.

Context Creation

Storage contexts represent a connection to a specific provider and data set. The SDK creates and manages contexts internally during upload(), but you can create them explicitly for data set management, retrieval, or split operations.

Single Context

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })

// All options are optional
await synapse.storage.createContext({
  providerId: 1n,           // specific provider (optional)
  dataSetId: 42n,           // specific data set (optional)
  metadata: { source: "my-app" },        // data set metadata for matching/creation
  withCDN: true,            // enable fast-retrieval (paid, optional)
  excludeProviderIds: [3n], // skip specific providers (optional)
})

Multiple Contexts

For multi-copy replication, use createContexts():

const contexts = await synapse.storage.createContexts({
  copies: 3,                   // number of contexts (default: 2)
  providerIds: [1n, 2n, 3n],   // specific providers (mutually exclusive with dataSetIds)
  dataSetIds: [10n, 20n, 30n], // specific data sets (mutually exclusive with providerIds)
})
const [primary, secondary] = contexts

View creation options for createContext()

View creation options for createContexts()

Data Set Matching

Metadata Matching for Cost Efficiency

The SDK reuses existing data sets when metadata matches exactly, avoiding duplicate payment rails. To maximize reuse:

• Use consistent metadata keys and values across uploads
• Avoid changing metadata unnecessarily
• Group related content with the same metadata

Example: If you create a data set with {Application: "MyApp", Version: "1.0"}, all subsequent uploads with the same metadata will reuse that data set and its payment rail.

The SDK intelligently manages data sets to minimize on-chain transactions. The selection behavior depends on the parameters you provide:

Selection Scenarios:

1. Explicit data set ID: If you specify dataSetId, that exact data set is used (must exist and be accessible)
2. Specific provider: If you specify providerId, the SDK searches for matching data sets only within that provider’s existing data sets
3. Automatic selection: Without specific parameters, the SDK searches across all your data sets with any approved provider

Exact Metadata Matching: In scenarios 2 and 3, the SDK will reuse an existing data set only if it has exactly the same metadata keys and values as requested. This ensures data sets remain organized according to your specific requirements.

Selection Priority: When multiple data sets match your criteria:

• Data sets with existing pieces are preferred over empty ones
• Within each group (with pieces vs. empty), the oldest data set (lowest ID) is selected

Provider Selection (when no matching data sets exist):

• If you specify a provider (via providerId), that provider is used
• Otherwise, the SDK selects from endorsed providers for the primary copy and any approved provider for secondaries
• Before finalizing selection, the SDK verifies the provider is reachable via a ping test
• If a provider fails the ping test, the SDK tries the next candidate
• A new data set will be created automatically during the first commit

Retrieval

Download from any provider that has the piece. The SDK resolves the provider automatically:

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: 'my-app' })

// Download using PieceCID from a previous upload
const pieceCid = "bafkzcib..." // from upload result
const bytes = await synapse.storage.download({ pieceCid })
const text = new TextDecoder().decode(bytes)
console.log("Downloaded:", text)

CDN-Accelerated Downloads

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

// Enable CDN globally
const synapse = Synapse.create({
  account: privateKeyToAccount("0x..."),
  source: 'my-app',
  withCDN: true,
})

const bytes = await synapse.storage.download({ pieceCid: "bafkzcib..." })

// Or per-download:
const bytes2 = await synapse.storage.download({
  pieceCid: "bafkzcib...",
  withCDN: true,
})

Context-Specific Downloads

When using a StorageContext, downloads are restricted to that specific provider:

import { Synapse, type PieceCID } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
const pieceCid = null as unknown as PieceCID;
// Downloads from the provider associated with this context
const data = await ctx.download({ pieceCid })

CDN Option Inheritance

The withCDN option follows a clear inheritance hierarchy:

1. Synapse level: Default setting for all operations
2. StorageContext level: Can override Synapse’s default
3. Method level: Can override instance settings

await synapse.storage.download({ pieceCid })                // Uses Synapse's withCDN: true
await ctx.download({ pieceCid })                            // Uses context's withCDN: false
await synapse.storage.download({ pieceCid, withCDN: false }) // Method override: CDN disabled

When withCDN: true is set, it adds { withCDN: '' } to the data set’s metadata, ensuring CDN-enabled and non-CDN data sets remain separate.

Data Set Management

When You Need This

These APIs are useful when you want to inspect existing data sets, query stored pieces, or retrieve metadata. For basic upload/download, you don’t need these.

Getting All Data Sets

Retrieve all data sets owned by your account to inspect piece counts, CDN status, and metadata:

import { Synapse } from "@filoz/synapse-sdk";
import { privateKeyToAccount } from 'viem/accounts'

const synapse = Synapse.create({ account: privateKeyToAccount('0x...'), source: 'my-app' });
const dataSets = await synapse.storage.findDataSets();

for (const ds of dataSets) {
  console.log(`Dataset ${ds.pdpVerifierDataSetId}:`, {
    live: ds.isLive,
    cdn: ds.withCDN,
    pieces: ds.activePieceCount,
    metadata: ds.metadata
  });
}

Getting Data Set Pieces

List all pieces stored in a specific data set by iterating through a context:

const context = await synapse.storage.createContext({ dataSetId });

const pieces = [];
for await (const piece of context.getPieces()) {
  pieces.push(piece);
}
console.log(`Found ${pieces.length} pieces`);

Getting Piece Metadata

Access custom metadata attached to individual pieces:

import { WarmStorageService } from "@filoz/synapse-sdk/warm-storage";
import { privateKeyToAccount } from 'viem/accounts'

const warmStorage =  WarmStorageService.create({ account: privateKeyToAccount('0x...') });

const metadata = await warmStorage.getPieceMetadata({ dataSetId, pieceId: piece.pieceId });
console.log("Piece metadata:", metadata);

Getting Piece Size

Extract the size directly from a PieceCID using Synapse Core:

import * as Piece from "@filoz/synapse-core/piece";

const pieceCid = "bafkzcib...";
const size = Piece.from(pieceCid).size;
console.log(`Piece size: ${size} bytes`);

Storage Information

Query service-wide pricing, available providers, and network parameters:

const info = await synapse.storage.getStorageInfo();
console.log("Price/TiB/month:", info.pricing.noCDN.perTiBPerMonth);
console.log("Providers:", info.providers.length);

const providerInfo = await synapse.getProviderInfo("0x...");
console.log("PDP URL:", providerInfo.pdp.serviceURL);

Lifecycle Management

Terminating the Storage Service

Irreversible Operation

Termination cannot be undone. Once initiated:

• The termination transaction is irreversible
• When the service ends, the provider may delete all data
• Payment rails associated with the data set will be terminated
• You cannot cancel the termination

Only terminate when you’re certain you no longer need the data.

To end the storage service for a data set and discontinue its payments, call context.terminate(), or synapse.storage.terminateService({ dataSetId }) when the data set ID is known (a context is not necessary).

There are two ways to terminate, chosen with the skipProvider option. The provider-mediated path is the recommended default: you sign an authorization, the provider relays it on chain, and if your account can settle in full the service and payment terminate immediately.

The direct on-chain path is an independent fallback. It is useful if the provider-mediated request is unavailable, account doesn’t have available funds to settle in full or you need to submit the transaction yourself. It does not require provider cooperation, but it uses the standard lockup wind-down: the service and payment continues until endEpoch, typically about 30 days, while the provider settles and winds down the data set before cleanup.

	Recommended default (via provider)	Fallback (skipProvider: true)
Who submits the transaction	The provider relays your signed authorization	You
Effect	Can terminate immediately and allow cleanup to begin	Service runs to the end of the lockup period, typically ~30 days
Cost to you	Small termination fee (USDFC, drawn from your account)	Transaction gas (FIL)
Requires	Provider online, your account able to settle in full	A funded wallet
Use when	Preferred path for ordinary termination	Provider-mediated termination cannot be completed, or you need independent submission

Provider-relayed termination never silently degrades: it either takes effect immediately or fails. The most common failure is an account that cannot cover full settlement; deposit funds or use skipProvider: true in that case. The result’s endEpoch reports the actual epoch at which the service stops; on the on-chain path this can be sooner than 30 days if your account stopped settling some time ago.

In both cases termination ends the service and its payments. The data set’s remaining on-chain state is cleaned up afterwards by the provider; that cleanup is not part of this call.

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
// Via context: relayed through this context's provider, immediate
const result = await ctx.terminate()
console.log("Service terminated, ends at epoch", result.endEpoch)

// Or directly by data set ID
const result2 = await synapse.storage.terminateService({ dataSetId: 42n })

// Or submit independently on chain (winds down over the lockup period)
const result3 = await synapse.storage.terminateService({
  dataSetId: 42n,
  skipProvider: true,
  onSubmitted: (txHash) => console.log("Submitted:", txHash),
})
console.log("Service ends at epoch", result3.endEpoch)

Deleting a Piece

To delete an individual piece from the data set, call context.deletePiece(). This method submits an on-chain transaction to initiate the deletion process.

Important: Piece deletion is irreversible and cannot be canceled once initiated.

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
// List all pieces in the data set
const pieces = []
for await (const piece of ctx.getPieces()) {
  pieces.push(piece)
}

if (pieces.length > 0) {
  await ctx.deletePiece({ piece: pieces[0].pieceId })
  console.log(
    `Piece ${pieces[0].pieceCid} (ID: ${pieces[0].pieceId}) deleted successfully`
  )
}

// Delete by PieceCID
await ctx.deletePiece({ piece: "bafkzcib..." })

Next Steps

• Upload Pipeline - Upload data with multi-copy durability, from simple one-liner to manual store/pull/commit.

• Storage Costs - Calculate your monthly costs and understand funding requirements.

• Payment Management - Manage deposits, approvals, and payment rails.