Provider Selection

The SDK automatically selects storage providers based on endorsement status, availability, and metadata matching. Understanding provider selection helps you optimize uploads and troubleshoot issues.

Provider Types

Filecoin Onchain Cloud uses a two-tier provider system:

Endorsed Providers

Curated, high-quality storage providers:

Manual vetting: Reviewed by the FilOzone team
Higher reliability: Subject to stricter SLAs
Primary role: Always used for the first copy
Smaller set: More exclusive list

Approved Providers

Pass automated quality checks:

Automated verification: Quality metrics from on-chain data
Broader selection: Larger pool of providers
Secondary role: Used for additional copies
Includes endorsed: All endorsed providers are also approved

Think of it as: Endorsed ⊆ Approved. The endorsed set is a strict subset of approved providers.

Default Selection

When you call upload() without options, the SDK:

Select Primary (Endorsed)

Choose 1 provider from the endorsed list:

const endorsedProviders = await getEndorsedProviders()
const primary = randomSelect(endorsedProviders)

Random selection distributes load across endorsed providers.

Select Secondaries (Approved)

Choose N-1 providers from the approved list:

const approvedProviders = await getApprovedProviders()
const secondaries = randomSelect(
  approvedProviders,
  count - 1,
  { exclude: [primary.id] }
)

Secondaries are chosen from approved providers, excluding the primary.

Match Metadata (Optional)

If metadata is provided, prefer providers with matching data sets:

const contexts = await synapse.storage.createContexts({
  metadata: { category: 'videos' },
})
// Reuses existing data sets with matching metadata

Example: Auto-Selection

const result = await synapse.storage.upload(data)

// SDK automatically selected:
// - 1 endorsed provider (primary)
// - 1 approved provider (secondary)

for (const copy of result.copies) {
  console.log(`${copy.role}: Provider ${copy.providerId}`)
}
// Output:
// primary: Provider 123
// secondary: Provider 456

Manual Provider Selection

Override automatic selection with explicit choices:

By Provider IDs

Specify exact providers to use:

const result = await synapse.storage.upload(data, {
  providerIds: [123n, 456n],
})

// Uses providers 123 and 456 exactly
// First in list becomes primary
// No retries on failure (explicit = user's choice)

With explicit providerIds, the SDK disables automatic retries. If these providers fail, the upload fails.

By Data Set IDs

Reuse existing data sets (and their providers):

// Find your existing data sets
const dataSets = await synapse.storage.findDataSets()

for (const ds of dataSets) {
  console.log(`Dataset ${ds.dataSetId}: Provider ${ds.providerId}`)
}

// Use specific data sets
const result = await synapse.storage.upload(data, {
  dataSetIds: [dataSets[0].dataSetId, dataSets[1].dataSetId],
})

// Pieces added to these existing data sets
// More efficient - batches pieces into same dataset

Benefits:

Reduces on-chain transactions
Groups related data together
Predictable provider assignment

Exclude Providers

Avoid specific providers:

const result = await synapse.storage.upload(data, {
  count: 2,
  excludeProviderIds: [999n, 888n],
})

// Auto-selects providers, but never uses 999 or 888
// Useful for excluding problematic providers

Metadata Matching

The SDK can match and reuse data sets based on metadata:

How It Works

Provide Metadata

Specify metadata when creating contexts:

const contexts = await synapse.storage.createContexts({
  metadata: { category: 'documents', project: 'alpha' },
})

Match Existing Data Sets

The SDK queries your existing data sets and looks for exact metadata matches:

// Finds data sets where:
// - payer = your wallet
// - metadata.category = 'documents'
// - metadata.project = 'alpha'

Reuse or Create

Match found: Reuse the data set and provider
No match: Create a new data set with this metadata

Example: Metadata-Based Grouping

// Upload multiple files to the same providers
const metadata = { project: 'website', env: 'production' }

const file1 = await synapse.storage.upload(data1, { metadata })
const file2 = await synapse.storage.upload(data2, { metadata })
const file3 = await synapse.storage.upload(data3, { metadata })

// All three files go to the same providers (metadata match)
// Efficient - pieces grouped into same data sets

CDN Metadata

The withCDN option sets special metadata:

const result = await synapse.storage.upload(data, {
  withCDN: true,
})

// Equivalent to:
const result = await synapse.storage.upload(data, {
  metadata: { withCDN: '' },
})

// Enables FilBeam CDN for this data set

The withCDN key in metadata has an empty string value by convention. The contract only checks for key presence.

Provider Discovery

Query available providers programmatically:

Get All Providers

const info = await synapse.storage.getStorageInfo()

console.log('Available providers:', info.providers.length)

for (const provider of info.providers) {
  console.log('Provider', provider.id)
  console.log('  Address:', provider.serviceProvider)
  console.log('  PDP URL:', provider.pdp.serviceURL)
  console.log('  Indexed:', provider.pdp.indexed)
}

Get Provider Details

// By ID
const provider = await synapse.getProviderInfo(123n)

// By address
const provider = await synapse.getProviderInfo('0x...')

console.log('Provider info:')
console.log('  ID:', provider.id)
console.log('  Service Provider:', provider.serviceProvider)
console.log('  Payee:', provider.payee)
console.log('  PDP Service URL:', provider.pdp.serviceURL)
console.log('  Indexed:', provider.pdp.indexed)

Using synapse-core

For low-level access:

import * as WarmStorage from '@filoz/synapse-core/warm-storage'
import * as SPRegistry from '@filoz/synapse-core/sp-registry'
import { createPublicClient, http } from 'viem'
import { calibration } from '@filoz/synapse-core/chains'

const client = createPublicClient({
  chain: calibration,
  transport: http(),
})

// Get approved provider IDs
const approvedIds = await WarmStorage.getApprovedProviders(client, {
  offset: 0n,
  limit: 100n,
})

// Get endorsed provider IDs
const endorsedIds = await WarmStorage.getEndorsedProviders(client)

// Get provider details
const provider = await SPRegistry.getProvider(client, {
  providerId: 123n,
})

Selection Algorithms

Three Mutually Exclusive Modes

The createContexts() method has three modes (you can only use one): Mode 1: Explicit Data Sets

const contexts = await synapse.storage.createContexts({
  dataSetIds: [100n, 200n, 300n],
})
// Uses exactly these data sets and their providers

Mode 2: Explicit Providers

const contexts = await synapse.storage.createContexts({
  providerIds: [123n, 456n, 789n],
})
// Uses exactly these providers
// Finds or creates matching data sets

Mode 3: Smart Selection

const contexts = await synapse.storage.createContexts({
  count: 3,
  metadata: { category: 'videos' },
  excludeProviderIds: [999n],
})
// Auto-selects 3 providers:
// - 1 endorsed (primary)
// - 2 approved (secondaries)
// - Matches metadata if possible
// - Excludes provider 999

Mixing modes throws an error. For example, you cannot specify both dataSetIds and providerIds.

Smart Selection Logic

1. Fetch provider lists:
   - Endorsed providers from contract
   - Approved providers from contract

2. Fetch client's data sets (for metadata matching)

3. Select primary (endorsed):
   - Filter endorsed providers by metadata match (if applicable)
   - Exclude any providers in excludeProviderIds
   - Random select from remaining

4. Select secondaries (approved):
   - Filter approved providers
   - Exclude primary
   - Exclude any in excludeProviderIds
   - Filter by metadata match (if applicable)
   - Random select remaining slots

5. For each selected provider:
   - Find existing data set with matching metadata
   - If found: reuse that data set
   - If not: new data set will be created on commit

Provider Health

Currently, provider selection is random within the endorsed/approved sets. Future versions may include:

Health scoring: Prefer providers with better uptime
Geographic distribution: Spread copies across regions
Cost optimization: Choose cheaper providers for secondaries
Performance metrics: Prefer faster providers

Provider selection is being actively improved. Follow the Synapse SDK releases for updates.

Best Practices

Use metadata for organization

Group related data with consistent metadata:

const projectMetadata = { project: 'app', env: 'prod' }

await synapse.storage.upload(config, { metadata: projectMetadata })
await synapse.storage.upload(assets, { metadata: projectMetadata })

// Both go to the same providers and data sets

Let the SDK choose providers

Unless you have specific requirements, let the SDK auto-select:

// Good (auto-select with retries)
await synapse.storage.upload(data, { count: 2 })

// Only if needed (no retries, no fallback)
await synapse.storage.upload(data, { providerIds: [123n, 456n] })

Exclude problematic providers

If a provider consistently fails:

const KNOWN_BAD = [999n, 888n]

await synapse.storage.upload(data, {
  excludeProviderIds: KNOWN_BAD,
})

Reuse data sets for batches

When uploading multiple related files:

const dataSets = await synapse.storage.findDataSets()
const projectDataSet = dataSets.find(
  ds => ds.metadata.project === 'myapp'
)

if (projectDataSet) {
  // Add new files to existing data set
  await synapse.storage.upload(data, {
    dataSetIds: [projectDataSet.dataSetId],
  })
}

Troubleshooting

”No approved service providers available”

This error means no providers are available after filtering:

// Too many exclusions?
const approvedIds = await synapse.storage.getStorageInfo()
console.log('Total approved:', approvedIds.providers.length)

// Don't exclude all providers
await synapse.storage.upload(data, {
  excludeProviderIds: [1n, 2n, 3n, ...], // Too many!
})

Solution: Reduce exclusions or check network health.

Metadata not matching

If metadata matching isn’t working:

// Check exact metadata format
const dataSets = await synapse.storage.findDataSets()

for (const ds of dataSets) {
  console.log('Dataset', ds.dataSetId)
  console.log('Metadata:', JSON.stringify(ds.metadata, null, 2))
}

// Metadata must match exactly (key and value)
const metadata = { category: 'docs' } // Not 'documents'