Storage Configuration

BioAgents supports S3-compatible storage for file uploads, research artifacts, and generated papers. Storage is optional but recommended for production deployments.

Overview

When storage is configured, BioAgents uses presigned URLs for direct client-to-S3 uploads, eliminating the need to proxy files through your server.

If storage is not configured, file upload features will be disabled.

Supported Providers

Amazon S3

AWS S3 buckets

DigitalOcean Spaces

S3-compatible object storage

MinIO

Self-hosted S3-compatible storage

Cloudflare R2

S3-compatible with zero egress fees

Quick Start

Choose Provider

Set the storage provider to s3:

STORAGE_PROVIDER=s3

Configure Credentials

Set your S3 credentials:

AWS_ACCESS_KEY_ID=your_access_key_id_here
AWS_SECRET_ACCESS_KEY=your_secret_access_key_here
AWS_REGION=us-east-1
S3_BUCKET=your-bucket-name-here

Optional: Custom Endpoint

For S3-compatible services (DigitalOcean Spaces, MinIO, Cloudflare R2):

S3_ENDPOINT=https://nyc3.digitaloceanspaces.com

Environment Variables

STORAGE_PROVIDER

string

Storage provider to use.Options:

s3 - Use S3-compatible storage
Empty or not set - Storage disabled

AWS Credentials

AWS_ACCESS_KEY_ID

string

required

S3 access key ID.Alternative: S3_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

string

required

S3 secret access key.Alternative: S3_SECRET_ACCESS_KEY

AWS_REGION

string

default:"us-east-1"

S3 region.Alternative: S3_REGION

S3_BUCKET

string

required

S3 bucket name.

S3_ENDPOINT

string

Custom S3 endpoint for S3-compatible services.Examples:

DigitalOcean Spaces: https://nyc3.digitaloceanspaces.com
MinIO: http://localhost:9000
Cloudflare R2: https://your-account.r2.cloudflarestorage.com

You can use either AWS_* or S3_* variants for credentials. Both are equivalent.

Provider Setup Guides

Amazon S3

Create S3 Bucket

Open AWS S3 Console
Click “Create bucket”
Enter bucket name and select region
Configure CORS (see below)

Create IAM User

Open IAM Console
Create new user with programmatic access
Attach policy with S3 permissions (see below)
Save access key ID and secret access key

Configure Environment

STORAGE_PROVIDER=s3
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS_REGION=us-east-1
S3_BUCKET=my-bioagents-bucket

DigitalOcean Spaces

Create Space

Open DigitalOcean Spaces
Click “Create Space”
Select region and name
Configure CORS (see below)

Generate API Keys

Go to API > Spaces Keys
Click “Generate New Key”
Save access key and secret key

Configure Environment

STORAGE_PROVIDER=s3
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_REGION=nyc3
S3_BUCKET=my-bioagents-space
S3_ENDPOINT=https://nyc3.digitaloceanspaces.com

MinIO (Self-Hosted)

Install MinIO

docker run -d \
  -p 9000:9000 \
  -p 9001:9001 \
  --name minio \
  -e "MINIO_ROOT_USER=minioadmin" \
  -e "MINIO_ROOT_PASSWORD=minioadmin" \
  minio/minio server /data --console-address ":9001"

Create Bucket

Open MinIO Console at http://localhost:9001
Login with minioadmin / minioadmin
Create new bucket
Configure CORS in bucket settings

Configure Environment

STORAGE_PROVIDER=s3
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin
AWS_REGION=us-east-1
S3_BUCKET=bioagents
S3_ENDPOINT=http://localhost:9000

Cloudflare R2

Create R2 Bucket

Open Cloudflare Dashboard
Go to R2 Object Storage
Create new bucket
Configure CORS (see below)

Generate API Token

Go to R2 > Manage R2 API Tokens
Create API token with R2 permissions
Save access key ID and secret access key

Configure Environment

STORAGE_PROVIDER=s3
AWS_ACCESS_KEY_ID=your_r2_access_key_id
AWS_SECRET_ACCESS_KEY=your_r2_secret_access_key
AWS_REGION=auto
S3_BUCKET=my-bioagents-bucket
S3_ENDPOINT=https://your-account-id.r2.cloudflarestorage.com

CORS Configuration

For direct client-to-S3 uploads, you must configure CORS on your S3 bucket.

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
    "AllowedOrigins": [
      "http://localhost:3000",
      "http://localhost:5173",
      "https://your-production-domain.com"
    ],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]

Replace http://localhost:* with your actual frontend URLs in production.

IAM Policy (AWS S3)

Minimum required permissions for S3:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:HeadObject"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name"
    }
  ]
}

File Organization

BioAgents organizes files in S3 with the following structure:

user/{userId}/
  conversation/{conversationId}/
    uploads/
      {filename}  # User-uploaded files
    artifacts/
      {artifactId}/  # Research artifacts (plots, tables)
    papers/
      {paperId}.pdf  # Generated papers
      {paperId}.tex  # LaTeX source

Implementation Details

Storage Provider Interface

The StorageProvider abstract class defines the storage interface:

src/storage/types.ts

export abstract class StorageProvider {
  abstract upload(path: string, buffer: Buffer, mimeType: string): Promise<string>;
  abstract download(path: string): Promise<Buffer>;
  abstract downloadRange(path: string, start: number, end: number): Promise<Buffer>;
  abstract delete(path: string): Promise<void>;
  abstract exists(path: string): Promise<boolean>;
  abstract getPresignedUrl(path: string, expiresIn?: number, filename?: string): Promise<string>;
  abstract getPresignedUploadUrl(path: string, contentType: string, expiresIn?: number, contentLength?: number): Promise<string>;
}

S3 Provider Implementation

The S3 provider uses AWS SDK v3:

src/storage/providers/s3.ts

export class S3StorageProvider extends StorageProvider {
  private client: S3Client;
  private bucket: string;

  constructor(config: {
    accessKeyId: string;
    secretAccessKey: string;
    region: string;
    bucket: string;
    endpoint?: string;
  }) {
    super();
    this.bucket = config.bucket;

    // For S3-compatible services, disable checksum features
    const isS3Compatible = !!config.endpoint;

    this.client = new S3Client({
      region: config.region,
      credentials: {
        accessKeyId: config.accessKeyId,
        secretAccessKey: config.secretAccessKey,
      },
      ...(config.endpoint && { endpoint: config.endpoint }),
      // Disable SDK checksum features for S3-compatible services
      ...(isS3Compatible && {
        requestChecksumCalculation: "WHEN_REQUIRED",
        responseChecksumValidation: "WHEN_REQUIRED",
      }),
    });
  }
}

Presigned URL Generation

For secure direct uploads:

src/storage/providers/s3.ts

async getPresignedUploadUrl(
  path: string,
  contentType: string,
  expiresIn: number = 3600,
  contentLength?: number,
): Promise<string> {
  const command = new PutObjectCommand({
    Bucket: this.bucket,
    Key: path,
    ContentType: contentType,
    // ContentLength prevents abuse: user cannot upload 5GB using a URL signed for 50MB
    ...(contentLength && { ContentLength: contentLength }),
  });

  const url = await getSignedUrl(this.client, command, { expiresIn });
  return url;
}

Storage Configuration Loading

src/storage/config.ts

export const STORAGE_CONFIG = {
  provider: process.env.STORAGE_PROVIDER?.toLowerCase(),
  s3: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID || process.env.S3_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY || process.env.S3_SECRET_ACCESS_KEY,
    region: process.env.AWS_REGION || process.env.S3_REGION || "us-east-1",
    bucket: process.env.S3_BUCKET,
    endpoint: process.env.S3_ENDPOINT,
  },
};

Upload Flow

BioAgents uses presigned URLs for efficient file uploads:

File Size Limits

Presigned URLs enforce file size limits to prevent abuse:

const presignedUrl = await storage.getPresignedUploadUrl(
  path,
  contentType,
  3600,  // 1 hour expiration
  fileSize  // S3 will reject uploads that don't match this size
);

S3 enforces the ContentLength parameter. Uploads with different sizes will be rejected.

Troubleshooting

CORS Errors

Symptom: No 'Access-Control-Allow-Origin' header errorsSolutions:

Configure CORS on your S3 bucket (see above)
Verify allowed origins match your frontend URL
Check browser console for specific CORS error

Access Denied

Symptom: AccessDenied or 403 Forbidden errorsSolutions:

Verify IAM policy grants required permissions
Check bucket policy doesn’t deny access
Ensure credentials are correct and not expired

Invalid Credentials

Symptom: InvalidAccessKeyId or SignatureDoesNotMatch errorsSolutions:

Double-check AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Ensure no extra spaces or newlines in credentials
Verify credentials are for the correct account

Bucket Not Found

Symptom: NoSuchBucket errorSolutions:

Verify bucket name is correct (case-sensitive)
Check bucket region matches AWS_REGION
For S3-compatible services, verify S3_ENDPOINT is correct

Upload Size Mismatch

Symptom: Upload fails with size errorSolutions:

Ensure file size passed to presigned URL matches actual upload
Don’t modify file after getting presigned URL
Check network didn’t truncate upload

Security Best Practices

Use Short-Lived Presigned URLs

Default expiration is 1 hour. Reduce for sensitive files:

const url = await storage.getPresignedUploadUrl(
  path,
  contentType,
  300  // 5 minutes
);

Enforce File Size Limits

Always pass contentLength to prevent abuse:

const url = await storage.getPresignedUploadUrl(
  path,
  contentType,
  3600,
  maxFileSize
);

Validate Content Types

Check file MIME types before generating presigned URLs:

const allowedTypes = ['application/pdf', 'text/csv'];
if (!allowedTypes.includes(contentType)) {
  throw new Error('Invalid file type');
}

Use IAM Least Privilege

Grant only required S3 permissions (see IAM policy above).

Next Steps

Environment Variables

View all configuration options

Authentication

Configure JWT or payment-based auth

Get Started

Core Concepts

Agents

Configuration

Features

Deployment

Advanced

Storage Configuration

Overview

Supported Providers

Amazon S3

DigitalOcean Spaces

MinIO

Cloudflare R2

Quick Start

Environment Variables

AWS Credentials

Provider Setup Guides

CORS Configuration

IAM Policy (AWS S3)

File Organization

Implementation Details

Storage Provider Interface

S3 Provider Implementation

Presigned URL Generation

Storage Configuration Loading

Upload Flow

File Size Limits

Troubleshooting

Security Best Practices

Next Steps

Environment Variables

Authentication

Build docs developers (and LLMs) love

Get Started

Core Concepts

Agents

Configuration

Features

Deployment

Advanced

​Overview

​Supported Providers

Amazon S3

DigitalOcean Spaces

MinIO

Cloudflare R2

​Quick Start

​Environment Variables

​AWS Credentials

​Provider Setup Guides

​CORS Configuration

​IAM Policy (AWS S3)

​File Organization

​Implementation Details

​Storage Provider Interface

​S3 Provider Implementation

​Presigned URL Generation

​Storage Configuration Loading

​Upload Flow

​File Size Limits

​Troubleshooting

​Security Best Practices

​Next Steps

Environment Variables

Authentication

Build docs developers (and LLMs) love

Overview

Supported Providers

Quick Start

Environment Variables

AWS Credentials

Provider Setup Guides

CORS Configuration

IAM Policy (AWS S3)

File Organization

Implementation Details

Storage Provider Interface

S3 Provider Implementation

Presigned URL Generation

Storage Configuration Loading

Upload Flow

File Size Limits

Troubleshooting

Security Best Practices

Next Steps