Amazon S3

Overview

The Amazon S3 Object Storage module introduces an object storage client that integrates with Amazon S3, providing a clean, abstracted interface for storing and retrieving objects in the cloud. Amazon S3 (Simple Storage Service) is a highly scalable, durable, and secure object storage service for storing and retrieving any amount of data from anywhere.

Installation

Intent.AmazonS3.ObjectStorage

The IObjectStorage Interface

The module generates an IObjectStorage interface that provides consistent object storage operations:

public interface IObjectStorage
{
    // Upload operations
    Task<string> UploadAsync(
        string bucketName,
        string key,
        Stream content,
        CancellationToken cancellationToken = default);
    
    Task<string> UploadStringAsync(
        string bucketName,
        string key,
        string content,
        CancellationToken cancellationToken = default);
    
    Task<string> UploadBytesAsync(
        string bucketName,
        string key,
        byte[] content,
        CancellationToken cancellationToken = default);
    
    // Download operations
    Task<Stream> DownloadAsync(
        string bucketName,
        string key,
        CancellationToken cancellationToken = default);
    
    Task<string> DownloadAsStringAsync(
        string bucketName,
        string key,
        CancellationToken cancellationToken = default);
    
    Task<byte[]> DownloadAsBytesAsync(
        string bucketName,
        string key,
        CancellationToken cancellationToken = default);
    
    // List operation
    IAsyncEnumerable<string> ListAsync(
        string bucketName,
        string prefix = null,
        CancellationToken cancellationToken = default);
    
    // Delete operation
    Task DeleteAsync(
        string bucketName,
        string key,
        CancellationToken cancellationToken = default);
    
    // Existence check
    Task<bool> ExistsAsync(
        string bucketName,
        string key,
        CancellationToken cancellationToken = default);
}

Configuration

Using AWS Credentials

appsettings.json:

{
  "AWS": {
    "Region": "us-east-1",
    "AccessKey": "your-access-key",
    "SecretKey": "your-secret-key"
  },
  "S3": {
    "BucketName": "my-app-bucket"
  }
}

Using IAM Roles (Recommended)

For applications running on AWS (EC2, ECS, Lambda): appsettings.json:

{
  "AWS": {
    "Region": "us-east-1"
  },
  "S3": {
    "BucketName": "my-app-bucket"
  }
}

The AWS SDK automatically uses IAM role credentials.

Using AWS Profile (Local Development)

appsettings.Development.json:

{
  "AWS": {
    "Region": "us-east-1",
    "Profile": "development"
  },
  "S3": {
    "BucketName": "my-app-dev-bucket"
  }
}

Usage Examples

Document Storage Service

public class DocumentStorageService
{
    private readonly IObjectStorage _objectStorage;
    private readonly string _bucketName;

    public DocumentStorageService(
        IObjectStorage objectStorage,
        IConfiguration configuration)
    {
        _objectStorage = objectStorage;
        _bucketName = configuration["S3:BucketName"];
    }

    public async Task<string> UploadDocumentAsync(
        string documentId,
        Stream content,
        string contentType,
        CancellationToken cancellationToken)
    {
        var key = $"documents/{documentId}";
        
        return await _objectStorage.UploadAsync(
            _bucketName,
            key,
            content,
            cancellationToken);
    }

    public async Task<Stream> DownloadDocumentAsync(
        string documentId,
        CancellationToken cancellationToken)
    {
        var key = $"documents/{documentId}";
        
        return await _objectStorage.DownloadAsync(
            _bucketName,
            key,
            cancellationToken);
    }

    public async Task DeleteDocumentAsync(
        string documentId,
        CancellationToken cancellationToken)
    {
        var key = $"documents/{documentId}";
        
        await _objectStorage.DeleteAsync(
            _bucketName,
            key,
            cancellationToken);
    }

    public async Task<List<string>> ListDocumentsAsync(
        CancellationToken cancellationToken)
    {
        var documents = new List<string>();
        
        await foreach (var key in _objectStorage.ListAsync(
            _bucketName,
            "documents/",
            cancellationToken))
        {
            documents.Add(key);
        }
        
        return documents;
    }
}

Image Upload Service

public class ImageService
{
    private readonly IObjectStorage _objectStorage;
    private readonly IImageProcessor _imageProcessor;
    private readonly string _bucketName;

    public async Task<string> UploadUserAvatarAsync(
        string userId,
        byte[] imageData,
        CancellationToken cancellationToken)
    {
        // Process image (resize, compress)
        var processedImage = await _imageProcessor.ResizeAsync(
            imageData, 
            width: 200, 
            height: 200);
        
        // Upload to S3
        var key = $"avatars/{userId}/profile.jpg";
        return await _objectStorage.UploadBytesAsync(
            _bucketName,
            key,
            processedImage,
            cancellationToken);
    }

    public async Task<byte[]> GetUserAvatarAsync(
        string userId,
        CancellationToken cancellationToken)
    {
        var key = $"avatars/{userId}/profile.jpg";
        
        if (!await _objectStorage.ExistsAsync(_bucketName, key, cancellationToken))
        {
            // Return default avatar
            return await GetDefaultAvatarAsync();
        }
        
        return await _objectStorage.DownloadAsBytesAsync(
            _bucketName,
            key,
            cancellationToken);
    }
}

Backup Service

public class BackupService
{
    private readonly IObjectStorage _objectStorage;
    private readonly string _bucketName;

    public async Task CreateBackupAsync(
        string backupName,
        string jsonData,
        CancellationToken cancellationToken)
    {
        var timestamp = DateTime.UtcNow.ToString("yyyy-MM-dd-HHmmss");
        var key = $"backups/{backupName}/{timestamp}.json";
        
        await _objectStorage.UploadStringAsync(
            _bucketName,
            key,
            jsonData,
            cancellationToken);
    }

    public async Task<string> RestoreLatestBackupAsync(
        string backupName,
        CancellationToken cancellationToken)
    {
        var backups = new List<string>();
        
        await foreach (var key in _objectStorage.ListAsync(
            _bucketName,
            $"backups/{backupName}/",
            cancellationToken))
        {
            backups.Add(key);
        }
        
        // Get most recent (sorted by timestamp in key)
        var latestKey = backups.OrderByDescending(k => k).FirstOrDefault();
        
        if (latestKey == null)
        {
            throw new InvalidOperationException("No backups found");
        }
        
        return await _objectStorage.DownloadAsStringAsync(
            _bucketName,
            latestKey,
            cancellationToken);
    }
}

S3 Key Naming

Best Practices

Use hierarchical structure with forward slashes:

// Good
var key = "users/12345/documents/invoice.pdf";
var key = "images/products/2024/01/product-abc.jpg";

// Avoid
var key = "user_12345_invoice.pdf";  // Flat structure
var key = "users\\12345\\documents\\invoice.pdf";  // Wrong separator

Include timestamps for versioning:

var timestamp = DateTime.UtcNow.ToString("yyyyMMdd-HHmmss");
var key = $"exports/{reportName}/{timestamp}.csv";

Use consistent naming conventions:

public class S3KeyBuilder
{
    public static string UserAvatar(string userId) => 
        $"avatars/users/{userId}/profile.jpg";
    
    public static string Document(string category, string documentId) =>
        $"documents/{category}/{documentId}";
    
    public static string Backup(string service, DateTime timestamp) =>
        $"backups/{service}/{timestamp:yyyy-MM-dd}/{timestamp:HHmmss}.bak";
}

AWS SDK Integration

Direct S3 Client Usage

For advanced scenarios, access the S3 client directly:

public class AdvancedS3Service
{
    private readonly IAmazonS3 _s3Client;

    public AdvancedS3Service(IAmazonS3 s3Client)
    {
        _s3Client = s3Client;
    }

    public async Task<string> GeneratePresignedUrlAsync(
        string bucketName,
        string key,
        TimeSpan expiration)
    {
        var request = new GetPreSignedUrlRequest
        {
            BucketName = bucketName,
            Key = key,
            Expires = DateTime.UtcNow.Add(expiration)
        };
        
        return _s3Client.GetPreSignedURL(request);
    }

    public async Task SetObjectMetadataAsync(
        string bucketName,
        string key,
        Dictionary<string, string> metadata)
    {
        var copyRequest = new CopyObjectRequest
        {
            SourceBucket = bucketName,
            SourceKey = key,
            DestinationBucket = bucketName,
            DestinationKey = key,
            MetadataDirective = S3MetadataDirective.REPLACE
        };
        
        foreach (var kvp in metadata)
        {
            copyRequest.Metadata.Add(kvp.Key, kvp.Value);
        }
        
        await _s3Client.CopyObjectAsync(copyRequest);
    }
}

Advanced Features

Pre-Signed URLs

Generate temporary download/upload URLs:

public async Task<string> GenerateDownloadLinkAsync(
    string documentId,
    TimeSpan validFor)
{
    var request = new GetPreSignedUrlRequest
    {
        BucketName = _bucketName,
        Key = $"documents/{documentId}",
        Expires = DateTime.UtcNow.Add(validFor),
        Verb = HttpVerb.GET
    };
    
    return _s3Client.GetPreSignedURL(request);
}

// Usage - generate 1-hour download link
var downloadUrl = await GenerateDownloadLinkAsync(
    "DOC-12345", 
    TimeSpan.FromHours(1));

Multipart Upload

For large files (>100 MB):

public async Task UploadLargeFileAsync(
    string bucketName,
    string key,
    Stream fileStream,
    IProgress<long> progress = null)
{
    var transferUtility = new TransferUtility(_s3Client);
    
    var uploadRequest = new TransferUtilityUploadRequest
    {
        BucketName = bucketName,
        Key = key,
        InputStream = fileStream,
        PartSize = 5 * 1024 * 1024, // 5 MB parts
    };
    
    if (progress != null)
    {
        uploadRequest.UploadProgressEvent += (sender, args) =>
        {
            progress.Report(args.TransferredBytes);
        };
    }
    
    await transferUtility.UploadAsync(uploadRequest);
}

Object Metadata

public async Task UploadWithMetadataAsync(
    string bucketName,
    string key,
    Stream content,
    Dictionary<string, string> metadata)
{
    var request = new PutObjectRequest
    {
        BucketName = bucketName,
        Key = key,
        InputStream = content
    };
    
    foreach (var kvp in metadata)
    {
        request.Metadata.Add(kvp.Key, kvp.Value);
    }
    
    await _s3Client.PutObjectAsync(request);
}

// Usage
await UploadWithMetadataAsync(
    "my-bucket",
    "documents/report.pdf",
    fileStream,
    new Dictionary<string, string>
    {
        { "author", "John Doe" },
        { "department", "Finance" },
        { "year", "2024" }
    });

Server-Side Encryption

public async Task UploadEncryptedAsync(
    string bucketName,
    string key,
    Stream content)
{
    var request = new PutObjectRequest
    {
        BucketName = bucketName,
        Key = key,
        InputStream = content,
        ServerSideEncryptionMethod = ServerSideEncryptionMethod.AES256
    };
    
    await _s3Client.PutObjectAsync(request);
}

S3 Storage Classes

Optimize costs with storage classes:

Class	Use Case	Retrieval
Standard	Frequently accessed	Immediate
Intelligent-Tiering	Unknown/changing access	Automatic
Standard-IA	Infrequently accessed	Immediate
One Zone-IA	Recreatable infrequent data	Immediate
Glacier Instant	Archive with instant retrieval	Immediate
Glacier Flexible	Archive	Minutes-hours
Glacier Deep Archive	Long-term archive	Hours

public async Task ArchiveOldDocumentsAsync(
    string bucketName,
    DateTime olderThan)
{
    await foreach (var key in _objectStorage.ListAsync(bucketName, "documents/"))
    {
        var metadata = await _s3Client.GetObjectMetadataAsync(bucketName, key);
        
        if (metadata.LastModified < olderThan)
        {
            var request = new CopyObjectRequest
            {
                SourceBucket = bucketName,
                SourceKey = key,
                DestinationBucket = bucketName,
                DestinationKey = key,
                StorageClass = S3StorageClass.Glacier
            };
            
            await _s3Client.CopyObjectAsync(request);
        }
    }
}

Lifecycle Policies

Automate data management:

{
  "Rules": [
    {
      "Id": "Move to IA after 30 days",
      "Status": "Enabled",
      "Prefix": "documents/",
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": {
        "Days": 365
      }
    },
    {
      "Id": "Delete temp files",
      "Status": "Enabled",
      "Prefix": "temp/",
      "Expiration": {
        "Days": 7
      }
    }
  ]
}

Best Practices

Security

Use IAM roles instead of access keys
Enable bucket versioning for critical data
Encrypt data at rest and in transit
Implement least privilege access
Use pre-signed URLs for temporary access
Enable S3 Block Public Access

Performance

Use CloudFront CDN for frequently accessed content
Implement multipart uploads for large files
Use Transfer Acceleration for global uploads
Parallelize operations when possible
Consider S3 Select for querying data

Cost Optimization

Use appropriate storage classes
Implement lifecycle policies
Delete incomplete multipart uploads
Monitor and analyze access patterns
Use S3 Intelligent-Tiering for unpredictable workloads

Reliability

Enable versioning for critical buckets
Implement cross-region replication
Use S3 Object Lock for compliance
Monitor with CloudWatch
Set up event notifications

Monitoring

CloudWatch Metrics

public class MonitoredObjectStorage : IObjectStorage
{
    private readonly IObjectStorage _inner;
    private readonly IAmazonCloudWatch _cloudWatch;

    public async Task<string> UploadAsync(
        string bucketName,
        string key,
        Stream content,
        CancellationToken cancellationToken)
    {
        var stopwatch = Stopwatch.StartNew();
        
        try
        {
            var result = await _inner.UploadAsync(
                bucketName, key, content, cancellationToken);
            
            await _cloudWatch.PutMetricDataAsync(new PutMetricDataRequest
            {
                Namespace = "MyApp/S3",
                MetricData = new List<MetricDatum>
                {
                    new MetricDatum
                    {
                        MetricName = "UploadDuration",
                        Value = stopwatch.ElapsedMilliseconds,
                        Unit = StandardUnit.Milliseconds,
                        Timestamp = DateTime.UtcNow
                    }
                }
            });
            
            return result;
        }
        catch (Exception ex)
        {
            // Track failures
            await TrackFailureAsync("Upload", ex);
            throw;
        }
    }
}

Local Development

LocalStack

Emulate AWS services locally: Docker Compose:

version: '3.8'
services:
  localstack:
    image: localstack/localstack
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3
      - DEBUG=1
      - DATA_DIR=/tmp/localstack/data
    volumes:
      - ./localstack:/tmp/localstack

Configuration:

{
  "AWS": {
    "Region": "us-east-1",
    "ServiceURL": "http://localhost:4566"
  }
}

MinIO

S3-compatible object storage:

docker run -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio server /data --console-address ":9001"

Resources

Amazon S3 Documentation

Official AWS documentation

Best Practices

Design and optimization guide

AWS SDK for .NET

.NET SDK reference

S3 Pricing

Cost calculator and pricing

Azure Functions

Azure Services

AWS Services

Message Brokers

Overview

Installation

The IObjectStorage Interface

Configuration

Using AWS Credentials

Using IAM Roles (Recommended)

Using AWS Profile (Local Development)

Usage Examples

Document Storage Service

Image Upload Service

Backup Service

S3 Key Naming

Best Practices

AWS SDK Integration

Direct S3 Client Usage

Advanced Features

Pre-Signed URLs

Multipart Upload

Object Metadata

Server-Side Encryption

S3 Storage Classes

Lifecycle Policies

Best Practices

Monitoring

CloudWatch Metrics

Local Development

LocalStack

MinIO

Resources

Amazon S3 Documentation

Best Practices

AWS SDK for .NET

S3 Pricing

Build docs developers (and LLMs) love

Azure Functions

Azure Services

AWS Services

Message Brokers

​Overview

​Installation

​The IObjectStorage Interface

​Configuration

​Using AWS Credentials

​Using IAM Roles (Recommended)

​Using AWS Profile (Local Development)

​Usage Examples

​Document Storage Service

​Image Upload Service

​Backup Service

​S3 Key Naming

​Best Practices

​AWS SDK Integration

​Direct S3 Client Usage

​Advanced Features

​Pre-Signed URLs

​Multipart Upload

​Object Metadata

​Server-Side Encryption

​S3 Storage Classes

​Lifecycle Policies

​Best Practices

​Monitoring

​CloudWatch Metrics

​Local Development

​LocalStack

​MinIO

​Resources

Amazon S3 Documentation

Best Practices

AWS SDK for .NET

S3 Pricing

Build docs developers (and LLMs) love

Overview

Installation

The IObjectStorage Interface

Configuration

Using AWS Credentials

Using IAM Roles (Recommended)

Using AWS Profile (Local Development)

Usage Examples

Document Storage Service

Image Upload Service

Backup Service

S3 Key Naming

Best Practices

AWS SDK Integration

Direct S3 Client Usage

Advanced Features

Pre-Signed URLs

Multipart Upload

Object Metadata

Server-Side Encryption

S3 Storage Classes

Lifecycle Policies

Best Practices

Monitoring

CloudWatch Metrics

Local Development

LocalStack

MinIO

Resources