System Architecture

​

Overview

​

Monorepo Structure

camera-workflow/
├── cmd/                    # CLI commands
├── internal/               # Core Go packages
├── apps/raycast/          # Raycast extension UI
├── scripts/               # Build and utility scripts
├── main.go                # Application entry point
└── Makefile               # Build system

​

Core Architecture

​

Entry Point Flow

1. main.go → Minimal entry point that delegates to cmd package
2. cmd/root.go → CLI interface using Cobra, flag binding, and pre-run validation
3. internal/converter/ → Main conversion orchestrator

package main

import "github.com/Azilone/Camera-Workflow/cmd"

func main() {
    cmd.Execute()  // Delegates to Cobra CLI
}

​

Component Overview

​

Internal Package Details

​

Configuration (internal/config/config.go)

type Config struct {
    // Directories
    SourceDir string
    DestDir   string

    // Processing
    MaxJobs        int
    DryRun         bool
    CopyOnly       bool
    VerifyChecksum bool
    JSONMode       bool

    // Image settings
    PhotoFormat      string  // "avif" or "webp"
    PhotoQualityAVIF int     // 1-100
    PhotoQualityWebP int     // 1-100

    // Video settings
    VideoCodec        string  // "h265", "h264", "av1"
    VideoCRF          int     // Constant Rate Factor
    VideoAcceleration bool    // Hardware acceleration

    // Adaptive worker management
    AdaptiveWorkers AdaptiveWorkerConfig

    // Security
    ConversionTimeoutPhoto time.Duration
    ConversionTimeoutVideo time.Duration
    MinOutputSizeRatio     float64
}

1. Command line flags
2. Environment variables
3. YAML config file
4. Default values

​

Converter (internal/converter/converter.go)

• File discovery and categorization
• Worker pool management (semaphore-based)
• Progress tracking and statistics
• Safety tests before batch processing
• Recovery and cleanup operations

┌─────────────────────────────────────────────────┐
│ 1. Initialization                               │
│    - Validate directories                       │
│    - Check dependencies (FFmpeg, ImageMagick)   │
│    - Initialize logger                          │
└─────────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────────┐
│ 2. Recovery Phase                               │
│    - Cleanup abandoned .tmp files               │
│    - Remove stale .processing markers           │
│    - Verify existing converted files            │
│    - Re-queue corrupted files                   │
└─────────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────────┐
│ 3. Safety Test                                  │
│    - Test conversion on sample file             │
│    - Verify output integrity                    │
│    - Skip in dry-run mode                       │
└─────────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────────┐
│ 4. File Discovery                               │
│    - Walk source directory tree                 │
│    - Skip system files and directories          │
│    - Categorize by format (photo/video)         │
└─────────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────────┐
│ 5. Parallel Conversion                          │
│    - Image conversions (N workers)              │
│    - Video conversions (2 workers or adaptive)  │
│    - Real-time progress tracking                │
└─────────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────────┐
│ 6. Final Report                                 │
│    - Statistics and timing                      │
│    - Space savings                              │
│    - S3 cost estimations                        │
└─────────────────────────────────────────────────┘

​

Image Conversion (internal/converter/image.go)

• Supports AVIF and WebP output formats
• Quality settings per format (AVIF: 80, WebP: 85)
• Metadata preservation
• Output file validation

​

Video Conversion (internal/converter/video.go)

• Codec support: H.265, H.264, AV1
• Hardware acceleration detection (macOS VideoToolbox, Linux VAAPI/NVENC)
• Progress tracking with time-based progress bars
• AWS S3 cost estimation
• Adaptive worker management

​

Security Checker (internal/security/security.go)

• VerifyOutputFile() - Validates converted files using external tools
• IsFileCorrupted() - Detects corrupted existing files
• VerifyFileIntegrity() - Comprehensive integrity checks

• CreateProcessingMarker() - Marks files being processed (PID + timestamp)
• RemoveProcessingMarker() - Cleans up after successful conversion
• FindAbandonedMarkers() - Detects dead processes
• CleanupAbandonedFiles() - Removes orphaned .tmp files

• Platform-specific implementations (Unix/Windows)
• Pre-conversion space validation
• Estimates needed space (50% of source for safety)

​

Logger (internal/logger/logger.go)

• Colored terminal output
• Progress bars and statistics
• File logging to {destination}/conversion.log

• Structured events to stdout
• File logging (no colors)
• Tauri frontend integration

• started - Conversion begins
• progress - Overall progress updates
• file_start - Individual file processing starts
• file_end - Individual file processing completes
• log - General log messages
• error - Error events
• complete - Conversion finished
• statistics - Detailed stats

​

Utilities (internal/utils/utils.go)

1. macOS mdls metadata (most reliable for RAW files)
2. EXIF data via ImageMagick (EXIF:DateTimeOriginal)
3. Video metadata via FFmpeg (creation_time)
4. File modification time (with validation)

• CreateDestinationPath() - Generates organized paths
• GetMonthName() - Multi-language month names (EN, FR, ES, DE)
• ShouldSkipSystemEntry() - Filters system files/directories

• FFmpeg and ffprobe
• ImageMagick (magick command)
• Hardware acceleration detection

​

Checksum (internal/checksum/checksum.go)

• xxHash64 implementation (faster than MD5/SHA256)
• Stream-based calculation (low memory usage)
• Duplicate detection
• Copy-mode verification

​

API (internal/api/json_writer.go)

// Example event structure
type JSONEvent struct {
    Type      EventType   `json:"type"`
    Timestamp time.Time   `json:"timestamp"`
    Data      interface{} `json:"data"`
}

​

Adaptive Worker Management

• AdaptiveLimiter - Dynamic semaphore with adjustable limits
• ResourceMonitor - Tracks CPU and memory usage
• AdaptiveController - Adjusts worker count based on thresholds

adaptive_workers:
  enabled: true
  min: 1
  max: 6
  cpu_high: 80.0   # Reduce workers above 80% CPU
  cpu_low: 50.0    # Increase workers below 50% CPU
  mem_low_percent: 20.0  # Reduce workers if <20% memory available
  interval_seconds: 3

​

Safety and Idempotence

​

Idempotent Operations

• Run multiple times safely - already converted files are skipped
• Resume after interruption (Ctrl+C, crashes, failures)
• Atomic conversions via .tmp files + atomic rename

​

Safety Guarantees

• Originals preserved by default (--keep-originals=true)
• Triple verification before deletion
• Processing markers track active conversions
• Automatic recovery from crashes
• Corruption detection

​

Recovery System

1. Skips successfully converted files
2. Cleans up temporary files from interrupted conversions
3. Re-converts partially processed or corrupted files
4. Continues from where it left off

​

Platform-Specific Code

• internal/security/diskspace_*.go - Disk space checks (Unix/Windows)
• internal/security/process_check_*.go - Process existence checks (Unix/Windows)
• internal/converter/resource_*.go - Resource monitoring (Darwin/Linux/Stub)

​

External Dependencies

​

Required Tools

• FFmpeg - Video conversion and metadata extraction
• FFprobe - Video analysis
• ImageMagick (magick) - Image conversion and EXIF data

​

Go Dependencies

• github.com/spf13/cobra - CLI framework
• github.com/spf13/viper - Configuration management
• github.com/sirupsen/logrus - Structured logging
• github.com/fatih/color - Terminal colors
• golang.org/x/sync - Semaphore for worker pools
• github.com/cespare/xxhash/v2 - Fast checksum calculation

​

Next Steps