Background Jobs

FullStackHero uses Hangfire for background job processing, enabling you to run tasks asynchronously, schedule recurring jobs, and implement the transactional outbox pattern for reliable event publishing.

Overview

The background jobs system provides:

Hangfire Integration: Persistent, reliable background job processing
Recurring Jobs: Schedule jobs to run on a cron schedule
Delayed Jobs: Schedule jobs to run at a specific time
Job Dashboard: Web UI for monitoring and managing jobs
Outbox Pattern: Reliably publish integration events after database transactions
Multi-Tenant Support: Jobs run in the correct tenant context
OpenTelemetry Tracing: Automatic instrumentation for job execution

Configuration

Configure Hangfire in appsettings.json:

appsettings.json

{
  "HangfireOptions": {
    "Username": "admin",
    "Password": "Secure1234!Me",
    "Route": "/jobs"
  },
  "DatabaseOptions": {
    "Provider": "POSTGRESQL",
    "ConnectionString": "Server=localhost;Database=fsh;User Id=postgres;Password=password",
    "MigrationsAssembly": "FSH.Playground.Migrations.PostgreSQL"
  }
}

HangfireOptions

Username

string

default:"admin"

Username for accessing the Hangfire dashboard.

Password

string

required

Password for accessing the Hangfire dashboard. Use a strong password in production.

Route

string

default:"/jobs"

URL path for the Hangfire dashboard (e.g., https://api.example.com/jobs).

Hangfire uses the same database as your application (configured in DatabaseOptions).

Hangfire Setup

Hangfire is configured in the Jobs building block:

Extensions.cs

public static IServiceCollection AddHeroJobs(this IServiceCollection services)
{
    ArgumentNullException.ThrowIfNull(services);

    services.AddOptions<HangfireOptions>()
        .BindConfiguration(nameof(HangfireOptions));

    services.AddHangfireServer(options =>
    {
        options.HeartbeatInterval = TimeSpan.FromSeconds(30);
        options.Queues = ["default", "email"];
        options.WorkerCount = 5;
        options.SchedulePollingInterval = TimeSpan.FromSeconds(30);
    });

    services.AddHangfire((provider, config) =>
    {
        var configuration = provider.GetRequiredService<IConfiguration>();
        var dbOptions = configuration.GetSection(nameof(DatabaseOptions))
            .Get<DatabaseOptions>() 
            ?? throw new CustomException("Database options not found");

        switch (dbOptions.Provider.ToUpperInvariant())
        {
            case DbProviders.PostgreSQL:
                CleanupStaleLocks(dbOptions.ConnectionString, provider);
                config.UsePostgreSqlStorage(o =>
                {
                    o.UseNpgsqlConnection(dbOptions.ConnectionString);
                });
                break;

            case DbProviders.MSSQL:
                config.UseSqlServerStorage(dbOptions.ConnectionString);
                break;

            default:
                throw new CustomException(
                    $"Hangfire storage provider {dbOptions.Provider} is not supported");
        }

        config.UseFilter(new FshJobFilter(provider));
        config.UseFilter(new LogJobFilter());
        config.UseFilter(new HangfireTelemetryFilter());
    });

    services.AddTransient<IJobService, HangfireService>();

    return services;
}

Key Configuration

Worker Configuration

Configures 5 workers processing jobs from the default and email queues.

Database Storage

Uses PostgreSQL or SQL Server for persistent job storage.

Filters

Applies custom filters for tenant context, logging, and OpenTelemetry tracing.

Stale Lock Cleanup

Removes stale database locks from crashed instances on startup.

Job Dashboard

Access the Hangfire dashboard at /jobs:

https://localhost:7030/jobs

Login Credentials (from appsettings.json):

Username: admin
Password: Secure1234!Me

The dashboard provides:

Jobs: View all jobs (enqueued, processing, succeeded, failed)
Recurring Jobs: Manage recurring job schedules
Servers: Monitor Hangfire server instances
Retries: View and retry failed jobs
Statistics: Job execution metrics

The Hangfire dashboard is protected with basic authentication. Change the default credentials in production.

Using Background Jobs

Enqueue a Job

Enqueue a job to run immediately:

public class MyService
{
    private readonly IJobService _jobService;

    public MyService(IJobService jobService)
    {
        _jobService = jobService;
    }

    public async Task ProcessOrderAsync(Guid orderId)
    {
        // Enqueue job
        var jobId = await _jobService.EnqueueAsync<OrderProcessor>(
            processor => processor.ProcessAsync(orderId, CancellationToken.None));
        
        Console.WriteLine($"Job {jobId} enqueued");
    }
}

Schedule a Delayed Job

Schedule a job to run at a specific time:

// Run in 1 hour
var jobId = await _jobService.ScheduleAsync<EmailService>(
    service => service.SendReminderAsync(userId, CancellationToken.None),
    TimeSpan.FromHours(1));

Create a Recurring Job

Define jobs that run on a schedule:

// Run every day at 2:00 AM
await _jobService.RecurringAsync<ReportGenerator>(
    "daily-report",
    generator => generator.GenerateAsync(CancellationToken.None),
    Cron.Daily(2));

Common cron expressions:

Cron.Minutely()

Outbox Pattern

The outbox pattern ensures integration events are reliably published, even if the message broker is temporarily unavailable.

How It Works

Store Event in Outbox

When a domain event occurs, save it to the OutboxMessages table within the same database transaction.

Commit Transaction

Commit the database transaction. The event is now persisted.

Background Job Processes Outbox

A recurring Hangfire job reads pending messages from the outbox and publishes them to the event bus.

Mark as Processed

Once published, the message is marked as processed in the outbox.

Outbox Message Entity

OutboxMessage.cs

public class OutboxMessage
{
    public Guid Id { get; set; }
    public DateTime CreatedOnUtc { get; set; }
    public string Type { get; set; } = default!;
    public string Payload { get; set; } = default!;
    public string? TenantId { get; set; }
    public string? CorrelationId { get; set; }
    public DateTime? ProcessedOnUtc { get; set; }
    public int RetryCount { get; set; }
    public string? LastError { get; set; }
    public bool IsDead { get; set; }
}

Adding Events to the Outbox

Use IOutboxStore to persist events:

public class GenerateTokenCommandHandler 
    : ICommandHandler<GenerateTokenCommand, TokenResponse>
{
    private readonly IOutboxStore _outboxStore;

    public async ValueTask<TokenResponse> Handle(
        GenerateTokenCommand command,
        CancellationToken ct)
    {
        // Business logic...
        var token = await _tokenService.IssueAsync(...);

        // Add integration event to outbox
        var integrationEvent = new TokenGeneratedIntegrationEvent(
            Id: Guid.NewGuid(),
            OccurredOnUtc: DateTime.UtcNow,
            TenantId: tenantId,
            CorrelationId: correlationId,
            Source: "Identity",
            UserId: subject,
            Email: command.Email,
            ClientId: clientId!,
            IpAddress: ip,
            UserAgent: ua,
            TokenFingerprint: fingerprint,
            AccessTokenExpiresAtUtc: token.AccessTokenExpiresAt);

        await _outboxStore.AddAsync(integrationEvent, ct)
            .ConfigureAwait(false);

        return token;
    }
}

Outbox Dispatcher

The OutboxDispatcher processes pending messages:

OutboxDispatcher.cs

public sealed class OutboxDispatcher
{
    private readonly IOutboxStore _outbox;
    private readonly IEventBus _bus;
    private readonly IEventSerializer _serializer;
    private readonly ILogger<OutboxDispatcher> _logger;
    private readonly EventingOptions _options;

    public async Task DispatchAsync(CancellationToken ct = default)
    {
        var batchSize = _options.OutboxBatchSize;
        if (batchSize <= 0) batchSize = 100;

        var messages = await _outbox.GetPendingBatchAsync(batchSize, ct)
            .ConfigureAwait(false);
        
        if (messages.Count == 0)
        {
            _logger.LogDebug("No outbox messages to dispatch.");
            return;
        }

        _logger.LogInformation(
            "Dispatching {Count} outbox messages (BatchSize={BatchSize})",
            messages.Count, batchSize);

        foreach (var message in messages)
        {
            try
            {
                var @event = _serializer.Deserialize(
                    message.Payload, message.Type);
                
                await _bus.PublishAsync(@event, ct).ConfigureAwait(false);
                await _outbox.MarkAsProcessedAsync(message, ct)
                    .ConfigureAwait(false);
                
                _logger.LogDebug(
                    "Outbox message {MessageId} dispatched and marked as processed.",
                    message.Id);
            }
            catch (Exception ex)
            {
                var maxRetries = _options.OutboxMaxRetries <= 0 
                    ? 5 : _options.OutboxMaxRetries;
                var isDead = message.RetryCount + 1 >= maxRetries;

                await _outbox.MarkAsFailedAsync(
                    message, ex.Message, isDead, ct).ConfigureAwait(false);

                if (isDead)
                {
                    _logger.LogError(ex, 
                        "Outbox message {MessageId} moved to dead-letter after {RetryCount} retries",
                        message.Id, message.RetryCount + 1);
                }
                else
                {
                    _logger.LogWarning(ex, 
                        "Outbox message {MessageId} failed (RetryCount={RetryCount}).",
                        message.Id, message.RetryCount + 1);
                }
            }
        }
    }
}

Scheduling the Outbox Dispatcher

await _jobService.RecurringAsync<OutboxDispatcher>(
    "outbox-dispatcher",
    dispatcher => dispatcher.DispatchAsync(CancellationToken.None),
    "*/30 * * * * *");  // Every 30 seconds

Job Filters

Hangfire filters add cross-cutting concerns to jobs:

Tenant Context Filter

Ensures jobs run in the correct tenant context:

FshJobFilter.cs

public class FshJobFilter : IServerFilter
{
    private readonly IServiceProvider _serviceProvider;

    public void OnPerforming(PerformingContext context)
    {
        var tenantId = context.GetJobParameter<string>("TenantId");
        if (!string.IsNullOrEmpty(tenantId))
        {
            // Set tenant context for job execution
            var tenantAccessor = _serviceProvider
                .GetRequiredService<IMultiTenantContextAccessor<AppTenantInfo>>();
            // Set tenant...
        }
    }
}

Logging Filter

Logs job execution:

LogJobFilter.cs

public class LogJobFilter : IServerFilter
{
    public void OnPerforming(PerformingContext context)
    {
        Console.WriteLine($"Starting job: {context.BackgroundJob.Job.Method.Name}");
    }

    public void OnPerformed(PerformedContext context)
    {
        Console.WriteLine($"Completed job: {context.BackgroundJob.Job.Method.Name}");
    }
}

Telemetry Filter

Adds OpenTelemetry tracing to jobs:

HangfireTelemetryFilter.cs

public class HangfireTelemetryFilter : IServerFilter
{
    public void OnPerforming(PerformingContext context)
    {
        var activitySource = new ActivitySource("FSH.Hangfire");
        var activity = activitySource.StartActivity(
            $"Job: {context.BackgroundJob.Job.Method.Name}",
            ActivityKind.Internal);
        
        activity?.SetTag("job.id", context.BackgroundJob.Id);
        activity?.SetTag("job.type", context.BackgroundJob.Job.Type.Name);
    }
}

Best Practices

Use Queues for Priority

Separate jobs into queues (e.g., default, email, reports) and configure workers accordingly.

Make Jobs Idempotent

Jobs may be retried. Ensure they can safely run multiple times without side effects.

Set Retry Policies

Configure automatic retries for transient failures, but limit retries to avoid infinite loops.

Monitor Job Failures

Set up alerts for failed jobs and dead-letter messages.

Use the Outbox Pattern for Events

Never publish events directly to a message broker. Always use the outbox for reliability.

Testing Background Jobs

Test that jobs execute correctly:

[Fact]
public async Task EnqueueJob_ExecutesSuccessfully()
{
    // Arrange
    var jobService = _serviceProvider.GetRequiredService<IJobService>();
    var processor = _serviceProvider.GetRequiredService<OrderProcessor>();

    // Act
    var jobId = await jobService.EnqueueAsync<OrderProcessor>(
        p => p.ProcessAsync(Guid.NewGuid(), CancellationToken.None));

    // Wait for job to complete (test environment)
    await Task.Delay(2000);

    // Assert
    var jobDetails = JobStorage.Current
        .GetMonitoringApi()
        .JobDetails(jobId);
    
    Assert.Equal("Succeeded", jobDetails.History[0].StateName);
}

Observability

Trace and monitor background job execution

Multi-Tenancy

Run jobs in tenant context

Health Checks

Monitor Hangfire server health

Integration Events

Learn about the outbox pattern and event-driven architecture

Overview

Getting Started

Architecture

Building Blocks

Modules

Development Guide

Features

Deployment

Overview

Configuration

HangfireOptions

Hangfire Setup

Key Configuration

Job Dashboard

Using Background Jobs

Enqueue a Job

Schedule a Delayed Job

Create a Recurring Job

Outbox Pattern

How It Works

Outbox Message Entity

Adding Events to the Outbox

Outbox Dispatcher

Scheduling the Outbox Dispatcher

Job Filters

Tenant Context Filter

Logging Filter

Telemetry Filter

Best Practices

Testing Background Jobs

Observability

Multi-Tenancy

Health Checks

Integration Events

Build docs developers (and LLMs) love

Overview

Getting Started

Architecture

Building Blocks

Modules

Development Guide

Features

Deployment

​Overview

​Configuration

​HangfireOptions

​Hangfire Setup

​Key Configuration

​Job Dashboard

​Using Background Jobs

​Enqueue a Job

​Schedule a Delayed Job

​Create a Recurring Job

​Outbox Pattern

​How It Works

​Outbox Message Entity

​Adding Events to the Outbox

​Outbox Dispatcher

​Scheduling the Outbox Dispatcher

​Job Filters

​Tenant Context Filter

​Logging Filter

​Telemetry Filter

​Best Practices

​Testing Background Jobs

​Related Topics

Observability

Multi-Tenancy

Health Checks

Integration Events

Build docs developers (and LLMs) love

Overview

Configuration

HangfireOptions

Hangfire Setup

Key Configuration

Job Dashboard

Using Background Jobs

Enqueue a Job

Schedule a Delayed Job

Create a Recurring Job

Outbox Pattern

How It Works

Outbox Message Entity

Adding Events to the Outbox

Outbox Dispatcher

Scheduling the Outbox Dispatcher

Job Filters

Tenant Context Filter

Logging Filter

Telemetry Filter

Best Practices

Testing Background Jobs

Related Topics