Background Jobs with Trigger.dev

Overview

Polaris IDE uses Trigger.dev for background job processing, enabling non-blocking AI operations and long-running tasks.

Background jobs run on Trigger.dev’s infrastructure, keeping the UI responsive while processing complex operations.

Why Background Jobs?

AI operations can take several seconds to complete:

Without Background Jobs

UI freezes during AI processing
Request timeouts on long operations
Poor user experience
No retry on failure

With Background Jobs

Instant UI response
No timeout limits
Real-time progress updates
Automatic retry and error handling

Architecture

Job Definitions

Polaris includes two main background jobs:

Process Message Task

Location: trigger/tasks/process-message.ts Handles AI chat responses with tool execution:

import { task } from "@trigger.dev/sdk/v3";
import { streamTextWithToolsPreferCerebras } from "@/lib/generate-text-with-tools";

export const processMessage = task({
  id: "process-message",
  run: async (payload: { messageId: Id<"messages"> }, { ctx }) => {
    // 1. Get message context from Convex
    const context = await convex.query(api.system.getMessageContext, {
      internalKey,
      messageId: payload.messageId
    });

    // 2. Prepare AI tools
    const tools = {
      ...createFileTools(context.projectId, internalKey),
      ...createLSPTools(context.projectId, internalKey),
      ...createSearchTools(context.projectId, internalKey),
      ...createTerminalTools(context.projectId, internalKey),
      ...createContextTools(context.projectId, internalKey)
    };

    // 3. Stream AI response
    const response = await streamTextWithToolsPreferCerebras({
      system: SYSTEM_PROMPT,
      messages: context.messages,
      tools,
      maxSteps: 10,
      onTextChunk: async (chunk, fullText) => {
        // Real-time update to Convex
        await convex.mutation(api.system.streamMessageContent, {
          messageId,
          content: fullText,
          isComplete: false
        });
      },
      onStepFinish: async ({ toolCalls, toolResults }) => {
        // Record tool usage
        for (const toolCall of toolCalls) {
          await convex.mutation(api.system.appendToolCall, {
            messageId,
            toolCall
          });
        }
      }
    });

    // 4. Mark complete
    await convex.mutation(api.system.streamMessageContent, {
      messageId,
      content: response.text,
      isComplete: true
    });

    return { success: true, messageId };
  }
});

Available Tools

File Management:

readFile - Read file contents
writeFile - Create/update files
deleteFile - Delete files/folders
listFiles - List directory contents
getProjectStructure - Get file tree

Code Analysis (LSP):

findSymbol - Search for functions, classes, variables
getReferences - Find symbol references
getDiagnostics - Get TypeScript errors
goToDefinition - Navigate to definitions

Code Search:

searchFiles - Regex content search
searchCodebase - AST-aware search
findFilesByPattern - Glob pattern matching

Context & Relevance:

getRelevantFiles - Find contextually relevant files

Terminal:

executeCommand - Run safe commands (npm, git, tsc, etc.)

Streaming Updates

Updates are sent to Convex every 100ms (throttled):

const STREAM_THROTTLE_MS = 100;
let lastStreamUpdate = 0;

onTextChunk: async (chunk: string, fullText: string) => {
  const now = Date.now();
  if (now - lastStreamUpdate >= STREAM_THROTTLE_MS) {
    lastStreamUpdate = now;
    await convex.mutation(api.system.streamMessageContent, {
      internalKey,
      messageId,
      content: fullText,
      isComplete: false
    });
  }
}

Error Handling

Failures are gracefully handled:

try {
  // Process message
} catch (error) {
  console.error("Error processing message:", error);
  
  await convex.mutation(api.system.updateMessageContent, {
    internalKey,
    messageId,
    content: "I encountered an error. Please try again.",
    status: "failed"
  });
  
  throw error; // Trigger.dev will retry
}

Generate Project Task

Location: trigger/tasks/generate-project.ts Creates complete projects from descriptions:

export const generateProject = task({
  id: "generate-project",
  run: async (payload: GenerateProjectPayload) => {
    const { projectId, description, internalKey } = payload;

    // Multi-step generation process
    await runGenerationStep({
      id: "generate-config-files",
      prompt: "Create package.json, tsconfig.json, vite.config.ts",
      maxSteps: 8
    });

    await runGenerationStep({
      id: "generate-source-structure",
      prompt: "Create src/main.tsx, src/App.tsx",
      maxSteps: 8
    });

    await runGenerationStep({
      id: "generate-components",
      prompt: "Create shared UI components",
      maxSteps: 8
    });

    await runGenerationStep({
      id: "generate-pages",
      prompt: "Create page components",
      maxSteps: 8
    });

    await runGenerationStep({
      id: "generate-hooks",
      prompt: "Create custom hooks",
      maxSteps: 6
    });

    await runGenerationStep({
      id: "generate-types",
      prompt: "Create type definitions",
      maxSteps: 4
    });

    await runGenerationStep({
      id: "generate-utilities",
      prompt: "Create utility functions",
      maxSteps: 6
    });

    await runGenerationStep({
      id: "finalize-readme",
      prompt: "Create README.md",
      maxSteps: 4
    });

    return { success: true, projectId };
  }
});

Step-by-Step Generation

Each step focuses on specific files:

Config files - package.json, tsconfig.json
Entry points - main.tsx, App.tsx
Components - UI building blocks
Pages - Route components
Hooks - Custom React hooks
Types - TypeScript definitions
Utilities - Helper functions
Documentation - README.md

Progress Tracking

Events logged to Convex for real-time UI updates:

await logEvent({
  type: "step",
  message: "Starting generate-components"
});

await logEvent({
  type: "file",
  message: "Created src/components/Button.tsx",
  filePath: "src/components/Button.tsx",
  preview: "export function Button({ ... })"
});

Tool Choice

Force specific tools for each step:

const toolChoice = (stepNumber: number) => 
  stepNumber === 0 
    ? { type: "tool", toolName: "writeFile" }
    : "auto";

// Ensures first action is always writing files

Invoking Jobs

From Frontend (via API Route)

// src/app/api/messages/route.ts
import { trigger } from "@/lib/trigger-client";

export async function POST(req: Request) {
  const { conversationId, content } = await req.json();
  
  // Create message in Convex
  const messageId = await convex.mutation(api.messages.create, {
    conversationId,
    content,
    role: "user"
  });
  
  // Trigger background job
  const run = await trigger.tasks.trigger("process-message", {
    messageId
  });
  
  return Response.json({ 
    messageId,
    runId: run.id 
  });
}

From Convex (via HTTP Action)

// convex/projects.ts
import { httpAction } from "./_generated/server";

export const generateProjectAction = httpAction(async (ctx, request) => {
  const { projectId, description } = await request.json();
  
  // Trigger background job
  const response = await fetch(
    `${process.env.TRIGGER_API_URL}/api/v1/tasks/generate-project/trigger`,
    {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${process.env.TRIGGER_SECRET_KEY}`,
        "Content-Type": "application/json"
      },
      body: JSON.stringify({ projectId, description })
    }
  );
  
  return new Response(JSON.stringify(await response.json()));
});

Monitoring Jobs

Trigger.dev Dashboard

Access at cloud.trigger.dev:

Run History

View all job executions with status and duration

Live Runs

Monitor currently executing jobs in real-time

Logs

Detailed execution logs and error traces

Metrics

Job success rate, duration, and retry statistics

Frontend Monitoring

Track job status in UI:

const MessageStatus = ({ messageId }: { messageId: string }) => {
  const message = useQuery(api.messages.getById, { messageId });
  
  if (!message) return null;
  
  return (
    <div>
      {message.status === "processing" && (
        <div className="flex items-center gap-2">
          <Loader className="animate-spin" />
          Processing...
        </div>
      )}
      
      {message.status === "completed" && (
        <div className="flex items-center gap-2">
          <Check className="text-green-500" />
          Complete
        </div>
      )}
      
      {message.status === "failed" && (
        <div className="flex items-center gap-2">
          <X className="text-red-500" />
          Failed - <button onClick={retry}>Retry</button>
        </div>
      )}
    </div>
  );
};

Creating New Tasks

Define task

Create trigger/tasks/my-task.ts:

import { task } from "@trigger.dev/sdk/v3";

export const myTask = task({
  id: "my-task",
  run: async (payload: { data: string }) => {
    console.log("Processing:", payload.data);
    
    // Your logic here
    
    return { success: true };
  }
});

Deploy task

npx trigger.dev deploy

Invoke from code

import { trigger } from "@/lib/trigger-client";

await trigger.tasks.trigger("my-task", {
  data: "Hello, world!"
});

Advanced Features

Retry Logic

export const unreliableTask = task({
  id: "unreliable-task",
  retry: {
    maxAttempts: 3,
    factor: 2,
    minTimeoutInMs: 1000,
    maxTimeoutInMs: 10000,
    randomize: true
  },
  run: async (payload) => {
    // May fail and retry automatically
  }
});

Scheduled Jobs

import { schedules } from "@trigger.dev/sdk/v3";

export const dailyCleanup = schedules.task({
  id: "daily-cleanup",
  cron: "0 2 * * *", // 2 AM daily
  run: async () => {
    // Clean up old conversations
    await cleanupOldData();
  }
});

Delayed Execution

import { trigger } from "@/lib/trigger-client";

await trigger.tasks.trigger("send-reminder", 
  { userId: "123" },
  { delay: 3600 } // 1 hour delay
);

Batching

export const batchProcessor = task({
  id: "batch-processor",
  run: async (payload: { items: string[] }) => {
    const results = [];
    
    for (const item of payload.items) {
      const result = await processItem(item);
      results.push(result);
    }
    
    return { results };
  }
});

Best Practices

Keep tasks focused

Each task should do one thing well
Break complex operations into multiple tasks
Use task chaining for multi-step workflows

Handle errors gracefully

Always catch and log errors
Return meaningful error messages
Use try-catch for external API calls
Let Trigger.dev handle retries

Provide progress updates

Stream updates for long operations
Update Convex records frequently
Show progress in UI
Log milestones for debugging

Optimize performance

Use throttling for real-time updates
Batch database operations
Cache expensive computations
Set appropriate timeouts

Test thoroughly

Test with development environment
Verify retry behavior
Check error handling
Monitor production metrics

Troubleshooting

Task not triggering

Symptoms: Job doesn’t startSolutions:

Verify TRIGGER_SECRET_KEY is set
Check task is deployed: npx trigger.dev deploy
Ensure task ID matches trigger call
Check Trigger.dev dashboard for errors

Job timing out

Symptoms: Job fails with timeout errorSolutions:

Increase timeout in task config
Break into smaller tasks
Optimize expensive operations
Use streaming for long-running AI calls

High failure rate

Symptoms: Jobs frequently failSolutions:

Check Trigger.dev logs for error details
Verify external API credentials
Add better error handling
Increase retry attempts
Monitor Sentry for exceptions

Slow execution

Symptoms: Jobs take too longSolutions:

Profile code for bottlenecks
Reduce AI token limits
Optimize database queries
Use parallel processing where possible
Consider caching results

Next Steps

Architecture

Understand the system design

Error Tracking

Monitor job failures with Sentry

Custom Extensions

Build AI tools for your tasks

Contributing

Add new background jobs

Development

Deployment

Advanced

Background Jobs with Trigger.dev

Overview

Why Background Jobs?

Without Background Jobs

With Background Jobs

Architecture

Job Definitions

Process Message Task

Generate Project Task

Invoking Jobs

From Frontend (via API Route)

From Convex (via HTTP Action)

Monitoring Jobs

Trigger.dev Dashboard

Run History

Live Runs

Logs

Metrics

Frontend Monitoring

Creating New Tasks

Advanced Features

Retry Logic

Scheduled Jobs

Delayed Execution

Batching

Best Practices

Troubleshooting

Next Steps

Architecture

Error Tracking

Custom Extensions

Contributing

Build docs developers (and LLMs) love

Development

Deployment

Advanced

​Overview

​Why Background Jobs?

Without Background Jobs

With Background Jobs

​Architecture

​Job Definitions

​Process Message Task

​Generate Project Task

​Invoking Jobs

​From Frontend (via API Route)

​From Convex (via HTTP Action)

​Monitoring Jobs

​Trigger.dev Dashboard

Run History

Live Runs

Logs

Metrics

​Frontend Monitoring

​Creating New Tasks

​Advanced Features

​Retry Logic

​Scheduled Jobs

​Delayed Execution

​Batching

​Best Practices

​Troubleshooting

​Next Steps

Architecture

Error Tracking

Custom Extensions

Contributing

Build docs developers (and LLMs) love

Overview

Why Background Jobs?

Architecture

Job Definitions

Process Message Task

Generate Project Task

Invoking Jobs

From Frontend (via API Route)

From Convex (via HTTP Action)

Monitoring Jobs

Trigger.dev Dashboard

Frontend Monitoring

Creating New Tasks

Advanced Features

Retry Logic

Scheduled Jobs

Delayed Execution

Batching

Best Practices

Troubleshooting

Next Steps