Architecture - OpenCouncil

High-level architecture

OpenCouncil is built as a modern web application with a decoupled task processing architecture. The system consists of three main components:

Next.js application

User-facing web application with Server Components and API routes

Task server

Background processing server for AI and media tasks

PostgreSQL database

Data persistence with PostGIS for geospatial features

Technology stack

Frontend layer

// Next.js 14 with App Router and TypeScript
// Server Components by default, Client Components when needed
import { Suspense } from 'react'
import MeetingList from '@/components/meetings/MeetingList'

export default function MeetingsPage() {
  return (
    <Suspense fallback={<LoadingSkeleton />}>
      <MeetingList />
    </Suspense>
  )
}

Key technologies:

Next.js 14: App Router with Server Components and React Server Actions
TypeScript: Strict mode enabled for type safety
Tailwind CSS: Utility-first styling with custom design system
Radix UI: Accessible component primitives
Framer Motion: Animation library
React Hook Form + Zod: Form handling and validation

Backend layer

// Prisma ORM with type-safe queries
import prisma from '@/lib/db/prisma'

export async function getCouncilMeetings(cityId: string) {
  return await prisma.councilMeeting.findMany({
    where: { cityId },
    include: {
      administrativeBody: true,
      subjects: true,
    },
    orderBy: { date: 'desc' },
  })
}

Key technologies:

Prisma ORM: Type-safe database access with migrations
Auth.js: Session-based authentication with email magic links
PostgreSQL 14+: Primary database with PostGIS extension
Elasticsearch: Full-text search and content discovery

AI and media processing

// Anthropic Claude for summaries and chat
import Anthropic from '@anthropic-ai/sdk'

const client = new Anthropic({
  apiKey: env.ANTHROPIC_API_KEY,
})

export async function generateSummary(text: string) {
  const response = await client.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 1024,
    messages: [{ role: 'user', content: text }],
  })
  return response.content
}

Key technologies:

Anthropic Claude: AI summaries and chat assistant
Google Cloud Speech-to-Text: Audio transcription
DigitalOcean Spaces: S3-compatible object storage for media
Custom task queue: Background job processing with callbacks

Application architecture

Directory structure

opencouncil/
├── src/
│   ├── app/                    # Next.js App Router
│   │   ├── [locale]/          # Locale-parameterized routes
│   │   │   ├── page.tsx       # Home page
│   │   │   ├── cities/        # City pages
│   │   │   ├── meetings/      # Meeting pages
│   │   │   └── search/        # Search interface
│   │   └── api/               # API routes
│   │       ├── cities/        # City CRUD operations
│   │       ├── meetings/      # Meeting operations
│   │       ├── search/        # Search endpoints
│   │       ├── chat/          # AI chat
│   │       └── cron/          # Scheduled tasks
│   ├── components/            # React components
│   │   ├── ui/               # Base UI components (Radix)
│   │   ├── meetings/         # Meeting-specific components
│   │   ├── chat/             # Chat interface
│   │   ├── map/              # Mapbox integration
│   │   └── search/           # Search components
│   ├── lib/                   # Business logic & services
│   │   ├── db/               # Data access layer
│   │   │   ├── cities.ts     # City queries
│   │   │   ├── meetings.ts   # Meeting queries
│   │   │   └── types/        # Shared Prisma types
│   │   ├── tasks/            # Task management
│   │   │   ├── tasks.ts      # Core task logic
│   │   │   ├── transcribe.ts # Transcription handler
│   │   │   ├── summarize.ts  # Summary handler
│   │   │   └── registry.ts   # Task handler registry
│   │   ├── search/           # Elasticsearch integration
│   │   ├── notifications/    # Multi-channel notifications
│   │   ├── ai.ts             # AI integration
│   │   ├── s3.ts             # Object storage
│   │   └── auth.ts           # Authentication helpers
│   ├── contexts/             # React Context providers
│   ├── hooks/                # Custom React hooks
│   └── types/                # TypeScript type definitions
├── prisma/
│   ├── schema.prisma         # Database schema
│   ├── migrations/           # Migration files
│   └── seed.ts               # Seed data script
├── public/                    # Static assets
└── tests/                     # Test suites

Data access patterns

OpenCouncil follows strict data access patterns for maintainability:

All database queries use centralized functions in src/lib/db/. This prevents scattered query logic and ensures consistent data access patterns.

// Centralized city queries
export async function getCities() {
  return await prisma.city.findMany({
    where: { status: 'listed' },
    orderBy: { name: 'asc' },
  })
}

export async function getCityById(id: string) {
  return await prisma.city.findUnique({
    where: { id },
    include: {
      administrativeBodies: true,
      parties: true,
    },
  })
}

Type storage:

Store shared Prisma types in src/lib/db/types/{entity}.ts
Re-export from src/lib/db/types/index.ts
Import from @/lib/db/types to prevent circular dependencies

Task workflow architecture

The task system is designed to offload long-running processes from the main application to a dedicated backend server.

Task lifecycle

Initiation

User action or cron job triggers startTask() function:

await startTask(
  'transcribe',
  { audioUrl, language: 'el' },
  meetingId,
  cityId
)

Creates TaskStatus record in database
Sends POST request to task server with callbackUrl

Execution

Backend task server processes the job:

Task added to custom in-memory queue
Worker process executes the task
Sends progress updates to callback URL

Completion

Task server sends final result:

Status: success or error
Result data or error message
Full response stored in TaskStatus.responseBody

Result handling

Next.js application processes the result:

// Task handler registry automatically routes to correct handler
const handler = taskHandlers[taskType]
await handler(taskId, result, options)

Updates database with task results
Triggers downstream tasks if needed
Sends Discord notifications to admins

Task architecture diagram

Task types

OpenCouncil supports multiple task types:

Transcribe

Converts audio/video to text with speaker recognition:

Uses Google Cloud Speech-to-Text or Whisper
Generates speaker segments and utterances
Creates word-level timestamps
Identifies speakers using voiceprints

Handler: src/lib/tasks/transcribe.ts

Summarize

Generates AI summaries of speeches:

Analyzes speaker utterances
Extracts key topics and subjects
Creates structured summaries
Links to existing subjects or creates new ones

Handler: src/lib/tasks/summarize.ts

Generate voiceprint

Creates speaker voice profiles:

Analyzes audio samples
Generates unique voiceprint
Enables automatic speaker identification

Handler: src/lib/tasks/generateVoiceprint.ts

Generate podcast

Creates podcast-style audio content:

Extracts key segments from meetings
Generates host narration
Produces structured audio content

Handler: src/lib/tasks/generatePodcast.ts

Poll decisions

Fetches decisions from Diavgeia:

Queries Greek government transparency portal
Links decisions to meeting subjects
Progressive backoff to avoid over-polling

Handler: src/lib/tasks/pollDecisions.ts

Task reprocessing

A key feature is the ability to reprocess task results without re-running the entire task:

// Reprocess stored task result
await processTaskResponse('transcribe', taskId, { force: true })

Force mode: Some tasks (like transcribe) create data that can’t be updated in place. The force flag triggers cleanup:

Delete existing SpeakerSegment, Utterance, and Word records
Recreate everything from stored responseBody

This prevents duplicates and ensures data consistency.

Database architecture

Schema overview

OpenCouncil uses PostgreSQL with the PostGIS extension for geospatial features.

model City {
  id                String   @id @default(cuid())
  name              String   // Greek name
  name_en           String   // English name
  timezone          String
  geometry          Unsupported("geometry")?
  status            CityStatus @default(pending)
  
  councilMeetings   CouncilMeeting[]
  parties           Party[]
  persons           Person[]
}

model CouncilMeeting {
  cityId            String
  id                String
  name              String
  name_en           String
  date              DateTime
  videoUrl          String?
  audioUrl          String?
  
  subjects          Subject[]
  speakerSegments   SpeakerSegment[]
  taskStatuses      TaskStatus[]
  
  @@id([cityId, id])
}

Composite keys

Many models use composite keys (cityId, id) for multi-tenant data isolation:

model CouncilMeeting {
  cityId  String
  id      String
  // ... other fields
  
  @@id([cityId, id])
}

This pattern ensures meetings are scoped to their city and prevents ID collisions.

Key relationships

City → Meetings → Subjects

City
  └── CouncilMeeting (many)
        └── Subject (many)
              ├── Decision (one, optional)
              └── Mentions (many to Person/Party)

Meeting → Transcription

CouncilMeeting
  └── SpeakerSegment (many)
        ├── Person (one, optional)
        └── Utterance (many)
              ├── Words (many)
              └── Summary (text)

Task processing

CouncilMeeting
  └── TaskStatus (many)
        ├── Type (transcribe, summarize, etc.)
        ├── Status (pending, processing, succeeded, failed)
        ├── RequestBody (JSON)
        └── ResponseBody (JSON, stored for reprocessing)

Search architecture

OpenCouncil uses Elasticsearch for full-text search across transcripts.

Search features

Natural language queries

Parse user queries and extract filters automatically

Filter extraction

Identify city, person, party, topic, date, and location filters

Multi-field search

Search across utterances, summaries, and subjects

Retry logic

Exponential backoff for reliability

Search implementation

// src/lib/search/index.ts
export async function search(
  request: SearchRequest
): Promise<SearchResponse> {
  // Extract filters from natural language query
  const filters = await extractFilters(request.query)
  
  // Build Elasticsearch query
  const query = buildSearchQuery({
    query: request.query,
    cityIds: filters.cityIds || request.cityIds,
    personIds: filters.personIds,
    dateRange: filters.dateRange,
  })
  
  // Execute with retry logic
  const results = await executeElasticsearchWithRetry(query)
  
  return results
}

Notification system

Multi-channel notification delivery with user preferences:

Matching engine

Match notifications to user preferences:

Topics of interest
Specific people or parties
Geographic locations
Meeting types

Approval workflow

For cities with NOTIFICATIONS_APPROVAL mode:

Notifications enter pending queue
Admins review and approve
Batch sending on approval

Multi-channel delivery

Send via configured channels:

Email: Resend API
WhatsApp: Bird API
SMS: Bird API

Rate limiting

Prevent overwhelming users:

500ms delays between sends
Batch processing
Per-user throttling

Key integrations

Anthropic Claude

AI summaries, chat assistant, and content generation

Elasticsearch

Full-text search and content discovery

Resend

Email authentication and notifications

Bird API

WhatsApp and SMS notifications

DigitalOcean Spaces

S3-compatible object storage for media

Google Calendar

Event scheduling and synchronization

Discord

Real-time admin alerts for system events

Mapbox

Interactive maps and geospatial features

Deployment architecture

Production setup

FROM node:18-alpine AS base

# Install dependencies
RUN apk add --no-cache libc6-compat
WORKDIR /app

# Copy dependencies
COPY package*.json ./
COPY prisma ./prisma/

# Install dependencies
RUN npm ci

# Build application
COPY . .
RUN npm run build

# Production image
FROM node:18-alpine AS runner
WORKDIR /app

ENV NODE_ENV production

COPY --from=base /app/.next/standalone ./
COPY --from=base /app/.next/static ./.next/static
COPY --from=base /app/public ./public

EXPOSE 3000

CMD ["node", "server.js"]

Scaling considerations

When scaling horizontally, consider:

Database connection pooling (use Prisma Accelerate or PgBouncer)
Redis for session storage
CDN for static assets
Load balancer for multiple app instances

Security architecture

Authentication flow

User requests sign-in

User enters email on login page

Magic link sent

Auth.js generates secure token and sends via Resend

User clicks link

Token validated and session created

Session persisted

Encrypted session cookie stored in browser

Authorization patterns

OpenCouncil uses centralized authorization helpers:

// src/lib/auth.ts

// For conditional UI (returns boolean)
const editable = await isUserAuthorizedToEdit({ cityId })

// For API routes (throws if unauthorized)
await withUserAuthorizedToEdit({ cityId })

Both methods are async and must be awaited to prevent auth bypass bugs.

Performance optimizations

Caching strategy

import { cache } from 'react'

// Cache function for request deduplication
export const getCityById = cache(async (id: string) => {
  return await prisma.city.findUnique({ where: { id } })
})

Database optimization

Indexes: Strategic indexes on frequently queried fields
Connection pooling: Prisma connection pool configuration
Query optimization: Use select to fetch only needed fields
Pagination: Cursor-based pagination for large datasets

Monitoring and observability

Discord integration

Admin alerts sent to Discord webhook:

// Task completion alerts
await sendTaskAdminAlert({
  taskType: 'transcribe',
  status: 'succeeded',
  meetingName: 'City Council Meeting',
  duration: '2h 15m',
})

// Error alerts
await sendTaskAdminAlert({
  taskType: 'summarize',
  status: 'failed',
  error: 'API rate limit exceeded',
})

Logging

Structured logging for search analytics
Task execution logging
API request logging
Error tracking with context

Next steps

Database schema

Detailed database schema documentation

API reference

Complete REST API documentation

Task system

Deep dive into task processing

Contributing

Join the development workflow

Get Started

Core Features

Deployment

Development

Guides

​High-level architecture

Next.js application

Task server

PostgreSQL database

​Technology stack

​Frontend layer

​Backend layer

​AI and media processing

​Application architecture

​Directory structure

​Data access patterns

​Task workflow architecture

​Task lifecycle

​Task architecture diagram

​Task types

​Task reprocessing

​Database architecture

​Schema overview

​Composite keys

​Key relationships

​Search architecture

​Search features

Natural language queries

Filter extraction

Multi-field search

Retry logic

​Search implementation

​Notification system

​Key integrations

Anthropic Claude

Elasticsearch

Resend

Bird API

DigitalOcean Spaces

Google Calendar

Discord

Mapbox

​Deployment architecture

​Production setup

​Scaling considerations

​Security architecture

​Authentication flow

​Authorization patterns

​Performance optimizations

​Caching strategy

​Database optimization

​Monitoring and observability

​Discord integration

​Logging

​Next steps

Database schema

API reference

Task system

Contributing

Build docs developers (and LLMs) love

High-level architecture

Technology stack

Frontend layer

Backend layer

AI and media processing

Application architecture

Directory structure

Data access patterns

Task workflow architecture

Task lifecycle

Task architecture diagram

Task types

Task reprocessing

Database architecture

Schema overview

Composite keys

Key relationships

Search architecture

Search features

Search implementation

Notification system

Key integrations

Deployment architecture

Production setup

Scaling considerations

Security architecture

Authentication flow

Authorization patterns

Performance optimizations

Caching strategy

Database optimization

Monitoring and observability

Discord integration

Logging

Next steps