Skip to main content

​
Overview

Meridian provides flexible data ingestion options to get your data into the platform quickly. You can upload CSV files directly or extract structured data from web pages using AI-powered extraction.

​
Upload Methods

The data upload interface provides two methods for importing data:

​
File Upload

Drag and drop CSV, XLSX, or XLS files (up to 10MB) directly into Meridian. Files are automatically:
  1. Uploaded to Cloudflare R2 storage
  2. Processed into DuckDB tables
  3. Made available for querying and analysis

​
URL Extraction

Extract structured data from any webpage using AI:
  1. Enter the URL of the webpage
  2. Provide a prompt describing what data to extract
  3. Meridian uses Firecrawl to scrape and extract structured data
  4. The extracted data is converted to a table automatically
import { Dropzone, MIME_TYPES } from '@mantine/dropzone'
import { useUploadFile } from '@convex-dev/r2/react'

const handleDrop = async (acceptedFiles: File[]) => {
  for (const file of acceptedFiles) {
    // Upload to R2
    const storageId = await uploadFile(file)
    
    // Save metadata
    const fileId = await saveFile({
      storageId,
      fileName: file.name,
      fileType: file.type,
      fileSize: file.size,
    })
    
    // Create DuckDB table for CSV files
    if (file.type === 'text/csv') {
      const result = await createTableFromCSV({
        data: { csvUrl, tableName }
      })
    }
  }
}

​
Implementation Details

The upload flow is implemented in FileUpload.tsx and follows this architecture:

​
File Processing Pipeline

1

Upload to Storage

Files are uploaded to Cloudflare R2 using the Convex R2 component:
const uploadFile = useUploadFile(api.r2)
const storageId = await uploadFile(file)
2

Save Metadata

File metadata is saved to Convex database:
const fileId = await saveFile({
  storageId,
  fileName: file.name,
  fileType: file.type,
  fileSize: file.size,
})
3

Create DuckDB Table

CSV files are processed into queryable DuckDB tables:
const result = await createTableFromCSV({
  data: { csvUrl, tableName }
})
4

Link Table to File

The DuckDB table name is linked back to the file record:
await updateDuckDBInfo({
  fileId,
  tableName: result.tableName,
})

​
Usage Patterns

​
Basic CSV Upload

<FileUpload 
  onUploadComplete={() => {
    // Refresh table list
    // Navigate to new table
  }} 
/>

​
URL Extraction with Custom Prompt

For URL extraction, craft descriptive prompts: 
Extract all product information including name, price, 
description, availability status, and customer ratings

​
Progress Tracking

The upload component provides real-time progress feedback: 
{uploading && (
  <Box mt="md">
    <Text size="sm" mb="xs">
      Uploading... {uploadProgress}%
    </Text>
    <Progress value={uploadProgress} animated />
  </Box>
)}
Progress stages:
  • 20%: File uploaded to R2
  • 50-65%: Metadata saved
  • 85-95%: DuckDB table created
  • 100%: Complete

​
Advanced Tips

Table names are automatically generated from filenames:
const tableName = file.name
  .replace(/\.csv$/i, '')
  .replace(/[^a-zA-Z0-9_]/g, '_')
  .toLowerCase()
Example: Sales Data 2024.csv → sales_data_2024
Files up to 10MB are supported. For larger datasets:
  • Split into multiple files
  • Use URL extraction with pagination
  • Consider direct DuckDB import (see Architecture docs)
If DuckDB table creation fails:
catch (duckdbError) {
  // File is still uploaded and accessible
  // Table creation can be retried
  notifications.show({
    title: 'Warning',
    message: 'File uploaded but table creation failed',
    color: 'yellow',
  })
}

​
Common Use Cases

​
Uploading Sales Data

  1. Prepare your CSV with clean column headers
  2. Drag and drop the file into the upload zone
  3. Wait for processing (typically 5-15 seconds)
  4. Start querying immediately

​
Extracting Web Data

  1. Find a webpage with structured data
  2. Click the From URL tab
  3. Paste the URL
  4. Write a clear extraction prompt
  5. Click Extract Data & Create Table

​
Batch Uploads

You can upload multiple files at once: 
<Dropzone
  onDrop={handleDrop}
  maxSize={10 * 1024 ** 2}
  accept={[MIME_TYPES.csv, MIME_TYPES.xlsx, MIME_TYPES.xls]}
>
  {/* Multiple files are processed sequentially */}
</Dropzone>

​
API Integration

The upload feature integrates with these Convex APIs:
  • api.r2 - R2 storage operations (from `/home/daytona/workspace/source/src/components/dashboard/FileUpload.tsx:131)
  • api.csv.saveFile - Save file metadata (from `/home/daytona/workspace/source/src/components/dashboard/FileUpload.tsx:35)
  • api.csv.updateDuckDBInfo - Link table to file (from `/home/daytona/workspace/source/src/components/dashboard/FileUpload.tsx:38)
  • api.csv.createTableFromURL - URL extraction (from `/home/daytona/workspace/source/src/components/dashboard/FileUpload.tsx:41)
For more details, see the API Reference.

​
Next Steps

Query Your Data

Learn how to write SQL queries against your uploaded data

AI Agents

Use AI agents to analyze and query your data automatically

Build docs developers (and LLMs) love

Get started for freeTalk to us