Global Resource Pool

Overview

The Global Resource Pool is a core architectural pattern in Study Sync that prevents duplication when the same resource (YouTube video, PDF, article, etc.) is used across multiple study plans.

The Problem

Without de-duplication:

Same YouTube video added to 10 different study plans → 10 duplicate database entries
Wasted storage space
Inconsistent metadata (e.g., video title updated in one plan but not others)
Difficulty tracking which resources are most popular

The Solution: URL-Based De-duplication

Architecture

┌─────────────────┐
│  Study Plan A   │────┐
└─────────────────┘    │
                       │       ┌──────────────────────┐
┌─────────────────┐    ├──────>│  Global Resources    │
│  Study Plan B   │────┤       │  Collection          │
└─────────────────┘    │       │                      │
                       │       │  - Unique by URL     │
┌─────────────────┐    │       │  - Shared metadata   │
│  Study Plan C   │────┘       │  - Referenced by ID  │
└─────────────────┘            └──────────────────────┘

How It Works

User adds a resource to their study plan
System checks if a resource with that URL already exists
If exists: Return the existing resource ObjectId
If new: Create resource, then return its ObjectId
Study plan stores only the resource ObjectId in its resourceIds array

Implementation

Database Schema

Collection: resources Location: src/lib/db.js:54

{
  _id: ObjectId,
  url: String,        // UNIQUE INDEX - de-duplication key
  title: String,
  type: String,       // "youtube-video", "pdf", "article", etc.
  metadata: Object,   // Type-specific metadata
  createdAt: Date,
  updatedAt: Date
}

Unique Index:

await collections.resources.createIndex({ url: 1 }, { unique: true });

This ensures MongoDB prevents duplicate URLs at the database level.

API Endpoint: Create or Get Resource

Endpoint: POST /api/resources Location: src/app/api/resources/route.js:84

Flow Diagram

Code Implementation

Single Resource (src/app/api/resources/route.js:238)

if (resourcesToCreate.length === 1) {
  // Check if resource already exists
  const existingResource = await resources.findOne({
    url: resourcesToCreate[0].url,
  });

  if (existingResource) {
    // Resource already exists, return it
    return createSuccessResponse({
      message: "Resource already exists",
      resource: existingResource,
      isNew: false,
    }, 200);
  }

  // Create new resource
  const result = await resources.insertOne(resourcesToCreate[0]);
  return createSuccessResponse({
    message: "Resource created successfully",
    resource: { ...resourcesToCreate[0], _id: result.insertedId },
    isNew: true,
  }, 201);
}

Multiple Resources (Playlist) (src/app/api/resources/route.js:268)

// Check which resources already exist
const urls = resourcesToCreate.map((r) => r.url);
const existingResources = await resources
  .find({ url: { $in: urls } })
  .toArray();
const existingUrlMap = new Map(existingResources.map((r) => [r.url, r]));

// Separate new and existing resources
const newResources = [];
const finalResources = [];

for (const resource of resourcesToCreate) {
  const existing = existingUrlMap.get(resource.url);
  if (existing) {
    finalResources.push(existing);  // Use existing
  } else {
    newResources.push(resource);    // Mark for insertion
  }
}

// Insert only new resources
if (newResources.length > 0) {
  const result = await resources.insertMany(newResources);
  const insertedResources = newResources.map((r, i) => ({
    ...r,
    _id: result.insertedIds[i],
  }));
  finalResources.push(...insertedResources);
}

return createSuccessResponse({
  message: `${newResources.length} new resource(s) created, ${existingResources.length} already existed`,
  resources: finalResources,
  newCount: newResources.length,
  existingCount: existingResources.length,
}, 201);

Resource Types

1. YouTube Videos

Type: youtube-video Metadata Fetching: src/lib/youtube.js Process:

Extract video ID from URL
Call YouTube Data API v3 to fetch:
- Title
- Duration (in seconds)
- Thumbnail URL
Store metadata in resource.metadata

Example:

{
  _id: ObjectId("..."),
  url: "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
  title: "Introduction to Algorithms",
  type: "youtube-video",
  metadata: {
    duration: 3600,
    videoId: "dQw4w9WgXcQ",
    thumbnail: "https://i.ytimg.com/vi/dQw4w9WgXcQ/maxresdefault.jpg"
  }
}

Implementation: src/app/api/resources/route.js:119

if (type === "youtube-video" && url) {
  const ytMetadata = await getVideoMetadata(url);
  finalMetadata = { ...ytMetadata };
  finalTitle = ytMetadata.title || finalTitle;
  
  resourcesToCreate.push({
    type,
    title: finalTitle,
    url,
    metadata: finalMetadata,
    addedBy: auth.user._id,
    createdAt: new Date(),
    updatedAt: new Date(),
  });
}

2. YouTube Playlists

Type: youtube-playlist (expanded to individual videos) Process:

Extract playlist ID from URL
Fetch all videos in playlist via YouTube API
Create individual youtube-video resources for each video
De-duplicate: skip videos already in database

Implementation: src/app/api/resources/route.js:141

else if (type === "youtube-playlist" && url) {
  const playlistId = url.match(/[?&]list=([^&]+)/)?.[1];
  const videos = await getPlaylistVideos(url);

  // Create individual resources for each video
  resourcesToCreate = videos.map((video) => ({
    type: "youtube-video",
    title: video.title,
    url: video.url,
    metadata: {
      duration: video.duration,
      videoId: video.videoId,
      thumbnail: video.thumbnailUrl,
    },
    addedBy: auth.user._id,
    createdAt: new Date(),
    updatedAt: new Date(),
  }));
}

3. PDF Documents

Type: pdf Metadata (user-provided):

pages: Number of pages
minsPerPage: Estimated minutes per page (default: 3)

Example:

{
  _id: ObjectId("..."),
  url: "https://example.com/textbook.pdf",
  title: "Data Structures Textbook - Chapter 5",
  type: "pdf",
  metadata: {
    pages: 42,
    minsPerPage: 4
  }
}

Total Time Calculation: 42 pages × 4 mins = 168 minutes Implementation: src/app/api/resources/route.js:172

else if (type === "pdf") {
  if (!title || !pages) {
    return createErrorResponse("Title and pages are required for PDF", 400);
  }

  finalMetadata = {
    pages: parseInt(pages),
    minsPerPage: parseInt(minsPerPage) || 3,
  };

  resourcesToCreate.push({ type, title, url, metadata: finalMetadata, ... });
}

4. Articles

Type: article Metadata:

estimatedMins: User-estimated reading time

Example:

{
  _id: ObjectId("..."),
  url: "https://blog.example.com/understanding-binary-trees",
  title: "Understanding Binary Trees",
  type: "article",
  metadata: {
    estimatedMins: 15
  }
}

5. Google Drive & Custom Links

Types: google-drive, custom-link Metadata: Optional estimatedMins Implementation: src/app/api/resources/route.js:212

else if (type === "google-drive" || type === "custom-link") {
  const metadata = {};
  if (estimatedMins) {
    metadata.estimatedMins = parseInt(estimatedMins);
  }
  resourcesToCreate.push({ type, title: title || url, url, metadata, ... });
}

Study Plan References

Storing Resource IDs

Collection: studyplans Field: resourceIds: [ObjectId] Location: src/lib/db.js:97

studyPlan: (data) => ({
  title: data.title,
  resourceIds: (data.resourceIds || [])
    .map((id) => toObjectId(id))
    .filter(Boolean),
  // ... other fields
})

Adding Resources to Plan

When a user adds a resource:

Create/get resource via POST /api/resources
Update study plan via PUT /api/study-plans/:id
Append resource ID to plan.resourceIds

Client-side flow:

// 1. Create or get resource
const { resource } = await createOrGetResource({ url, type, ... }, token);

// 2. Add to study plan
await updateStudyPlan(planId, {
  resourceIds: [...existingResourceIds, resource._id]
}, token);

Fetching Plan Resources

API Endpoint: GET /api/study-plans/:id Query:

const plan = await studyPlans.findOne({ _id: planId });
const planResources = await resources
  .find({ _id: { $in: plan.resourceIds } })
  .toArray();

Result: Full resource objects with metadata Sorting: Resources are returned in the order specified by plan.resourceIds Implementation: src/app/api/resources/route.js:59

switch (sortBy) {
  case "order":
  default:
    // Maintain the order from resourceIds array
    const orderMap = new Map(
      plan.resourceIds.map((id, index) => [id.toString(), index])
    );
    sortedResources.sort((a, b) => {
      const orderA = orderMap.get(a._id.toString()) ?? 999999;
      const orderB = orderMap.get(b._id.toString()) ?? 999999;
      return orderA - orderB;
    });
    break;
}

Benefits of Global Resource Pool

1. Storage Efficiency

Before:

Plan A: { resources: [{ url: "video1", title: "Intro" }] }
Plan B: { resources: [{ url: "video1", title: "Intro" }] }  // DUPLICATE
Plan C: { resources: [{ url: "video1", title: "Intro" }] }  // DUPLICATE

After:

Resources: [{ _id: "r1", url: "video1", title: "Intro" }]  // Stored once
Plan A: { resourceIds: ["r1"] }
Plan B: { resourceIds: ["r1"] }
Plan C: { resourceIds: ["r1"] }

2. Metadata Consistency

If YouTube video title changes, one database update affects all study plans.

3. Analytics Potential

Find most popular resources (count references)
Track which resources appear in successful study plans
Recommend resources based on patterns

Example Query:

// Find most-referenced resources
const popularResources = await studyPlans.aggregate([
  { $unwind: "$resourceIds" },
  { $group: { _id: "$resourceIds", count: { $sum: 1 } } },
  { $sort: { count: -1 } },
  { $limit: 10 }
]);

4. Bulk Operations

When importing a playlist with 50 videos:

Check all 50 URLs against database in one query
Insert only new videos (maybe 10 are already in DB)
Save API quota and processing time

Edge Cases

1. Same Video, Different URLs

Problem: youtube.com/watch?v=abc vs youtu.be/abc are the same video but different URLs. Solution: Normalize URLs before storage (future enhancement).

2. URL Changes

Problem: Video moves to a different URL. Current: Creates a new resource. Old resource remains orphaned. Future: Implement URL redirect detection or manual merge tools.

3. Deleted Resources

Problem: YouTube video is deleted, but resource still in DB. Current: Resource remains, but video is inaccessible. Future: Periodic validation job to check resource availability.

Duration Calculation

Helper Function: src/lib/db.js:156

export function getResourceTotalTime(resource) {
  if (resource.type === "youtube-video") {
    return resource.metadata?.duration || 0;  // seconds
  } else if (resource.type === "pdf") {
    return (resource.metadata?.pages || 0) * (resource.metadata?.minsPerPage || 0);
  } else if (resource.type === "article" || resource.type === "google-drive" || resource.type === "custom-link") {
    return resource.metadata?.estimatedMins || 0;
  }
  return 0;
}

Usage: Calculate total study plan duration

const totalTime = planResources.reduce((sum, resource) => {
  return sum + getResourceTotalTime(resource);
}, 0);

Future Enhancements

1. Resource Versioning

Track when resource metadata changes:

{
  _id: ObjectId,
  url: String,
  currentVersion: ObjectId,
  versions: [
    { title: "Old Title", updatedAt: Date },
    { title: "New Title", updatedAt: Date }
  ]
}

2. Resource Tagging

{
  url: "...",
  tags: ["algorithms", "sorting", "beginner"],
  difficulty: "intermediate"
}

3. User-Contributed Metadata

Allow users to suggest better titles or durations:

{
  url: "...",
  suggestedMetadata: [
    { userId: ObjectId, title: "Better Title", votes: 12 }
  ]
}

4. Smart Recommendations

// Users who used resource A also used:
const recommendations = await studyPlans.aggregate([
  { $match: { resourceIds: resourceA } },
  { $unwind: "$resourceIds" },
  { $group: { _id: "$resourceIds", count: { $sum: 1 } } },
  { $sort: { count: -1 } }
]);

Summary

Key Principles:

Resources are stored once in a global pool
URL uniqueness prevents duplicates
Study plans reference resources by ObjectId
Automatic de-duplication on creation
Metadata consistency across all plans

Files to Review:

src/app/api/resources/route.js - Resource creation endpoint
src/lib/db.js - Schema definitions and helpers
src/lib/youtube.js - YouTube API integration

Get Started

Core Features

User Guides

Configuration

Architecture

​Overview

​The Problem

​The Solution: URL-Based De-duplication

​Architecture

​How It Works

​Implementation

​Database Schema

​API Endpoint: Create or Get Resource

​Flow Diagram

​Code Implementation

​Resource Types

​1. YouTube Videos

​2. YouTube Playlists

​3. PDF Documents

​4. Articles

​5. Google Drive & Custom Links

​Study Plan References

​Storing Resource IDs

​Adding Resources to Plan

​Fetching Plan Resources

​Benefits of Global Resource Pool

​1. Storage Efficiency

​2. Metadata Consistency

​3. Analytics Potential

​4. Bulk Operations

​Edge Cases

​1. Same Video, Different URLs

​2. URL Changes

​3. Deleted Resources

​Duration Calculation

​Future Enhancements

​1. Resource Versioning

​2. Resource Tagging

​3. User-Contributed Metadata

​4. Smart Recommendations

​Summary

Build docs developers (and LLMs) love

Overview

The Problem

The Solution: URL-Based De-duplication

Architecture

How It Works

Implementation

Database Schema

API Endpoint: Create or Get Resource

Flow Diagram

Code Implementation

Resource Types

1. YouTube Videos

2. YouTube Playlists

3. PDF Documents

4. Articles

5. Google Drive & Custom Links

Study Plan References

Storing Resource IDs

Adding Resources to Plan

Fetching Plan Resources

Benefits of Global Resource Pool

1. Storage Efficiency

2. Metadata Consistency

3. Analytics Potential

4. Bulk Operations

Edge Cases

1. Same Video, Different URLs

2. URL Changes

3. Deleted Resources

Duration Calculation

Future Enhancements

1. Resource Versioning

2. Resource Tagging

3. User-Contributed Metadata

4. Smart Recommendations

Summary