Documentation Scraping

User: Convert this to a Next.js Server Action using https://nextjs.org/docs/app/building-your-application/data-fetching/server-actions

AI: *scrapes Next.js docs*
    *learns Server Action pattern*
    *generates compliant code*

User: Add Stripe checkout following https://docs.stripe.com/payments/checkout/how-checkout-works

AI: *scrapes Stripe docs*
    *understands the flow*
    *creates checkout with correct parameters*

User: Use Zod validation based on https://zod.dev/?id=basic-usage

AI: *scrapes Zod documentation*
    *applies correct schema syntax*
    *generates type-safe validation*

User: Create a button using https://ui.shadcn.com/docs/components/button

AI: *scrapes shadcn/ui docs*
    *follows component patterns*
    *includes proper imports and variants*

// Selected code:
const response = await fetch('/api/data');

// Instruction:
// "add error handling using https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#checking_that_the_fetch_was_successful"

// Result (uses MDN patterns):
const response = await fetch('/api/data');
if (!response.ok) {
  throw new Error(`HTTP error! status: ${response.status}`);
}
const data = await response.json();

const URL_REGEX = /https?:\/\/[^\s)>\]]+/g;
const urls: string[] = instruction.match(URL_REGEX) || [];

if (urls.length > 0) {
  const scrapedResults = await Promise.all(
    urls.map(async (url) => {
      const result = await firecrawl.scrape(url, {
        formats: ["markdown"],
      });
      return result.markdown;
    })
  );
  // Inject into AI prompt
}

scrapeUrls({
  urls: [
    "https://docs.stripe.com/api/checkout/sessions",
    "https://docs.stripe.com/webhooks"
  ]
})

User: Build a subscription system using Stripe. Reference:
- https://docs.stripe.com/billing/subscriptions/overview
- https://docs.stripe.com/billing/subscriptions/build-subscriptions

AI:
1. Scrapes both Stripe documentation URLs
2. Reads your current project structure
3. Creates subscription management files following Stripe patterns
4. Implements webhook handlers for subscription events
5. Responds with setup instructions

const URL_REGEX = /https?:\/\/[^\s)>\]]+/g;

const urls: string[] = instruction.match(URL_REGEX) || [];
let documentationContext = "";

if (urls.length > 0) {
  const scrapedResults = await Promise.all(
    urls.map(async (url) => {
      try {
        const result = await firecrawl.scrape(url, {
          formats: ["markdown"],
        });
        if (result.markdown) {
          return `<doc url="${url}">\n${result.markdown}\n</doc>`;
        }
        return null;
      } catch {
        return null;
      }
    })
  );
  
  const validResults = scrapedResults.filter(Boolean);
  if (validResults.length > 0) {
    documentationContext = `<documentation>\n${validResults.join("\n\n")}\n</documentation>`;
  }
}

export const createScrapeUrlsTool = () => {
  return createTool({
    name: "scrapeUrls",
    description:
      "Scrape content from URLs to get documentation or reference material. " +
      "Use this when the user provides URLs or references external documentation. " +
      "Returns markdown content from the scraped pages.",
    parameters: z.object({
      urls: z.array(z.string()).describe("Array of URLs to scrape for content"),
    }),
    handler: async (params, { step: toolStep }) => {
      const results: { url: string; content: string }[] = [];
      
      for (const url of urls) {
        const result = await firecrawl.scrape(url, {
          formats: ["markdown"],
        });
        if (result.markdown) {
          results.push({ url, content: result.markdown });
        }
      }
      
      return JSON.stringify(results);
    }
  });
};

import Firecrawl from '@mendable/firecrawl-js';

export const firecrawl = new Firecrawl({
  apiKey: process.env.FIRECRAWL_API_KEY
});

"add Zod validation following https://zod.dev"

"use React Query patterns from https://tanstack.com/query/latest/docs/framework/react/overview"

"implement auth with https://next-auth.js.org/getting-started/example"

"style with Tailwind using https://tailwindcss.com/docs/utility-first"

"Build a file upload component using https://uploadthing.com/docs"

"Create API routes following https://nextjs.org/docs/app/api-reference/file-conventions/route"

"Add database queries using https://orm.drizzle.team/docs/sql-schema-declaration"

"Implement real-time features with https://docs.convex.dev/functions"