Web Content

pip install beautifulsoup4

pip install beautifulsoup4 defusedxml

# Tech Blog
Latest technology news and tutorials

## New Python 3.12 Features
Published on: Mon, 15 Feb 2024 10:00:00 GMT
Python 3.12 introduces several exciting new features including...

## Understanding Async Programming  
Published on: Sun, 14 Feb 2024 15:30:00 GMT
Asynchronous programming can be challenging...

# Personal Blog
Thoughts and tutorials

## My Journey Learning Rust
Updated on: 2024-02-15T10:00:00Z
Rust has been an interesting language to learn...

## Web Development Tips
Updated on: 2024-02-14T14:00:00Z  
Here are some tips for modern web development...

pip install beautifulsoup4

# Python (programming language)

Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability with the use of significant indentation.

## History

Python was conceived in the late 1980s by Guido van Rossum...

## Features and philosophy

Python is a multi-paradigm programming language...

pip install beautifulsoup4 youtube-transcript-api

# YouTube

## Learn Python in 10 Minutes

### Video Metadata
- **Views:** 1,234,567
- **Keywords:** python, programming, tutorial
- **Runtime:** PT10M30S

### Description
In this video, we'll cover the basics of Python programming including variables, loops, and functions. Perfect for beginners!

### Transcript
Welcome to this Python tutorial. Today we're going to learn the basics of Python programming. Let's start with variables. A variable is a container for storing data values...

# Specify preferred languages (in order of preference)
result = md.convert_stream(
    stream=response,
    url=youtube_url,
    youtube_transcript_languages=["es", "fr", "en"]
)

pip install beautifulsoup4

import requests
import io
from markitdown import MarkItDown

md = MarkItDown()
query = "python programming"
url = f"https://www.bing.com/search?q={query}"

response = requests.get(url)
result = md.convert_stream(
    stream=io.BytesIO(response.content),
    url=url
)

print(result.markdown)

## A Bing search for 'python programming' found the following results:

[**Python.org**](https://www.python.org/)
Welcome to Python.org. The official home of the Python Programming Language.

[**Learn Python - Free Interactive Python Tutorial**](https://www.learnpython.org/)
Learn Python, one of today's most in-demand programming languages...

[**Python Tutorial - W3Schools**](https://www.w3schools.com/python/)
Well organized and easy to understand Web building tutorials...

import requests
import io
from markitdown import MarkItDown

def convert_url(url: str) -> str:
    """Fetch and convert any web URL to Markdown."""
    md = MarkItDown()
    
    response = requests.get(url, headers={
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
    })
    response.raise_for_status()
    
    result = md.convert_stream(
        stream=io.BytesIO(response.content),
        url=url
    )
    
    return result.markdown

# Usage
markdown = convert_url("https://en.wikipedia.org/wiki/Markdown")
print(markdown)

import requests
import io
from markitdown import MarkItDown
from urllib.parse import urlparse

md = MarkItDown()
urls = [
    "https://en.wikipedia.org/wiki/Python_(programming_language)",
    "https://www.youtube.com/watch?v=example",
    "https://example.com/feed.xml"
]

for url in urls:
    response = requests.get(url)
    result = md.convert_stream(
        stream=io.BytesIO(response.content),
        url=url
    )
    
    # Save with domain as filename
    domain = urlparse(url).netloc.replace('www.', '')
    filename = f"{domain}.md"
    
    with open(filename, 'w') as f:
        f.write(f"# {result.title or 'Untitled'}\n\n")
        f.write(f"Source: {url}\n\n")
        f.write(result.markdown)
    
    print(f"Saved {filename}")

import requests
import io
from markitdown import MarkItDown
from markitdown._exceptions import MissingDependencyException

md = MarkItDown()

try:
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    
    result = md.convert_stream(
        stream=io.BytesIO(response.content),
        url=url
    )
    print(result.markdown)
    
except requests.RequestException as e:
    print(f"Failed to fetch URL: {e}")
except MissingDependencyException as e:
    print(f"Missing dependency: {e}")
    print("Install with: pip install beautifulsoup4")
except Exception as e:
    print(f"Conversion error: {e}")

packages/markitdown/src/markitdown/converters/
├── _html_converter.py       # Generic HTML
├── _rss_converter.py        # RSS/Atom feeds
├── _wikipedia_converter.py  # Wikipedia articles
├── _youtube_converter.py    # YouTube videos
├── _bing_serp_converter.py  # Bing search results
└── _markdownify.py          # HTML to Markdown conversion