Overview
Azure OpenAI Service provides the same powerful OpenAI models (GPT-4o, GPT-4o-mini) through Microsoft’s Azure cloud platform, offering:
- Enterprise Security: Azure’s compliance and security features
- Private Network: Deploy within your Azure virtual network
- SLA Guarantees: 99.9% uptime commitment
- Data Residency: Keep data within specific regions
- Cost Management: Azure billing and cost controls
Perfect for enterprise deployments requiring compliance, security, and governance.
Prerequisites
Create Azure OpenAI Resource
- Go to Azure Portal
- Search for “Azure OpenAI”
- Click “Create”
- Select subscription, resource group, and region
- Wait for deployment to complete
Deploy a Model
- Go to your Azure OpenAI resource
- Navigate to “Model deployments”
- Click “Create new deployment”
- Select model (e.g.,
gpt-4o-mini)
- Give it a deployment name (e.g.,
gpt-4o-mini-deployment)
Get Credentials
From Azure OpenAI resource overview:
- Endpoint:
https://YOUR-RESOURCE-NAME.openai.azure.com/
- API Key: Found in “Keys and Endpoint” section
- API Version: e.g.,
2024-02-15-preview
Install ScrapeGraphAI
pip install scrapegraphai
playwright install
Basic Configuration
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
load_dotenv()
graph_config = {
"llm": {
"api_key": os.getenv("AZURE_OPENAI_API_KEY"),
"model": "azure_openai/gpt-4o-mini",
"api_version": "2024-02-15-preview",
"azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"),
"deployment_name": "gpt-4o-mini-deployment",
},
"verbose": True,
"headless": False,
}
smart_scraper_graph = SmartScraperGraph(
prompt="Extract the main article content",
source="https://www.wired.com",
config=graph_config,
)
result = smart_scraper_graph.run()
print(result)
Environment Variables
Create a .env file:
AZURE_OPENAI_API_KEY=your-azure-api-key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_DEPLOYMENT=gpt-4o-mini-deployment
AZURE_OPENAI_API_VERSION=2024-02-15-preview
import os
from dotenv import load_dotenv
load_dotenv()
graph_config = {
"llm": {
"api_key": os.getenv("AZURE_OPENAI_API_KEY"),
"model": "azure_openai/gpt-4o-mini",
"api_version": os.getenv("AZURE_OPENAI_API_VERSION"),
"azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"),
"deployment_name": os.getenv("AZURE_OPENAI_DEPLOYMENT"),
},
}
Always use environment variables for credentials - never hardcode them in your source code.
Available Models
GPT-4o Mini (Best Value)
"llm": {
"api_key": os.getenv("AZURE_OPENAI_API_KEY"),
"model": "azure_openai/gpt-4o-mini",
"api_version": "2024-02-15-preview",
"azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"),
"deployment_name": "gpt-4o-mini-deployment",
}
- Context: 128K tokens
- Best for: Most scraping tasks
- Deployment: Available in most regions
GPT-4o (Best Quality)
"llm": {
"api_key": os.getenv("AZURE_OPENAI_API_KEY"),
"model": "azure_openai/gpt-4o",
"api_version": "2024-02-15-preview",
"azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"),
"deployment_name": "gpt-4o-deployment",
}
- Context: 128K tokens
- Best for: Complex scraping tasks
Models available on Azure OpenAI:| Model | Context Window | Best For |
|---|
gpt-4o | 128K | Best accuracy |
gpt-4o-mini | 128K | Best value |
gpt-4-turbo | 128K | Vision tasks |
gpt-4 | 8K | Legacy support |
gpt-3.5-turbo | 16K | Cost-effective |
Configuration Options
Required Parameters
graph_config = {
"llm": {
# Required
"api_key": "your-azure-api-key",
"model": "azure_openai/gpt-4o-mini",
"azure_endpoint": "https://your-resource.openai.azure.com/",
"deployment_name": "your-deployment-name",
"api_version": "2024-02-15-preview",
},
}
The deployment_name is the name you gave when deploying the model in Azure Portal, NOT the model name itself.
Optional Parameters
graph_config = {
"llm": {
"api_key": os.getenv("AZURE_OPENAI_API_KEY"),
"model": "azure_openai/gpt-4o-mini",
"azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"),
"deployment_name": "gpt-4o-mini",
"api_version": "2024-02-15-preview",
# Optional
"temperature": 0, # Deterministic output
"max_tokens": 4000, # Limit response length
"top_p": 1.0, # Nucleus sampling
"frequency_penalty": 0, # Reduce repetition
"presence_penalty": 0, # Encourage topic diversity
},
}
Authentication Methods
API Key (Recommended)
Azure Active Directory
Managed Identity
Use API key authentication (simplest method):graph_config = {
"llm": {
"api_key": os.getenv("AZURE_OPENAI_API_KEY"),
"model": "azure_openai/gpt-4o-mini",
"azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"),
"deployment_name": "gpt-4o-mini",
"api_version": "2024-02-15-preview",
},
}
Use Azure AD authentication for enhanced security:from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
token = credential.get_token("https://cognitiveservices.azure.com/.default")
graph_config = {
"llm": {
"api_key": token.token, # Use token instead of API key
"model": "azure_openai/gpt-4o-mini",
"azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"),
"deployment_name": "gpt-4o-mini",
"api_version": "2024-02-15-preview",
},
}
Requires azure-identity package: pip install azure-identity
For Azure VMs and services with managed identity:from azure.identity import ManagedIdentityCredential
credential = ManagedIdentityCredential()
token = credential.get_token("https://cognitiveservices.azure.com/.default")
graph_config = {
"llm": {
"api_key": token.token,
"model": "azure_openai/gpt-4o-mini",
"azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"),
"deployment_name": "gpt-4o-mini",
"api_version": "2024-02-15-preview",
},
}
Complete Examples
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
load_dotenv()
graph_config = {
"llm": {
"api_key": os.getenv("AZURE_OPENAI_API_KEY"),
"model": "azure_openai/gpt-4o-mini",
"api_version": "2024-02-15-preview",
"azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"),
"deployment_name": os.getenv("AZURE_OPENAI_DEPLOYMENT"),
"temperature": 0,
},
"verbose": True,
"headless": False,
}
smart_scraper = SmartScraperGraph(
prompt="Extract the main article with title and author",
source="https://www.wired.com",
config=graph_config,
)
result = smart_scraper.run()
print(json.dumps(result, indent=4))
graph_exec_info = smart_scraper.get_execution_info()
print(prettify_exec_info(graph_exec_info))
Troubleshooting
Error: DeploymentNotFound: The API deployment for this resource does not existSolution:
- Check deployment name in Azure Portal
- Ensure it matches exactly (case-sensitive)
- Verify deployment is in “Succeeded” state
# Correct
"deployment_name": "gpt-4o-mini-deployment" # As shown in Portal
# Wrong
"deployment_name": "gpt-4o-mini" # Model name, not deployment name
Error: InvalidApiVersion: The API version is not supportedSolution: Use a valid API version:"api_version": "2024-02-15-preview" # Current stable version
Error: RateLimitError: Requests to the API are being rate limitedSolution:
- Increase quota in Azure Portal
- Add retry logic with backoff
- Distribute load across multiple deployments
import time
from tenacity import retry, wait_exponential, stop_after_attempt
@retry(wait=wait_exponential(min=1, max=60), stop=stop_after_attempt(5))
def scrape_with_retry(url):
return scraper.run()
Error: AuthenticationError: Access deniedSolution:
- Verify API key is correct
- Check key hasn’t expired
- Ensure resource is not paused/deleted
- Verify network access (if using Private Endpoint)
Best Practices
Use Key Vault
Store credentials in Azure Key Vault:from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential
client = SecretClient(
vault_url="https://myvault.vault.azure.net/",
credential=DefaultAzureCredential()
)
api_key = client.get_secret("azure-openai-key").value
Multi-Region Deployment
Deploy to multiple regions for high availability:
- Primary: East US
- Failover: West Europe
- Backup: Southeast Asia
Monitor Usage
Use Azure Monitor to track:
- Token usage
- Request latency
- Error rates
- Cost trends
Cost Management
Implement cost controls:
- Set budget alerts
- Use gpt-4o-mini for most tasks
- Cache common responses
- Monitor token usage
Regional Considerations
Model availability varies by Azure region. Popular regions for Azure OpenAI:
- East US: Best availability, all models
- West Europe: GDPR compliance
- UK South: UK data residency
- Australia East: APAC customers
- Canada Central: Canadian data residency
Next Steps
Advanced Configuration
Learn about proxy rotation and browser settings
OpenAI
Compare with standard OpenAI API
