Data Sources - ChartsMaze EDL Pipeline

Overview

The EDL Pipeline integrates data from 12+ endpoints across Dhan’s market analytics platform and NSE archives. This page documents the exact URLs, request payloads, response structures, and rate limits for each endpoint.

1. Full Market Data

Endpoint Details

Property	Value
Script	`fetch_dhan_data.py`
URL	`https://ow-scanx-analytics.dhan.co/customscan/fetchdt`
Method	`POST`
Page Size	`count: 5000` (returns all ~2,775 in one call)
Output	`dhan_data_response.json`, `master_isin_map.json`

Request Payload

{
  "data": {
    "type": "full",
    "whichpage": "nse_total_market",
    "filters": [],
    "sort": "Mcap",
    "sorder": "desc",
    "count": 5000,
    "page": 1
  }
}

Response Structure

[
  {
    "Sym": "RELIANCE",
    "Ltp": 2456.75,
    "PPerchange": 1.23,
    "Mcap": 1658234.5,
    "High1Yr": 2800.0,
    "DaySMA200CurrentCandle": 2400.0,
    "DaySMA50CurrentCandle": 2480.0,
    "DayRSI14CurrentCandle": 62.5,
    "PricePerchng1week": 2.1,
    "PricePerchng1mon": -3.5,
    "PricePerchng3mon": 8.2,
    "PricePerchng1year": 15.6,
    "idxlist": [
      {"Indexid": 13, "Name": "NIFTY 50"},
      {"Indexid": 51, "Name": "NIFTY 100"}
    ],
    "Isin": "INE002A01018",
    "Sid": "1333"
  }
]

Key Fields

Sym: Stock symbol (e.g., “RELIANCE”)
Isin: ISIN identifier for regulatory filings
Sid: Security ID required for OHLCV and advanced indicators
Ltp: Last Traded Price
Mcap: Market Cap in Crores
idxlist: Array of index memberships

This endpoint produces master_isin_map.json, which is required by ALL Phase 2 scripts. If this script fails, the entire pipeline cannot continue.

2. Fundamental Data (Results & Ratios)

Endpoint Details

Property	Value
Script	`fetch_fundamental_data.py`
URL	`https://open-web-scanx.dhan.co/scanx/fundamental`
Method	`POST`
Pagination	Per-ISIN (iterates `master_isin_map.json`)
Timeout	30s per request
Output	`fundamental_data.json` (35 MB)

Request Payload

{
  "data": {
    "isin": "INE002A01018"
  }
}

Response Structure

{
  "Symbol": "RELIANCE",
  "Name": "Reliance Industries Ltd",
  "incomeStat_cq": {
    "YEAR": "Q3-FY24|Q2-FY24|Q1-FY24|Q4-FY23|Q3-FY23",
    "NET_PROFIT": "18500|17200|16800|15900|17800",
    "EPS": "27.5|25.6|25.0|23.7|26.5",
    "SALES": "234000|228000|225000|220000|232000",
    "OPM": "12.5|11.8|12.1|11.5|12.3"
  },
  "incomeStat_cy": {
    "YEAR": "FY23|FY22|FY21|FY20|FY19",
    "EPS": "98.5|92.3|85.6|78.2|72.1",
    "SALES": "900000|850000|780000|720000|650000"
  },
  "TTM_cy": {
    "OPM": 12.2,
    "EPS": 102.3
  },
  "CV": {
    "INDUSTRY_NAME": "Refineries",
    "SECTOR": "Energy",
    "MARKET_CAP": 1658234.5,
    "STOCK_PE": 24.5
  },
  "roce_roe": {
    "ROE": 14.8,
    "ROCE": 16.2
  },
  "sHp": {
    "FII": "22.5|21.8",
    "DII": "15.3|15.1",
    "PROMOTER": "50.4|50.4"
  },
  "bs_c": {
    "NON_CURRENT_LIABILITIES": "150000|145000",
    "TOTAL_EQUITY": "520000|510000"
  }
}

Data Sections

incomeStat_cq: Quarterly income statement (pipe-separated, last 5 quarters)
incomeStat_cy: Annual income statement (pipe-separated, last 5 years)
TTM_cy: Trailing Twelve Months metrics
CV: Company valuation and classification
roce_roe: Return ratios
sHp: Shareholding pattern (latest 2 quarters)
bs_c: Balance sheet current period

3. Company Filings (Hybrid)

Endpoint Details

Property	Value
Script	`fetch_company_filings.py`
URL 1	`https://ow-static-scanx.dhan.co/staticscanx/company_filings`
URL 2	`https://ow-static-scanx.dhan.co/staticscanx/lodr`
Method	`POST`
Page Size	`count: 100, pg_no: 1`
Threads	20
Deduplication	By `news_id` + `news_date` + `caption`
Output	`company_filings/{SYMBOL}_filings.json`

Why Two Endpoints?

The script fetches from both endpoints and merges results for maximum coverage:

URL 1 (company_filings): Legacy filings, broader coverage
URL 2 (lodr): LODR (Listing Obligations and Disclosure Requirements) filings

Request Payload

{
  "data": {
    "isin": "INE002A01018",
    "pg_no": 1,
    "count": 100
  }
}

Response Structure

[
  {
    "news_id": "123456",
    "news_date": "03-Mar-2024",
    "caption": "Outcome of Board Meeting - Quarterly Results",
    "filing_type": "SEBI Reg 30",
    "pdf_url": "https://...",
    "category": "Results"
  },
  {
    "news_id": "123455",
    "news_date": "28-Feb-2024",
    "caption": "Insider Trading - SEBI Reg 7(2)",
    "filing_type": "SEBI Reg 7(2)",
    "pdf_url": "https://...",
    "category": "Insider Trading"
  }
]

Deduplication Logic

Filings are deduplicated using a composite key:

key = f"{news_id}_{news_date}_{caption}"

4. Live Announcements

Endpoint Details

Property	Value
Script	`fetch_new_announcements.py`
URL	`https://ow-static-scanx.dhan.co/staticscanx/announcements`
Method	`POST`
Threads	40
Output	`all_company_announcements.json`

Request Payload

{
  "data": {
    "isin": "INE002A01018"
  }
}

Response Structure

[
  {
    "announcement_date": "03-Mar-2024",
    "subject": "Board Meeting Intimation",
    "description": "Notice of Board Meeting to consider Quarterly Results",
    "exchange": "NSE",
    "attachment_url": "https://..."
  }
]

5. Advanced Indicators (Pivot, EMA, SMA)

Endpoint Details

Property	Value
Script	`fetch_advanced_indicators.py`
URL	`https://ow-static-scanx.dhan.co/staticscanx/indicator`
Method	`POST`
Threads	50
Requires	`Sid` (Security ID from `master_isin_map.json`)
Output	`advanced_indicator_data.json` (8.3 MB)

Request Payload

{
  "exchange": "NSE",
  "segment": "E",
  "security_id": "1333",
  "isin": "INE002A01018",
  "symbol": "RELIANCE",
  "minute": "D"
}

Response Structure

{
  "Symbol": "RELIANCE",
  "SMA": [
    {"Indicator": "20-SMA", "Value": 2480.5, "Signal": "Bullish"},
    {"Indicator": "50-SMA", "Value": 2450.0, "Signal": "Bullish"},
    {"Indicator": "200-SMA", "Value": 2400.0, "Signal": "Bullish"}
  ],
  "EMA": [
    {"Indicator": "20-EMA", "Value": 2485.0, "Signal": "Bullish"},
    {"Indicator": "50-EMA", "Value": 2460.0, "Signal": "Bullish"},
    {"Indicator": "200-EMA", "Value": 2410.0, "Signal": "Bullish"}
  ],
  "TechnicalIndicators": [
    {"Indicator": "RSI", "Action": "Neutral"},
    {"Indicator": "MACD", "Action": "Bullish"}
  ],
  "Pivots": [
    {
      "Classic": {
        "PP": 2456.0,
        "R1": 2480.0,
        "R2": 2510.0,
        "S1": 2430.0,
        "S2": 2400.0
      }
    }
  ]
}

The minute parameter supports:

D = Daily
60 = Hourly
15, 30 = Intraday intervals

Pipeline uses D (Daily) for consistency.

6. Market News Feed

Endpoint Details

Property	Value
Script	`fetch_market_news.py`
URL	`https://news-live.dhan.co/v2/news/getLiveNews`
Method	`POST`
Page Size	`limit: 50` (per stock)
Max Tested	`limit: 100` works, pagination via `page_no`
Threads	15
Output	`market_news/{SYMBOL}_news.json`

Request Payload

{
  "categories": ["ALL"],
  "page_no": 0,
  "limit": 50,
  "first_news_timeStamp": 0,
  "last_news_timeStamp": 0,
  "news_feed_type": "live",
  "stock_list": ["INE002A01018"],
  "entity_id": ""
}

Response Structure

[
  {
    "title": "Reliance Q3 Results Beat Estimates",
    "description": "Net profit rises 12% YoY to Rs 18,500 crore...",
    "sentiment": "positive",
    "published_date": "2024-03-03T14:30:00Z",
    "source": "Economic Times",
    "news_url": "https://..."
  },
  {
    "title": "Jio Subscriber Growth Slows in Q3",
    "description": "Jio adds 5.2 million subscribers, down from 8.1 million...",
    "sentiment": "neutral",
    "published_date": "2024-03-02T10:15:00Z",
    "source": "Bloomberg",
    "news_url": "https://..."
  }
]

Sentiment Analysis

API includes AI-generated sentiment:

positive — Bullish news
neutral — Factual reporting
negative — Bearish news

7. Corporate Actions

Endpoint Details

Property	Value
Script	`fetch_corporate_actions.py`
URL	`https://ow-scanx-analytics.dhan.co/customscan/fetchdt`
Method	`POST`
Page Size	`count: 5000`
Modes	History (2 years back) + Upcoming (2 months ahead)
Output	`history_corporate_actions.json`, `upcoming_corporate_actions.json`

Request Payload (Upcoming)

{
  "data": {
    "type": "full",
    "whichpage": "corporate_action",
    "filters": [
      {
        "field": "CorpAct.ActDate",
        "op": "GT",
        "val": "2024-03-01"
      }
    ],
    "count": 5000,
    "page": 1
  }
}

Request Payload (History)

{
  "data": {
    "type": "full",
    "whichpage": "corporate_action",
    "filters": [
      {
        "field": "CorpAct.ActDate",
        "op": "LT",
        "val": "2024-03-01"
      },
      {
        "field": "CorpAct.ActDate",
        "op": "GT",
        "val": "2022-03-01"
      }
    ],
    "count": 5000,
    "page": 1
  }
}

Response Structure

[
  {
    "Symbol": "RELIANCE",
    "ActionType": "Dividend",
    "ActDate": "2024-03-15",
    "ExDate": "2024-03-10",
    "RecordDate": "2024-03-11",
    "Details": "Rs 9 per share",
    "BCStartDate": "2024-03-12",
    "BCEndDate": "2024-03-15"
  },
  {
    "Symbol": "INFY",
    "ActionType": "Bonus",
    "ActDate": "2024-04-20",
    "Details": "1:1",
    "ExDate": "2024-04-15"
  }
]

Action Types

Dividend — Cash dividend
Bonus — Bonus issue (e.g., 1:1)
Split — Stock split (e.g., 1:5)
Rights — Rights issue
AGM/EGM — Shareholder meetings

8. Surveillance Lists (ASM/GSM)

Endpoint Details

Property	Value
Script	`fetch_surveillance_lists.py`
URL	Google Sheets Gviz endpoint (fallback: Dhan Next.js API)
Method	`GET`
Output	`nse_asm_list.json`, `nse_gsm_list.json`

ASM (Additional Surveillance Measure)

NSE places stocks in ASM framework for:

High price volatility
Abnormal trading volumes
Market manipulation concerns

GSM (Graded Surveillance Measure)

More stringent than ASM, includes:

Trading restrictions
Price bands
Settlement obligations

Response Structure

[
  {
    "symbol": "YESBANK",
    "series": "EQ",
    "surveillance_stage": "Stage 1 - ASM",
    "date_of_inclusion": "2024-02-15"
  }
]

9. Circuit Stocks

Endpoint Details

Property	Value
Script	`fetch_circuit_stocks.py`
URL	`https://ow-scanx-analytics.dhan.co/customscan/fetchdt`
Method	`POST`
Page Size	`count: 500`
Output	`upper_circuit_stocks.json`, `lower_circuit_stocks.json`

Request Payload (Upper Circuit)

{
  "data": {
    "type": "full",
    "whichpage": "circuit_filter",
    "filters": [
      {"field": "circuit", "op": "EQ", "val": "upper"}
    ],
    "count": 500,
    "page": 1
  }
}

10. Bulk/Block Deals

Endpoint Details

Property	Value
Script	`fetch_bulk_block_deals.py`
URL	`https://ow-static-scanx.dhan.co/staticscanx/deal`
Method	`POST`
Page Size	`pagecount: 50` (auto-paginates all pages)
Output	`bulk_block_deals.json`

Request Payload

{
  "data": {
    "defaultpage": "N",
    "pageno": 1,
    "pagecount": 50
  }
}

Response Structure

[
  {
    "symbol": "RELIANCE",
    "deal_date": "03-Mar-2024",
    "deal_type": "Bulk",
    "client_name": "XYZ Investments Ltd",
    "quantity": 1500000,
    "price": 2456.75,
    "buy_sell": "Buy"
  }
]

11. Price Bands

Incremental Price Bands

Property	Value
Script	`fetch_incremental_price_bands.py`
URL	`https://nsearchives.nseindia.com/content/equities/eq_band_changes_{date}.csv`
Method	`GET` (CSV download)
Output	`incremental_price_bands.json`

Complete Price Bands

Property	Value
Script	`fetch_complete_price_bands.py`
URL	`https://nsearchives.nseindia.com/content/equities/sec_list_{date}.csv`
Method	`GET` (CSV download)
Output	`complete_price_bands.json`

CSV Structure

SYMBOL,SERIES,DATE OF LISTING,FACE VALUE,APPLICABLE MARGIN,PRICE BAND
RELIANCE,EQ,29-Nov-1977,10,20%,20%
INFY,EQ,11-Feb-1993,5,20%,20%
TCS,EQ,25-Aug-2004,1,20%,20%

12. Historical OHLCV

Endpoint Details

Property	Value
Script	`fetch_all_ohlcv.py`
URL	`https://openweb-ticks.dhan.co/getDataH`
Method	`POST`
Threads	15
Start	`215634600` (Oct 31, 1976 — forces max history)
Interval	`D` (Daily candles)
Output	`ohlcv_data/{SYMBOL}.csv`

Request Payload

{
  "EXCH": "NSE",
  "SYM": "RELIANCE",
  "SEG": "E",
  "INST": "EQUITY",
  "SEC_ID": "1333",
  "EXPCODE": 0,
  "INTERVAL": "D",
  "START": 215634600,
  "END": 1709481600
}

Response CSV Structure

Date,Open,High,Low,Close,Volume
2024-03-01,2450.0,2480.5,2445.0,2456.75,12500000
2024-03-02,2460.0,2475.0,2440.0,2450.25,11800000

Smart Incremental Updates

The script automatically:

Checks existing CSV files
Reads the last date
Fetches only new candles from last_date + 1 to today
Appends to existing CSV

Result: ~2-5 min updates (vs ~30 min full download)

Rate Limits & Best Practices

Threading Guidelines

Endpoint	Max Threads	Notes
Advanced Indicators	50	Highest throughput
Announcements	40	Good stability
Company Filings	20	Two endpoints, be conservative
Market News	15	External API, lower limit
OHLCV	15	Large payloads, avoid overload

Timeout Recommendations

Fundamental Data: 30s (large JSON responses)
OHLCV: 30s (multi-year history)
News/Filings: 15s (moderate payloads)
All Others: 10s

Error Handling

try:
    response = requests.post(url, json=payload, timeout=30)
    response.raise_for_status()
    data = response.json()
except requests.Timeout:
    # Retry with exponential backoff
except requests.HTTPError as e:
    if e.response.status_code == 429:
        # Rate limit hit, sleep 60s
    elif e.response.status_code >= 500:
        # Server error, retry

Get Started

Core Concepts

Pipeline Scripts

Standalone Scripts

Configuration

​Overview

​1. Full Market Data

​Endpoint Details

​Request Payload

​Response Structure

​Key Fields

​2. Fundamental Data (Results & Ratios)

​Endpoint Details

​Request Payload

​Response Structure

​Data Sections

​3. Company Filings (Hybrid)

​Endpoint Details

​Why Two Endpoints?

​Request Payload

​Response Structure

​Deduplication Logic

​4. Live Announcements

​Endpoint Details

​Request Payload

​Response Structure

​5. Advanced Indicators (Pivot, EMA, SMA)

​Endpoint Details

​Request Payload

​Response Structure

​6. Market News Feed

​Endpoint Details

​Request Payload

​Response Structure

​Sentiment Analysis

​7. Corporate Actions

​Endpoint Details

​Request Payload (Upcoming)

​Request Payload (History)

​Response Structure

​Action Types

​8. Surveillance Lists (ASM/GSM)

​Endpoint Details

​ASM (Additional Surveillance Measure)

​GSM (Graded Surveillance Measure)

​Response Structure

​9. Circuit Stocks

​Endpoint Details

​Request Payload (Upper Circuit)

​10. Bulk/Block Deals

​Endpoint Details

​Request Payload

​Response Structure

​11. Price Bands

​Incremental Price Bands

​Complete Price Bands

​CSV Structure

​12. Historical OHLCV

​Endpoint Details

​Request Payload

​Response CSV Structure

​Smart Incremental Updates

​Rate Limits & Best Practices

​Threading Guidelines

​Timeout Recommendations

​Error Handling

​Next Steps

Pipeline Architecture

Output Schema

Build docs developers (and LLMs) love

Overview

1. Full Market Data

Endpoint Details

Request Payload

Response Structure

Key Fields

2. Fundamental Data (Results & Ratios)

Endpoint Details

Request Payload

Response Structure

Data Sections

3. Company Filings (Hybrid)

Endpoint Details

Why Two Endpoints?

Request Payload

Response Structure

Deduplication Logic

4. Live Announcements

Endpoint Details

Request Payload

Response Structure

5. Advanced Indicators (Pivot, EMA, SMA)

Endpoint Details

Request Payload

Response Structure

6. Market News Feed

Endpoint Details

Request Payload

Response Structure

Sentiment Analysis

7. Corporate Actions

Endpoint Details

Request Payload (Upcoming)

Request Payload (History)

Response Structure

Action Types

8. Surveillance Lists (ASM/GSM)

Endpoint Details

ASM (Additional Surveillance Measure)

GSM (Graded Surveillance Measure)

Response Structure

9. Circuit Stocks

Endpoint Details

Request Payload (Upper Circuit)

10. Bulk/Block Deals

Endpoint Details

Request Payload

Response Structure

11. Price Bands

Incremental Price Bands

Complete Price Bands

CSV Structure

12. Historical OHLCV

Endpoint Details

Request Payload

Response CSV Structure

Smart Incremental Updates

Rate Limits & Best Practices

Threading Guidelines

Timeout Recommendations

Error Handling

Next Steps