Indices & ETF Scripts - ChartsMaze EDL Pipeline

These two standalone scripts fetch comprehensive data for NSE Indices and ETFs. They are not included in the main pipeline by default and run only when FETCH_OPTIONAL = True in run_full_pipeline.py.

Overview

Script	Output File	Records	Purpose
`fetch_all_indices.py`	`all_indices_list.json`	~194 indices	NSE market indices with live OHLCV + volume
`fetch_etf_data.py`	`etf_data_response.json`	~361 ETFs	Exchange-traded funds with fundamentals

fetch_all_indices.py

What It Does

Fetches all NSE indices (sectoral, thematic, and benchmark) with live intraday data including today’s volume, OHLC, and 52-week high/low.

API Endpoint

URL: https://ow-scanx-analytics.dhan.co/customscan/fetchdt
Method: POST

Payload:
{
  "data": {
    "sort": "Sym",
    "sorder": "asc",
    "count": 500,
    "fields": [
      "Sym", "DispSym", "Sid", "Exch", "Seg", "Inst",
      "Open", "High", "Low", "Ltp", "Pchange", "PPerchange",
      "High1Yr", "Low1Yr",
      "Min1TotalVolPrevCandle"  // Today's cumulative volume
    ],
    "params": [
      {"field": "Inst", "op": "", "val": "IDX"},
      {"field": "Exch", "op": "", "val": "IDX"}
    ],
    "pgno": 1
  }
}

Filter Logic:

Inst = IDX: Instrument type is Index
Exch = IDX: Exchange is Index segment (excludes equity stocks)

Output Format

File: all_indices_list.json (Array of ~194 indices)

[
  {
    "IndexName": "NIFTY 50",
    "Symbol": "Nifty 50",
    "IndexID": "13",
    "Exchange": "IDX",
    "Segment": "IDX",
    "Instrument": "IDX",
    "Open": 21450.30,
    "High": 21523.80,
    "Low": 21398.50,
    "Ltp": 21487.65,
    "Chng": 37.35,
    "PChng": 0.17,
    "Volume": 0,
    "52W_High": 22526.60,
    "52W_Low": 19281.75
  },
  {
    "IndexName": "NIFTY Bank",
    "Symbol": "Nifty Bank",
    "IndexID": "25",
    "Exchange": "IDX",
    "Segment": "IDX",
    "Instrument": "IDX",
    "Open": 46523.40,
    "High": 46789.20,
    "Low": 46412.10,
    "Ltp": 46651.35,
    "Chng": 128.95,
    "PChng": 0.28,
    "Volume": 0,
    "52W_High": 48788.30,
    "52W_Low": 40450.55
  }
]

Volume Data Behavior

Volume is 0 for all indices because NSE does not publish volume for index instruments. The field Min1TotalVolPrevCandle returns 0 or -1 (overflow) for indices.For constituent stock volumes, use the main pipeline’s equity data or fetch_dhan_data.py.

Index Categories

Benchmark Indices (5)

NIFTY 50: Top 50 large-cap stocks
NIFTY 500: Broad market representation
NIFTY Midcap 150: Mid-cap universe
NIFTY Smallcap 250: Small-cap universe
NIFTY Next 50: Stocks next in line for NIFTY 50

Sectoral Indices (11)

NIFTY Bank: Banking sector
NIFTY IT: Information Technology
NIFTY Pharma: Pharmaceuticals
NIFTY Auto: Automobile sector
NIFTY FMCG: Fast-Moving Consumer Goods
NIFTY Metal: Metals & Mining
NIFTY Realty: Real Estate
NIFTY Energy: Oil & Gas
NIFTY Financial Services: Broad financial sector
NIFTY PSU Bank: Public Sector Banks
NIFTY Media: Media & Entertainment

Thematic Indices (30+)

NIFTY Dividend Opportunities 50: High dividend yield stocks
NIFTY Alpha 50: High alpha stocks
NIFTY100 Quality 30: Quality factor index
NIFTY200 Momentum 30: Momentum strategy
NIFTY100 Low Volatility 30: Low volatility stocks
And 25+ other thematic/strategy indices

Custom & Sectoral Sub-Indices (148+)

Specialized indices like NIFTY Commodities, NIFTY Infrastructure, NIFTY MNC, etc.

Usage

python3 fetch_all_indices.py

Output:

Fetching NSE Indices & Today's Volume via ScanX Pro API...
Successfully identified 194 Indices with Today's Volume.

Use Cases

Market Breadth

Track sectoral rotation by comparing index performances

Index Screeners

Build watchlists based on index membership

Benchmark Comparison

Compare stock returns against relevant sectoral indices

Correlation Analysis

Identify inter-sector correlations for portfolio diversification

fetch_etf_data.py

What It Does

Fetches all NSE-listed Exchange Traded Funds (ETFs) with live pricing, fundamentals, and performance metrics.

API Endpoint

URL: https://ow-scanx-analytics.dhan.co/customscan/fetchdt
Method: POST

Payload:
{
  "data": {
    "sort": "Mcap",
    "sorder": "desc",
    "count": 1000,
    "fields": [
      "Isin", "OgInst", "DispSym", "Mcap", "Pe", "DivYeild",
      "Volume", "Ltp", "Sym",
      "PricePerchng1year", "PricePerchng3year", "PricePerchng5year",
      "DaySMA50CurrentCandle", "DaySMA200CurrentCandle",
      "DayRSI14CurrentCandle",
      "ExpenseRatio", "MfCoCode"
    ],
    "params": [
      {"field": "OgInst", "op": "", "val": "ETF"},
      {"field": "Exch", "op": "", "val": "NSE"}
    ],
    "pgno": 0
  }
}

Filter Logic:

OgInst = ETF: Original Instrument type is ETF
Exch = NSE: Only NSE-listed ETFs (excludes BSE)

Output Format

File: etf_data_response.json (Array of ~361 ETFs)

[
  {
    "Isin": "INF204KA1NT8",
    "OgInst": "ETF",
    "DispSym": "SBI - ETF Nifty 50",
    "Sym": "SETFNIF50",
    "Mcap": 4532.15,
    "Ltp": 214.35,
    "Volume": 125430,
    "Pe": 22.5,
    "DivYeild": 1.2,
    "ExpenseRatio": 0.07,
    "PricePerchng1year": 18.3,
    "PricePerchng3year": 54.2,
    "PricePerchng5year": 89.7,
    "DaySMA50CurrentCandle": 210.45,
    "DaySMA200CurrentCandle": 198.30,
    "DayRSI14CurrentCandle": 61.2,
    "High5yr": 225.80,
    "RtAwayFrom5YearHigh": -5.08,
    "MfCoCode": "SBI_MF"
  }
]

ETF Categories

Index ETFs (170+)
Gold & Silver ETFs (45+)
International ETFs (30+)
Debt & Liquid ETFs (50+)
Thematic & Strategy ETFs (66+)

Track major NSE indices:

NIFTY 50 ETFs: Largest cap-weighted index
NIFTY Next 50: Mid-cap index ETFs
NIFTY Bank: Banking sector ETFs
Sectoral ETFs: IT, Pharma, Auto, PSU, Infrastructure

Expense Ratio Insights

# ETFs sorted by expense ratio (lowest first)
Cheapest ETFs:
1. SETFNIF50  - 0.05%  (NIFTY 50)
2. JUNIORBEES - 0.06%  (NIFTY Next 50)
3. BANKBEES   - 0.10%  (NIFTY Bank)

Average Expense Ratio:
- Index ETFs: 0.05% - 0.20%
- Sectoral ETFs: 0.30% - 0.50%
- Thematic ETFs: 0.40% - 0.75%
- International ETFs: 0.50% - 1.00%

Expense ratios below 0.20% are considered highly competitive for passive index ETFs.

Usage

python3 fetch_etf_data.py

Output:

Fetching ETF data from https://ow-scanx-analytics.dhan.co/...
Successfully fetched 361 ETFs. Saved to etf_data_response.json

Use Cases

ETF Screener

Filter ETFs by expense ratio, tracking error, and liquidity

Passive Portfolio Construction

Build diversified portfolios using low-cost index ETFs

Sector Rotation

Identify outperforming sectoral ETFs for tactical allocation

Benchmark Comparison

Compare ETF performance vs underlying index (tracking error)

Running Optional Scripts in Pipeline

By default, these scripts are excluded from the main pipeline. To include them:

Step 1: Enable Optional Fetch

Edit run_full_pipeline.py (lines 66-67):

# Set to True to also fetch standalone data (Indices, ETFs)
FETCH_OPTIONAL = True  # Change from False to True

Step 2: Run Pipeline

python3 run_full_pipeline.py

Output (Phase 6):

📋 PHASE 6: Optional Standalone Data
────────────────────────────────────────
  ▶ Running fetch_all_indices.py...
  ✅ fetch_all_indices.py (2.3s)
  ▶ Running fetch_etf_data.py...
  ✅ fetch_etf_data.py (1.8s)

These files are NOT merged into all_stocks_fundamental_analysis.json. They remain as standalone reference datasets.

Data Freshness

Both scripts fetch live data from Dhan’s ScanX API:

Field	Update Frequency
`Ltp`, `Open`, `High`, `Low`	Real-time (15-second delay)
`Volume`	Cumulative intraday
`52W High/Low`	Daily (post-market)
Technical Indicators (SMA, RSI)	End-of-day

Comparison: Indices vs ETFs

Feature	Indices (`fetch_all_indices.py`)	ETFs (`fetch_etf_data.py`)
Nature	Calculated benchmark (non-tradable)	Tradable securities
Volume	Not applicable (0)	Real trading volume
Expense Ratio	N/A	0.05% - 1.00%
Dividends	Not applicable	Distributed to ETF holders
Tracking Error	N/A	Deviation from underlying index
Use Case	Benchmarking, market analysis	Direct investment, trading

Error Handling

Both scripts include robust error handling:

try:
    response = requests.post(url, json=payload, headers=get_headers(), timeout=15)
    if response.status_code == 200:
        raw_data = response.json().get('data', [])
        # Process and save
        return True
    else:
        print(f"API Error: Status {response.status_code}")
        return False
except Exception as e:
    print(f"Critical API Failure: {e}")
    return False

If the ScanX API is down, these scripts will fail gracefully without crashing the pipeline.

Next Steps

F&O Data Scripts

Fetch Futures & Options data

OHLCV History

Download lifetime price history

Pipeline Flags

Configure FETCH_OHLCV and FETCH_OPTIONAL

Output Reference

Explore all 86 fields in master JSON

Get Started

Core Concepts

Pipeline Scripts

Standalone Scripts

Configuration

​Overview

​fetch_all_indices.py

​What It Does

​API Endpoint

​Output Format

​Volume Data Behavior

​Index Categories

​Usage

​Use Cases

Market Breadth

Index Screeners

Benchmark Comparison

Correlation Analysis

​fetch_etf_data.py

​What It Does

​API Endpoint

​Output Format

​ETF Categories

​Expense Ratio Insights

​Usage

​Use Cases

ETF Screener

Passive Portfolio Construction

Sector Rotation

Benchmark Comparison

​Running Optional Scripts in Pipeline

​Step 1: Enable Optional Fetch

​Step 2: Run Pipeline

​Data Freshness

​Comparison: Indices vs ETFs

​Error Handling

​Next Steps

F&O Data Scripts

OHLCV History

Pipeline Flags

Output Reference

Build docs developers (and LLMs) love

Overview

fetch_all_indices.py

What It Does

API Endpoint

Output Format

Volume Data Behavior

Index Categories

Usage

Use Cases

fetch_etf_data.py

What It Does

API Endpoint

Output Format

ETF Categories

Expense Ratio Insights

Usage

Use Cases

Running Optional Scripts in Pipeline

Step 1: Enable Optional Fetch

Step 2: Run Pipeline

Data Freshness

Comparison: Indices vs ETFs

Error Handling

Next Steps