Skip to main content
The EDL Pipeline allows customization of API request payloads to filter stocks by various criteria. This guide shows how to modify filters for targeted data fetching.

​
Filter Locations

Filters are defined in individual fetch scripts as request payloads. The primary filter entry point is fetch_dhan_data.py.

​
Default Filters

​
Fetch All NSE Stocks

By default, fetch_dhan_data.py fetches all NSE equity stocks (lines 14-40): 
payload = {
    "data": {
        "sort": "Mcap",
        "sorder": "desc",
        "count": 5000,  # Large count to get all stocks
        "fields": [...],  # 50+ field list
        "params": [
            {"field": "OgInst", "op": "", "val": "ES"},  # Equity Segment
            {"field": "Exch", "op": "", "val": "NSE"}    # NSE Exchange
        ],
        "pgno": 0,
        "sorder": "desc",
        "sort": "Mcap"
    }
}
This default configuration fetches ~2,775 NSE-listed equity stocks sorted by market capitalization.

​
Common Filter Customizations

​
1. Filter by Market Cap Range

Add market cap filters to the params array: 
payload = {
    "data": {
        "sort": "Mcap",
        "sorder": "desc",
        "count": 5000,
        "fields": [...],
        "params": [
            {"field": "OgInst", "op": "", "val": "ES"},
            {"field": "Exch", "op": "", "val": "NSE"},
            # Only large-cap stocks (> ₹10,000 Cr.)
            {"field": "Mcap", "op": "GT", "val": "10000"}
        ],
        "pgno": 0,
        "sorder": "desc",
        "sort": "Mcap"
    }
}
Operators: GT (greater than), LT (less than), GTE (greater than or equal), LTE (less than or equal), EQ (equals)

​
2. Filter by Sector

payload = {
    "data": {
        "sort": "Mcap",
        "sorder": "desc",
        "count": 5000,
        "fields": [...],
        "params": [
            {"field": "OgInst", "op": "", "val": "ES"},
            {"field": "Exch", "op": "", "val": "NSE"},
            # Only IT sector
            {"field": "Sector", "op": "EQ", "val": "IT"}
        ],
        "pgno": 0
    }
}
Common Sectors: IT, Banking, Pharma, Auto, Metals, Energy, FMCG

​
3. Filter by P/E Ratio

payload = {
    "data": {
        "sort": "Mcap",
        "sorder": "desc",
        "count": 5000,
        "fields": [...],
        "params": [
            {"field": "OgInst", "op": "", "val": "ES"},
            {"field": "Exch", "op": "", "val": "NSE"},
            # P/E between 10 and 25
            {"field": "Pe", "op": "GT", "val": "10"},
            {"field": "Pe", "op": "LT", "val": "25"}
        ],
        "pgno": 0
    }
}

​
4. F&O Stocks Only

payload = {
    "data": {
        "sort": "Mcap",
        "sorder": "desc",
        "count": 5000,
        "fields": [...],
        "params": [
            {"field": "OgInst", "op": "", "val": "ES"},
            {"field": "Exch", "op": "", "val": "NSE"},
            # Only F&O stocks
            {"field": "FnoFlag", "op": "EQ", "val": "1"}
        ],
        "pgno": 0
    }
}
This filter reduces the dataset from ~2,775 to ~207 F&O-enabled stocks, significantly speeding up pipeline execution.

​
5. Volume Filter (High Liquidity)

payload = {
    "data": {
        "sort": "volume",
        "sorder": "desc",
        "count": 5000,
        "fields": [...],
        "params": [
            {"field": "OgInst", "op": "", "val": "ES"},
            {"field": "Exch", "op": "", "val": "NSE"},
            # Volume > 1,000,000 shares
            {"field": "volume", "op": "GT", "val": "1000000"}
        ],
        "pgno": 0
    }
}

​
Advanced Filter Combinations

​
Quality Stock Screen

Combine multiple fundamental criteria: 
payload = {
    "data": {
        "sort": "Mcap",
        "sorder": "desc",
        "count": 5000,
        "fields": [...],
        "params": [
            {"field": "OgInst", "op": "", "val": "ES"},
            {"field": "Exch", "op": "", "val": "NSE"},
            {"field": "Mcap", "op": "GT", "val": "5000"},     # > ₹5,000 Cr.
            {"field": "Roe", "op": "GT", "val": "15"},        # ROE > 15%
            {"field": "Pe", "op": "LT", "val": "30"},         # P/E < 30
            {"field": "DivYeild", "op": "GT", "val": "1"}    # Dividend Yield > 1%
        ],
        "pgno": 0
    }
}

​
Momentum Screen (Technical)

payload = {
    "data": {
        "sort": "PricePerchng1mon",
        "sorder": "desc",
        "count": 500,
        "fields": [...],
        "params": [
            {"field": "OgInst", "op": "", "val": "ES"},
            {"field": "Exch", "op": "", "val": "NSE"},
            {"field": "PricePerchng1mon", "op": "GT", "val": "10"},  # 1M return > 10%
            {"field": "DayRSI14CurrentCandle", "op": "GT", "val": "50"},  # RSI > 50
            {"field": "DaySMA50CurrentCandle", "op": "GT", "val": "0"}   # Above SMA50
        ],
        "pgno": 0
    }
}

​
Available Filter Fields

Common fields you can filter on:
FieldDescriptionExample Values
McapMarket Cap (₹ Cr.)1000, 10000, 50000
PeP/E Ratio15, 20, 30
RoeROE (%)10, 15, 20
ROCEROCE (%)15, 20, 25
DivYeildDividend Yield (%)1, 2, 3
volumeVolume100000, 1000000
SectorSectorIT, Banking, Pharma
FnoFlagF&O Enabled1 (yes), 0 (no)
PricePerchng1mon1M Return (%)5, 10, 20
PricePerchng1year1Y Return (%)10, 20, 50
DayRSI14CurrentCandleRSI (14)30, 50, 70
DaySMA50CurrentCandleDistance from SMA50Positive/negative values

​
Implementing Custom Filters

1

Edit fetch_dhan_data.py

Open fetch_dhan_data.py and locate the payload dictionary (lines 16-40).
2

Modify params array

Add your filter criteria to the params array using the format:
{"field": "FieldName", "op": "Operator", "val": "Value"}
3

Test the filter

Run the script standalone to verify results:
python3 fetch_dhan_data.py
Check dhan_data_response.json to confirm filtered stock count.
4

Run full pipeline

Once verified, run the complete pipeline:
python3 run_full_pipeline.py

​
Pagination for Large Results

If your filter returns > 5,000 results, implement pagination: 
all_data = []
page = 0

while True:
    payload = {
        "data": {
            "sort": "Mcap",
            "sorder": "desc",
            "count": 5000,
            "fields": [...],
            "params": [...],
            "pgno": page  # Increment page
        }
    }
    
    response = requests.post(url, json=payload, headers=headers)
    data = response.json().get('data', [])
    
    if not data:
        break  # No more pages
    
    all_data.extend(data)
    page += 1
    
print(f"Fetched {len(all_data)} total stocks across {page} pages")

​
Downstream Impact

Custom filters affect all downstream scripts:
If you filter fetch_dhan_data.py to 100 stocks, all subsequent scripts will only process those 100 stocks. The master_isin_map.json file is used by all Phase 2+ scripts.
Affected scripts:
  • fetch_fundamental_data.py - Fetches data for filtered stocks only
  • fetch_company_filings.py - ~100 filings instead of ~2,775
  • fetch_market_news.py - News for filtered stocks only
  • All other enrichment scripts

​
Pre-built Filter Presets

Create filter preset files for common scenarios: filters/large_cap.json: 
[
  {"field": "Mcap", "op": "GT", "val": "10000"}
]
filters/quality.json: 
[
  {"field": "Mcap", "op": "GT", "val": "5000"},
  {"field": "Roe", "op": "GT", "val": "15"},
  {"field": "Pe", "op": "LT", "val": "30"}
]
Load dynamically: 
import json

with open('filters/large_cap.json', 'r') as f:
    custom_filters = json.load(f)

payload['data']['params'].extend(custom_filters)

​
Performance Considerations

  • Smaller datasets = faster pipeline execution
  • F&O filter (207 stocks): ~3 min total pipeline time
  • Large-cap filter (500 stocks): ~4-5 min total pipeline time
  • No filter (2,775 stocks): ~6-9 min total pipeline time (without OHLCV)
For daily updates, consider filtering to liquid, high-volume stocks to reduce execution time from 6-9 min to 3-4 min.

​
Next Steps

Error Handling

Handle API errors and retries for filtered queries

Performance Tuning

Optimize threading and batching for filtered datasets

Build docs developers (and LLMs) love

Get started for freeTalk to us