​

Data Sources

​

Required Data Files

1. DEM Tiles or GeoTIFF: Terrain elevation data
2. DVNL GeoTIFF: Light pollution / night lights data
3. Gaia Star Catalog: Stellar positions and photometry
4. DE421 Ephemeris: Solar system body positions

​

Directory Structure

TerraLab/
├── data/
│   ├── stars/
│   │   ├── gaia_stars.json
│   │   └── de421.bsp
│   ├── light_pollution/
│   │   └── C_DVNL 2022.tif
│   └── dem/
│       ├── tiles/              # For ASC tile format
│       │   ├── tile_001.asc
│       │   └── ...
│       └── europe_dem.tif      # For single large GeoTIFF
└── ...

​

1. Digital Elevation Models (DEM)

​

Format A: Tiled ESRI ASCII Grid (.asc)

• Format: ESRI ASCII Grid (.asc) or NumPy binary cache (.npy)
• Coordinate System: UTM Zone 31N (EPSG:25831)
• Resolution: Typically 5m, 15m, or 30m per pixel
• Coverage: Regional (e.g., Catalonia)

• ICGC Download: https://www.icgc.cat/en/Downloads/Elevation-data
• Select “Digital Elevation Model” (MET)
• Download tiles covering your area of interest

dem/tiles/
├── E320000_N4580000.asc
├── E325000_N4580000.asc
└── ...

​

Format B: Large GeoTIFF

• Format: Cloud-Optimized GeoTIFF (COG) or standard GeoTIFF
• Coordinate System: Various (auto-detected), typically EPSG:4326 or EPSG:3035
• Resolution: 25m to 90m per pixel
• Coverage: Continental (e.g., all of Europe)
• File Size: Can exceed 18 GB for continental coverage

• Copernicus EU-DEM: https://land.copernicus.eu/imagery-in-situ/eu-dem
• NASA SRTM: https://www.earthdata.nasa.gov/
• USGS Earth Explorer: https://earthexplorer.usgs.gov/

1. Click “Browse” and select your GeoTIFF or tiles directory
2. Path is saved to ~/.terralab_config.json
3. Reopen dialog from settings menu to change

• TerraLab uses windowed reading for large files
• Data loaded into RAM for region of interest only
• Coordinate transformations handled automatically

​

Coordinate System Handling

• Lat/Lon converted to UTM meters using WGS84 ellipsoid formulas
• Local flat-earth approximation for small regions
• Meters-per-degree scaling: ~111,320 m/deg latitude
• Longitude scaling: 111,320 × cos(latitude) m/deg

• Native CRS detected via rasterio
• Automatic transformation between UTM and native CRS
• Uses rasterio.warp.transform for accuracy

​

2. Light Pollution Data (DVNL)

• Format: GeoTIFF
• Coordinate System: Equal-area projection (typically EPSG:8857)
• Units: Radiance (nW/cm²/sr)
• Resolution: ~500m globally
• Temporal: Annual composite
• File Size: ~2-5 GB (global) or ~200-500 MB (regional)

• Official source: https://eogdata.mines.edu/products/vnl/
• Select “VIIRS Night Lights - Annual VNL V2”
• Download the most recent year (e.g., 2022)
• Choose the “vcm-orm-ntl” (cloud-masked, outlier-removed) product

C_DVNL 2022.tif

1. Convolution (optional preprocessing):
   • Gaussian kernel (σ = 1.5 km) simulates light scatter
   • Script: scripts/dvnl_convolve.py
   • Creates *_convolved.tif
2. SQM Calibration:
   • Converts radiance → Sky Quality Meter reading
   • Formula: SQM = 22.0 - 2.4 × log₁₀(radiance + 0.001)
   • Script: scripts/calibrate_sqm.py
3. Bortle Mapping:
   • Converts SQM → Bortle class (1-9)
   • Thresholds defined in light_pollution/bortle.py

​

Bortle Class Thresholds

​

3. Gaia Star Catalog

• Format: JSON (list of arrays) or dict format
• Fields required:
  • source_id or id: Unique star identifier
  • ra: Right Ascension (degrees, ICRS)
  • dec: Declination (degrees, ICRS)
  • phot_g_mean_mag or mag: G-band magnitude
  • bp_rp: Color index (B_P - R_P)
  • designation or name: Star name (optional)
• Typical size: 10,000 to 500,000 stars
• Magnitude range: Typically mag < 7.5 for visual use

• URL: https://gea.esac.esa.int/archive/
• Use ADQL query language
• Example query for bright stars:

SELECT source_id, ra, dec, phot_g_mean_mag, bp_rp, designation
FROM gaiadr3.gaia_source
WHERE phot_g_mean_mag < 7.5
  AND ra BETWEEN 0 AND 360
  AND dec BETWEEN -30 AND 60

• Export as CSV, convert to JSON

• Many astronomy projects provide Gaia subsets
• Check for “visual observing” or “planetarium” catalogs
• Ensure magnitude limit appropriate for your needs

{
  "data": [
    ["1234567890", "Sirius", 101.287, -16.716, -1.46, -0.05],
    ["9876543210", "Betelgeuse", 88.793, 7.407, 0.42, 1.85],
    ...
  ]
}

[
  {
    "source_id": "1234567890",
    "designation": "Sirius",
    "ra": 101.287,
    "dec": -16.716,
    "phot_g_mean_mag": -1.46,
    "bp_rp": -0.05
  },
  ...
]

TerraLab/data/stars/gaia_stars.json

• gaia_cache_ra.npy
• gaia_cache_dec.npy
• gaia_cache_mag.npy
• gaia_cache_r.npy, gaia_cache_g.npy, gaia_cache_b.npy (computed colors)
• gaia_cache_ids.npy
• gaia_cache_names.npy
• gaia_cache_bprp.npy

​

Color Calculation

​

4. DE421 Ephemeris

• Format: SPICE Binary Kernel (.bsp)
• Coverage: 1900 to 2050
• Bodies included:
  • Sun, Moon, Mercury, Venus, Mars
  • Jupiter, Saturn, Uranus, Neptune, Pluto
  • Earth-Moon barycenter
• Coordinate system: ICRF/J2000
• File size: ~17 MB

• URL: https://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/planets/
• Direct file: de421.bsp
• wget command:

wget https://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/planets/de421.bsp

from skyfield.api import load
eph = load('de421.bsp')  # Downloads if not present

TerraLab/data/stars/de421.bsp

ts = load.timescale()
eph = load('de421.bsp')
earth = eph['earth']
moon = eph['moon']
sun = eph['sun']

​

Alternatives

• More accurate, larger files (100+ MB)
• Extended coverage (1550-2650 for DE440)
• Use if precision critical or observing distant dates

• Older, smaller (6 MB)
• Coverage: 1600-2200
• Adequate for most visual observing

​

Configuration File

~/.terralab_config.json

{
  "raster_path": "/path/to/dem/tiles/",
  "horizon_quality": 20,
  "last_location_lat": 41.3874,
  "last_location_lon": 2.1686,
  "bortle_manual": 4,
  "auto_bortle": true,
  "copernicus_api_key": "your-api-key",
  "copernicus_api_url": "https://cds.climate.copernicus.eu/api"
}

• get_config_value(key, default)
• set_config_value(key, value)
• Auto-saves on every write

​

Data Loading Process

​

Startup Sequence

1. Configuration Check (__main__.py:38-49):
   • Load config from ~/.terralab_config.json
   • If raster_path missing or invalid, show TerrainConfigDialog
   • User selects DEM source
2. Catalog Loading (background thread):
   • CatalogLoaderWorker loads Gaia JSON
   • Builds NumPy arrays for vectorized rendering
   • Creates binary cache if not present
   • Emits catalog_ready signal
3. Ephemeris Loading (background thread):
   • SkyfieldLoaderWorker loads DE421
   • Initializes timescale
   • Emits skyfield_ready signal
4. First Render:
   • Sky canvas renders with available stars
   • Horizon profile computed on-demand when location set

​

Horizon Baking

1. Region Preparation:
   • Calculate bounding box: [lat-radius, lon-radius, lat+radius, lon+radius]
   • Load DEM tiles or GeoTIFF window into memory
   • Show progress: ”⏳ Loading maps: X/Y (Z%)”
2. Raycasting:
   • Cast rays in 360° sweep (default 720 rays, 0.5° spacing)
   • Depth bands: 10-60 bands (quality setting)
   • Apply Earth curvature correction
   • Sample light pollution at ray-terrain intersections
3. Profile Caching:
   • Save to .npz file in cache directory
   • Keyed by: lat_lon_quality_hash
   • Automatic reload on next session

​

Data Updates and Maintenance

​

When to Update

• New data releases every ~18-24 months
• Update if working with precision astrometry
• Visual users: Update optional

• Annual releases (e.g., 2021, 2022, 2023)
• Update yearly to track urban light growth
• Critical for accurate Bortle classification

• Update when new LiDAR surveys published
• ICGC updates Catalonia DEM periodically
• Important for mountainous regions with new infrastructure

• Valid until 2050
• Update only if:
  • Working beyond 2050
  • Requiring higher precision (switch to DE440)
  • Studying historical dates before 1900

​

Verifying Data Integrity

• gaia_stars.json: 5 MB - 50 MB (depends on star count)
• de421.bsp: ~17 MB
• C_DVNL 2022.tif: 200 MB - 5 GB
• DEM tiles: Varies widely

python -m TerraLab 2>&1 | grep -E "(Loaded|Error|Warning)"

• [CatalogLoader] Loaded N stars
• [SkyfieldLoader] Initialized
• [TiffRasterWindowProvider] Opening dataset
• No error messages

​

Troubleshooting

​

”Catalog not found” Error

1. Verify file exists: ls TerraLab/data/stars/gaia_stars.json
2. Check JSON syntax: python -m json.tool < gaia_stars.json
3. Fallback: TerraLab generates 500 random stars for testing

​

”GeoTIFF not found” Error

1. Open terrain config dialog (Settings menu)
2. Browse to correct path
3. Verify path exists: ls -lh /path/to/dem.tif

​

”Rasterio error” on GeoTIFF

1. Verify with gdalinfo: gdalinfo dem.tif
2. Check for compression: COG preferred
3. Re-download if corrupted
4. Convert with: gdal_translate -co COMPRESS=LZW input.tif output.tif

​

Slow Star Catalog Loading

1. Wait for first load (creates NPY cache)
2. Subsequent loads will be ~10× faster
3. Consider reducing catalog size (mag < 7.0 sufficient for visual)

​

Missing Light Pollution Data

1. Check TerraLab/data/light_pollution/ exists
2. Verify C_DVNL 2022.tif present
3. Manual Bortle mode: Disable “Auto Bortle” in settings
4. Set manual Bortle class (1-9)

​

Advanced: Custom Data Pipelines

​

Creating Custom Star Catalogs

import json

# Load Gaia bright stars
with open('gaia_bright.json') as f:
    gaia = json.load(f)['data']

# Add Messier objects manually
messier = [
    ["M31", "Andromeda Galaxy", 10.685, 41.269, 3.4, 0.5],
    # ... more objects
]

combined = gaia + messier
with open('TerraLab/data/stars/gaia_stars.json', 'w') as f:
    json.dump({'data': combined}, f)

​

Preprocessing DVNL for Performance

# Run TerraLab convolution script
python scripts/dvnl_convolve.py \
  --input C_DVNL_2022.tif \
  --output europe_convolved.tif \
  --sigma 1.5

# Clip to bounding box (faster loading)
gdal_translate -projwin lon_min lat_max lon_max lat_min \
  europe_convolved.tif region_lp.tif

set_config_value('light_pollution_path', 'data/light_pollution/region_lp.tif')

​

Data Licensing and Attribution

• License: CC BY-SA 3.0 IGO
• Attribution: “This work has made use of data from the European Space Agency (ESA) mission Gaia.”

• License: Public domain (U.S. Government work)
• Attribution: “Earth Observation Group, Payne Institute, Colorado School of Mines.”

• License: CC BY 4.0
• Attribution: “Institut Cartogràfic i Geològic de Catalunya (ICGC)”

• License: Free and open access
• Attribution: “Copernicus Land Monitoring Service, European Environment Agency.”

• License: Public domain (NASA/JPL)
• Attribution: “Jet Propulsion Laboratory, California Institute of Technology.”