Data Management

Overview

The Product Distribution Dashboard loads data from JSON sources, validates it, persists it to a PostgreSQL database, and automatically refreshes when changes are detected. This document explains the data models, loading process, and refresh mechanisms.

Data Sources

The system loads three types of data from JSON files or URLs:

Products

Product catalog with available sizes

Stores

Retail locations with demand and capacity

Warehouses

Distribution centers with inventory

Configuration

Data source URLs are configured in application.properties:

data.products.url=https://example.com/products.json
data.stores.url=https://example.com/stores.json
data.warehouses.url=https://example.com/warehouses.json

URLs can point to:

External HTTP/HTTPS endpoints
Local file system paths (file://)
Cloud storage URLs

Data Models

Product Model

Represents a product in the catalog:

@Entity
@Table(name = "products")
public class Product {
    @Id
    private String id;

    private String brandId;
    @ElementCollection
    private List<String> sizes;
}

String

required

Unique product identifier

brandId

String

Brand identifier for grouping products

sizes

List<String>

Available sizes for this product (e.g., [“S”, “M”, “L”, “XL”])

Example JSON:

{
  "id": "P1",
  "brandId": "BRAND_A",
  "sizes": ["S", "M", "L", "XL"]
}

Store Model

Represents a retail location with demand:

@Entity
@Table(name = "stores")
public class Store {
    @Id
    private String id;

    private Double latitude;
    private Double longitude;
    private String country;
    private Integer maxStockCapacity;
    @JsonProperty("expected_return_rate")
    private Double expectedReturnRate;
    
    private Integer remainingCapacity;

    @OneToMany(cascade = CascadeType.ALL, orphanRemoval = true, fetch = FetchType.LAZY)
    @JoinColumn(name = "store_id")
    private List<ProductItem> demand;
}

String

required

Unique store identifier

latitude

Double

required

Geographic latitude coordinate

longitude

Double

required

Geographic longitude coordinate

country

String

Country code or name

maxStockCapacity

Integer

required

Maximum number of units the store can hold

expectedReturnRate

Double

Expected return rate as decimal (0.0 - 1.0). Used to adjust demand calculations.

demand

List<ProductItem>

List of products and quantities the store needs

Example JSON:

{
  "id": "S1",
  "latitude": 48.8566,
  "longitude": 2.3522,
  "country": "France",
  "max_stock_capacity": 1000,
  "expected_return_rate": 0.15,
  "demand": [
    {
      "productId": "P1",
      "size": "M",
      "quantity": 50
    }
  ]
}

Capacity Management:

public int calculateAdjustedDemand(int originalDemand) {
    return (int) Math.ceil(originalDemand * (1 - this.expectedReturnRate));
}

public boolean hasCapacityFor(int quantity) {
    return this.remainingCapacity >= quantity;
}

public boolean tryAllocateCapacity(int quantity) {
    if (!hasCapacityFor(quantity)) {
        return false;
    }
    this.remainingCapacity -= quantity;
    return true;
}

Warehouse Model

Represents a distribution center with inventory:

@Entity
@Table(name = "warehouses")
public class Warehouse {
    @Id
    private String id;

    private Double latitude;
    private Double longitude;
    private String country;

    @OneToMany(cascade = CascadeType.ALL, orphanRemoval = true, fetch = FetchType.LAZY)
    @JoinColumn(name = "warehouse_id")
    private List<ProductItem> stock;
}

String

required

Unique warehouse identifier

latitude

Double

required

Geographic latitude coordinate

longitude

Double

required

Geographic longitude coordinate

country

String

Country code or name

stock

List<ProductItem>

List of products and quantities available in inventory

Example JSON:

{
  "id": "W1",
  "latitude": 51.5074,
  "longitude": -0.1278,
  "country": "United Kingdom",
  "stock": [
    {
      "productId": "P1",
      "size": "M",
      "quantity": 500
    }
  ]
}

Stock Lookup Methods:

public Optional<ProductItem> findStock(String productId, String size) {
    return this.stock.stream()
        .filter(item -> item.getProductId().equals(productId) 
                     && item.getSize().equals(size))
        .findFirst();
}

public int getStockForProduct(String productId, String size) {
    return findStock(productId, size)
        .map(ProductItem::getQuantity)
        .orElse(0);
}

Data Loading Process

The JsonDataLoaderService handles loading data from configured URLs:

@Override
public List<Product> loadProducts() {
    return loadFromUrl(productsUrl, new TypeReference<List<Product>>() {});
}

@Override
public List<Store> loadStores() {
    return loadFromUrl(storesUrl, new TypeReference<List<Store>>() {});
}

@Override
public List<Warehouse> loadWarehouses() {
    return loadFromUrl(warehousesUrl, new TypeReference<List<Warehouse>>() {});
}

Loading Implementation

private <T> T loadFromUrl(String url, TypeReference<T> typeReference) {
    try {
        String jsonContent = webClient.get()
            .uri(url)
            .retrieve()
            .bodyToMono(String.class)
            .block();
        
        if (jsonContent == null || jsonContent.trim().isEmpty()) {
            throw new DataLoadingException("Empty response from URL: " + url);
        }
        
        return objectMapper.readValue(jsonContent, typeReference);
    } catch (WebClientException ex) {
        throw new DataLoadingException("HTTP error loading from URL: " + url, ex);
    } catch (IOException ex) {
        throw new DataLoadingException("JSON parsing error loading from URL: " + url, ex);
    }
}

Error Handling:

Throws DataLoadingException for HTTP errors
Throws DataLoadingException for JSON parsing errors
Validates non-empty response
Uses blocking WebClient for synchronous loading

Database Persistence

Data is persisted to PostgreSQL using Spring Data JPA:

Repositories
Schema Management

public interface ProductRepository extends JpaRepository<Product, String> {}
public interface StoreRepository extends JpaRepository<Store, String> {}
public interface WarehouseRepository extends JpaRepository<Warehouse, String> {}
public interface StockAssignmentRepository extends JpaRepository<StockAssignment, UUID> {}
public interface UnfulfilledDemandRepository extends JpaRepository<UnfulfilledDemand, UUID> {}

spring.jpa.hibernate.ddl-auto=validate

Schema validation mode ensures database schema matches entity definitions. Use migrations (Flyway/Liquibase) for schema changes in production.

Entity Relationships:

Products: Standalone entities
Stores: Have one-to-many relationship with ProductItem (demand)
Warehouses: Have one-to-many relationship with ProductItem (stock)
StockAssignments: Created by distribution algorithm
UnfulfilledDemands: Created when demand cannot be satisfied

Scheduled Refresh

The system automatically checks for data changes and triggers redistribution:

@PostConstruct
public void onStartup() {
    logger.info("🔁 Executing initial distribution on application startup...");
    distributionService.distributeProducts();
    jsonChangeWatcher.updateContentHashes();
}

Startup Process:

Application starts
Initial distribution runs automatically
Content hashes are stored for change detection

Scheduled Check

@Scheduled(cron = "${scheduler.distribution.cron}")
public void nightlyDistribution() {
    logger.info("🔍 Checking if JSON files have changed (comparing content hashes)...");

    if (jsonChangeWatcher.hasAnyChanged()) {
        logger.info("🔁 Changes detected in JSON files. Executing redistribution...");
        distributionService.distributeProducts();
        logger.info("✅ Distribution completed");
    } else {
        logger.info("⏭️  No changes detected in files. Distribution skipped.");
    }
}

Configuration:

# Example: Run at 2 AM daily
scheduler.distribution.cron=0 0 2 * * *

Change Detection:

Compares content hashes (not timestamps)
Only triggers redistribution if actual content changed
Efficient for large files
Avoids unnecessary processing

Data Validation

Validation occurs at multiple levels:

JSON Schema Validation

Jackson ObjectMapper validates JSON structure matches entity annotations:

Required fields must be present
Data types must match
Collections must be properly formatted

JPA Validation

Hibernate validates entities before persistence:

ID fields must be unique
Foreign key constraints enforced
Data types and lengths validated

Business Logic Validation

Distribution service validates business rules:

Capacity cannot be negative
Quantities must be positive
Geographic coordinates must be valid
Return rates must be between 0 and 1

Data Flow

Load from Source

JsonDataLoaderService fetches data from configured URLs

Parse JSON

Jackson ObjectMapper deserializes JSON to entity objects

Validate

Entities validated against constraints and business rules

Persist

JPA repositories save entities to PostgreSQL database

Process

Distribution algorithm processes data and creates assignments

Store Results

Stock assignments and unfulfilled demands saved to database

Cache Metrics

Metrics calculated and cached for quick access

Best Practices

Data Consistency

Keep JSON files synchronized
Use transactions for multi-entity updates
Validate data before committing
Monitor error logs for parsing issues

Performance

Schedule refreshes during low-traffic periods
Use content hashing to avoid unnecessary processing
Index frequently queried fields
Monitor database query performance

Reliability

Implement retry logic for failed loads
Log all data loading operations
Set up alerts for loading failures
Maintain backup data sources

Scalability

Use lazy loading for entity relationships
Implement pagination for large datasets
Consider data partitioning for massive scale
Monitor memory usage during loads

Troubleshooting

Data Loading Fails

Symptoms: DataLoadingException in logsSolutions:

Check URL accessibility
Verify JSON format validity
Review network connectivity
Check authentication/authorization
Validate SSL certificates for HTTPS

JSON Parsing Errors

Symptoms: IOException or mapping errorsSolutions:

Validate JSON against schema
Check field name case sensitivity
Verify data types match entity definitions
Look for missing required fields
Check for special characters in strings

Database Constraint Violations

Symptoms: ConstraintViolationExceptionSolutions:

Verify unique IDs across entities
Check foreign key references
Validate data ranges (e.g., capacity > 0)
Review entity relationships
Check for duplicate entries

Distribution Not Triggering

Symptoms: No redistribution despite data changesSolutions:

Verify cron expression is valid
Check scheduler is enabled (not in test profile)
Review JsonChangeWatcher hash comparison
Ensure content actually changed (not just timestamp)
Check application logs for scheduler execution

API Access

Access raw data via REST endpoints:

GET /api/products
GET /api/products/{id}

Get Started

Installation

Core Features

Configuration

Overview

Data Sources

Products

Stores

Warehouses

Configuration

Data Models

Product Model

Store Model

Warehouse Model

Data Loading Process

Loading Implementation

Database Persistence

Scheduled Refresh

Scheduled Check

Data Validation

Data Flow

Best Practices

Data Consistency

Performance

Reliability

Scalability

Troubleshooting

API Access

Next Steps

Distribution Algorithm

Metrics

Build docs developers (and LLMs) love

Get Started

Installation

Core Features

Configuration

​Overview

​Data Sources

Products

Stores

Warehouses

​Configuration

​Data Models

​Product Model

​Store Model

​Warehouse Model

​Data Loading Process

​Loading Implementation

​Database Persistence

​Scheduled Refresh

​Scheduled Check

​Data Validation

​Data Flow

​Best Practices

Data Consistency

Performance

Reliability

Scalability

​Troubleshooting

​API Access

​Next Steps

Distribution Algorithm

Metrics

Build docs developers (and LLMs) love

Overview

Data Sources

Configuration

Data Models

Product Model

Store Model

Warehouse Model

Data Loading Process

Loading Implementation

Database Persistence

Scheduled Refresh

Scheduled Check

Data Validation

Data Flow

Best Practices

Troubleshooting

API Access

Next Steps