Data Flow Diagram
The system processes data through a unidirectional pipeline:Data Structures
Input: HTTP Request
Endpoint:POST /scrapping
Request Body:
Stage 1: Database Records
Data retrieved from Corprecam MySQL database via PHP APIs:Compra (Purchase Order Header)
Type:types/types.ts:3-7
api/php.ts:8-23
Example:
CompraItem (Line Items)
Type:types/types.ts:13-21
api/php.ts:25-42
Example:
Material (Product Details)
Type:types/types.ts:23-28
api/php.ts:44-59
Example:
emp_id_fk determines which company (Corprecam or Reciclemos) the product belongs to.
Micro (Route Information)
Type:types/types.ts:29-31
api/php.ts:61-76
Example:
Stage 2: Intermediate Products
After joiningCompraItem with Material, the system creates intermediate product records:
Code: utils/transformDs.ts:35-45
Stage 3: DocumentoSoporte (Final Structure)
The transformed data structure passed to Playwright: Type:types/types.ts:39-44
utils/transformDs.ts:50-67
Data Flow by Component
server.ts Data Flow
Input: HTTP POST body withcompra ID
Processing:
ds object passed to run_playwright()
main.ts Data Flow
Input:DocumentoSoporte object
Processing:
transformDs.ts Data Flow
Input: Raw database records (4 separate arrays) Processing Steps:- Join CompraItem with Material:
- Partition by Company:
- Construct Final Object:
DocumentoSoporte object
utils/functions.ts Data Flow
These functions consume data and produce browser interactions:login()
Input Data:username: Siigo account usernamepassword: Siigo account passworddocumentoSoporteLabelCode: “25470”nit: Supplier NIT fromproveedor_idnit_empresa: Company NIT (“900142913” or “901328575”)
selectProducto()
Input Data:codigo: Product code fromProducts.codigo(e.g., “PLAST-001”)
llenarCantidadValor()
Input Data:cantidad: Quantity fromProducts.cantidadvalor: Unit price fromProducts.precio
seleccionarPago()
Input Data:cuentaNombre: Account name (” CAJA RIOHACHA ” or ” Efectivo ”)
Data Validation
The system performs minimal validation:Implicit Validation
- Required Fields: TypeScript interfaces enforce type structure
-
Material Lookup: Uses
find()which may returnundefined- Fallback: Empty string for
codigo(|| "") - Risk: May attempt to create products with empty codes
- Fallback: Empty string for
-
Company Assignment: Products without
emp_id_fk === 1or2go toreciclemosarray by default (theelsebranch)
Missing Validation
Not Checked:- Whether
compraarray is empty - Whether
compraItemshas matching materials - Numeric ranges (quantity > 0, price > 0)
- String formats (NIT validity, product codes)
- Duplicate products
Data Persistence
The system does NOT persist data: No Local Storage:- No database writes
- No file system writes
- No cache or session storage
- Source: Corprecam MySQL database (read-only)
- Destination: Siigo Nube (write-only, via browser automation)
Data Security
In Transit
API Calls: All Corprecam APIs use HTTPS (https://corprecam.codesolutions.com.co)
Siigo Login: HTTPS with credentials transmitted via form fields
In Memory
Sensitive Data:- Siigo credentials stored in
configobject (loaded from.env) - Supplier NITs passed through function parameters
- Product prices and quantities in memory during processing
Logging
Currently logs to console:Data Transformation Rules
Company Assignment Logic
Rule: Based onMaterial.emp_id_fk
emp_id_fk | Company | NIT | Account |
|---|---|---|---|
| 1 | Corprecam | 900142913 | CAJA RIOHACHA |
| 2 (or other) | Reciclemos | 901328575 | Efectivo |
utils/transformDs.ts:50-60
Field Mapping
From Database to Automation:| Source | Field | Destination | Usage |
|---|---|---|---|
| Compra | comp_asociado | proveedor_id | Supplier search in Siigo |
| Micro | mic_nom | micro_id | Not used in automation (reference only) |
| Material | mat_codigo | codigo | Product search in Siigo |
| CompraItem | citem_cantidad | cantidad | Quantity field |
| CompraItem | citem_valor_unitario | precio | Unit value field |
Data Loss
Fields NOT Transferred to Siigo:com_codigo(purchase order number)citem_total(line total - calculated by Siigo)citem_rechazo(rejection quantity)mat_nom(product name - looked up by Siigo)mic_nom(route - not used)
Data Flow Error Scenarios
Missing Material
If aCompraItem references a non-existent mat_id:
Empty Product Arrays
If all products belong to one company:API Failure
If any API call fails (network error, 500 response):- Express middleware catches error
- Returns HTTP 500 to client
- No Playwright execution
- No partial data in Siigo
Output Data
HTTP Response
Format: JSON Success:server.ts:38-40
Meaning: Playwright automation completed without throwing errors. Does NOT indicate:
- Document was finalized in Siigo
- All products were added successfully
- Data accuracy
Siigo Side Effects
The actual output is a draft document in Siigo Nube containing:- Document type: “Documento soporte” (type 25470 for Corprecam)
- Supplier: Matched by NIT from
proveedor_id - Consecutive number: Auto-generated by Siigo
- Line items:
- Product code, description (from Siigo catalog)
- Warehouse: “BODEGA DE RIOHACHA” (Corprecam only)
- Quantity and unit price
- Payment account: “CAJA RIOHACHA” or “Efectivo”
ngrok Registration
On startup, the system writes its public URL to Corprecam: API:api/php.ts:78-89
Request: