Skip to main content

​
Architecture Philosophy

Tokenizador is built with a professional modular architecture that prioritizes maintainability, testability, and separation of concerns. The application follows object-oriented design principles with clear boundaries between different layers of functionality.
The architecture is designed to be simple yet scalable, using vanilla JavaScript with ES6 classes to avoid framework overhead while maintaining clean code organization.

​
System Architecture

The application follows a layered architecture pattern with four distinct layers:

​
Core Design Principles

Separation of Concerns

Each component has a single, well-defined responsibility. The UI layer never directly handles tokenization logic, and services never manipulate the DOM.

Dependency Injection

Components receive their dependencies through constructor injection, making the code more testable and flexible.

Event-Driven Communication

The UI layer communicates with the application layer through event handlers, creating loose coupling between components.

Configuration-Driven

Model data, encodings, and pricing are centralized in configuration files, making updates easy without code changes.

​
Project Structure

The codebase is organized into logical directories that reflect the architectural layers: 
trae/
├── index.html                    # Application entry point
├── styles.css                    # Global styles
├── js/
│   ├── token-analyzer.js         # Application orchestrator (main class)
│   ├── config/
│   │   └── models-config.js      # Model data, encodings, pricing
│   ├── services/
│   │   └── tokenization-service.js  # Tokenization business logic
│   ├── controllers/
│   │   └── ui-controller.js      # UI state and DOM manipulation
│   └── utils/
│       └── statistics-calculator.js # Statistics computation
This structure makes it easy to:
  • Locate code - Each file has a clear purpose and location
  • Test components - Business logic is isolated from UI concerns
  • Add features - New capabilities can be added without modifying existing code
  • Maintain consistency - Related code lives together

​
Component Interaction Flow

Here’s how the components work together during a typical tokenization operation:
1

User Input

User types text or changes model selection in the browser
2

Event Handling

UIController captures the DOM event and calls the registered event handler
3

Orchestration

TokenAnalyzer coordinates between services to process the request
4

Business Logic

TokenizationService performs the actual tokenization using tiktoken
5

Calculation

StatisticsCalculator computes metrics like cost and context utilization
6

Display Update

UIController updates the DOM with the results

​
Data Flow Architecture

Tokenizador uses unidirectional data flow for predictable state management:
Unidirectional flow means data always flows in one direction: from user input through processing layers to display output. This makes the application behavior predictable and easier to debug.

​
Initialization Sequence

When the application starts, components are initialized in a specific order: 
// From token-analyzer.js:9-15
constructor() {
    this.tokenizationService = new TokenizationService();
    this.uiController = new UIController();
    this.statisticsCalculator = new StatisticsCalculator();
    
    this.init();
}

// Create core services
this.tokenizationService = new TokenizationService();
this.uiController = new UIController();
this.statisticsCalculator = new StatisticsCalculator();

​
Error Handling Strategy

The architecture includes multiple levels of error handling:
  1. Service Level - TokenizationService has fallback tokenization when tiktoken fails
  2. Application Level - TokenAnalyzer catches errors and displays user-friendly messages
  3. UI Level - UIController validates input and handles DOM errors gracefully
// From token-analyzer.js:101-104
try {
    // ... tokenization logic
} catch (error) {
    console.error('Error durante el análisis:', error);
    this.showError('Error al analizar el texto. Por favor, inténtalo de nuevo.');
}
The application never crashes on tokenization errors. If tiktoken fails, the system automatically falls back to approximation-based tokenization.

​
State Management

Tokenizador uses a simple state management approach:
  • No global state - Each component manages its own internal state
  • UI as source of truth - Form inputs (text, model selection) are the source of truth
  • Computed values - Statistics and tokens are computed on-demand, not stored
// From token-analyzer.js:179-186
getState() {
    return {
        currentText: this.uiController.getTextInput(),
        selectedModel: this.uiController.getSelectedModel(),
        isInitialized: this.tokenizationService.isInitialized,
        availableModels: Object.keys(MODELS_DATA)
    };
}

​
Extensibility Points

The architecture makes it easy to extend functionality:
Simply add an entry to MODELS_DATA in models-config.js. No code changes needed in services or UI.

​
Performance Considerations

The architecture includes several performance optimizations:

DOM Caching

UIController caches DOM element references in initializeElements() to avoid repeated queries

Async Operations

Tokenization runs asynchronously to avoid blocking the UI thread

Single Responsibility

Each component does one thing well, avoiding complex computations that could slow down the app

Direct Token Processing

Tokens are processed in a single pass without intermediate storage

​
Technology Stack

Tokenizador intentionally uses a minimal technology stack:
LayerTechnologyWhy?
CoreVanilla JavaScript (ES6+)No build step, no framework overhead
Tokenizationtiktoken (WASM)Industry-standard, same as OpenAI uses
ArchitectureES6 ClassesClean OOP design without frameworks
UINative DOM APIsFast, no virtual DOM overhead
StylingCSS Custom PropertiesThemeable without preprocessors
By avoiding frameworks, Tokenizador loads instantly and has zero runtime dependencies (except tiktoken for tokenization accuracy).

​
Next Steps

Component Details

Explore each component in detail with code examples

API Reference

View the complete API documentation

How to Use

Learn how to use Tokenizador

Contributing

Contribute to the project

Build docs developers (and LLMs) love

Get started for freeTalk to us