Asynchronous Programming

Asynchronous programming is fundamental to Node.js. Understanding the evolution from callbacks to Promises to async/await is essential for writing efficient, maintainable Node.js applications.

Why Asynchronous?

Node.js uses a single-threaded event loop to handle concurrent operations. Asynchronous programming prevents blocking the thread while waiting for I/O operations to complete.

// Synchronous (blocks the thread)
const data = fs.readFileSync('file.txt', 'utf8');
console.log(data);
console.log('After reading');

// Asynchronous (non-blocking)
fs.readFile('file.txt', 'utf8', (err, data) => {
  if (err) throw err;
  console.log(data);
});
console.log('After initiating read');
// Output: "After initiating read" appears first

Synchronous operations block the entire event loop, preventing Node.js from processing other requests. Use async operations for I/O-bound tasks.

Callback Pattern

The callback pattern is the original approach to asynchronous programming in Node.js.

Error-First Callbacks

Node.js uses the error-first callback convention:

function callback(err, result) {
  if (err) {
    // Handle error
    return;
  }
  // Use result
}

The error-first callback pattern ensures errors are always handled explicitly. The first argument is always reserved for an error object (or null if no error), and subsequent arguments contain the result data.

Real Example from Node.js Core

// From lib/fs.js - simplified
function readFile(path, options, callback) {
  callback = maybeCallback(callback || options);
  options = getOptions(options, {});
  
  const req = new FSReqCallback();
  req.oncomplete = callback;
  
  binding.readFile(path, options.encoding, req);
}

// Usage
fs.readFile('file.txt', 'utf8', (err, data) => {
  if (err) {
    console.error('Error reading file:', err);
    return;
  }
  console.log(data);
});

Callback Hell

Nested callbacks can lead to deeply indented, hard-to-read code:

// Callback hell (pyramid of doom)
fs.readFile('file1.txt', (err, data1) => {
  if (err) throw err;
  fs.readFile('file2.txt', (err, data2) => {
    if (err) throw err;
    fs.readFile('file3.txt', (err, data3) => {
      if (err) throw err;
      console.log(data1, data2, data3);
    });
  });
});

Solutions to Callback Hell

Named functions: Extract callbacks into named functions
Modularization: Break code into smaller modules
Control flow libraries: Use libraries like async.js
Promises: Use Promises or async/await (modern approach)

Promises

Promises provide a cleaner way to handle asynchronous operations and avoid callback hell.

Promise Basics

A Promise represents a value that may be available now, later, or never:

const promise = new Promise((resolve, reject) => {
  // Async operation
  const success = true;
  
  if (success) {
    resolve('Success!');
  } else {
    reject(new Error('Failed!'));
  }
});

promise
  .then(result => console.log(result))
  .catch(error => console.error(error));

Promise States

Pending

Initial state, operation not yet completed.

Fulfilled

Operation completed successfully, promise has a value.

Rejected

Operation failed, promise has a reason (error).

Once a Promise is fulfilled or rejected, it cannot change state. Promises are immutable after settlement.

Promise Chaining

Promises can be chained for sequential operations:

const fs = require('fs').promises;

fs.readFile('file1.txt', 'utf8')
  .then(data1 => {
    console.log('File 1:', data1);
    return fs.readFile('file2.txt', 'utf8');
  })
  .then(data2 => {
    console.log('File 2:', data2);
    return fs.readFile('file3.txt', 'utf8');
  })
  .then(data3 => {
    console.log('File 3:', data3);
  })
  .catch(err => {
    console.error('Error:', err);
  });

Promise Combinators

Node.js provides several ways to work with multiple promises:

// Promise.all - wait for all promises (fails fast)
Promise.all([
  fetch('url1'),
  fetch('url2'),
  fetch('url3')
])
  .then(results => console.log('All done:', results))
  .catch(err => console.error('One failed:', err));

// Promise.allSettled - wait for all, regardless of success/failure
Promise.allSettled([
  Promise.resolve('success'),
  Promise.reject('error'),
  Promise.resolve('another success')
])
  .then(results => {
    // [{ status: 'fulfilled', value: 'success' },
    //  { status: 'rejected', reason: 'error' },
    //  { status: 'fulfilled', value: 'another success' }]
  });

// Promise.race - returns first settled promise
Promise.race([
  timeout(1000),
  fetchData()
])
  .then(result => console.log('First:', result));

// Promise.any - returns first fulfilled promise
Promise.any([
  fetchFromServer1(),
  fetchFromServer2(),
  fetchFromServer3()
])
  .then(result => console.log('Fastest success:', result));

Use Promise.all() when you need all results and want to fail fast. Use Promise.allSettled() when you want all results regardless of individual failures.

Promisification

Converting callback-based APIs to Promises:

const { promisify } = require('util');
const fs = require('fs');

// Convert callback-based function to Promise
const readFileAsync = promisify(fs.readFile);

readFileAsync('file.txt', 'utf8')
  .then(data => console.log(data))
  .catch(err => console.error(err));

// From lib/internal/util.js - simplified promisify implementation
function promisify(original) {
  return function (...args) {
    return new Promise((resolve, reject) => {
      original.call(this, ...args, (err, ...values) => {
        if (err) {
          return reject(err);
        }
        resolve(values.length === 1 ? values[0] : values);
      });
    });
  };
}

Async/Await

Async/await is syntactic sugar over Promises, making asynchronous code look synchronous.

Async Functions

An async function always returns a Promise:

async function fetchData() {
  return 'data';
}

// Equivalent to:
function fetchData() {
  return Promise.resolve('data');
}

fetchData().then(data => console.log(data));

Await Expressions

The await keyword pauses execution until a Promise settles:

const fs = require('fs').promises;

async function readFiles() {
  try {
    const data1 = await fs.readFile('file1.txt', 'utf8');
    console.log('File 1:', data1);
    
    const data2 = await fs.readFile('file2.txt', 'utf8');
    console.log('File 2:', data2);
    
    const data3 = await fs.readFile('file3.txt', 'utf8');
    console.log('File 3:', data3);
  } catch (err) {
    console.error('Error:', err);
  }
}

readFiles();

await can only be used inside async functions (or at the top level in ES modules). It pauses the function execution but does not block the event loop.

Parallel Execution with Async/Await

// Sequential (slow) - operations run one after another
async function sequential() {
  const result1 = await operation1(); // Wait for operation1
  const result2 = await operation2(); // Then wait for operation2
  const result3 = await operation3(); // Then wait for operation3
  return [result1, result2, result3];
}

// Parallel (fast) - operations run concurrently
async function parallel() {
  const [result1, result2, result3] = await Promise.all([
    operation1(),
    operation2(),
    operation3()
  ]);
  return [result1, result2, result3];
}

Using await in a loop executes operations sequentially. Use Promise.all() for parallel execution when operations are independent.

Real-World Example

const fs = require('fs').promises;
const path = require('path');

async function processDirectory(dirPath) {
  try {
    // Read directory contents
    const files = await fs.readdir(dirPath);
    
    // Process files in parallel
    const results = await Promise.all(
      files.map(async (file) => {
        const filePath = path.join(dirPath, file);
        const stats = await fs.stat(filePath);
        
        if (stats.isFile()) {
          const content = await fs.readFile(filePath, 'utf8');
          return {
            name: file,
            size: stats.size,
            lines: content.split('\n').length
          };
        }
        return null;
      })
    );
    
    return results.filter(Boolean);
  } catch (err) {
    console.error('Error processing directory:', err);
    throw err;
  }
}

processDirectory('./src')
  .then(results => console.log(results))
  .catch(err => console.error(err));

Error Handling

Proper error handling is critical in asynchronous code.

Callback Error Handling

fs.readFile('file.txt', 'utf8', (err, data) => {
  if (err) {
    if (err.code === 'ENOENT') {
      console.error('File not found');
    } else {
      console.error('Error reading file:', err);
    }
    return;
  }
  // Process data
});

Promise Error Handling

fetchData()
  .then(data => processData(data))
  .then(result => saveResult(result))
  .catch(err => {
    // Handles errors from any step in the chain
    console.error('Error:', err);
  })
  .finally(() => {
    // Always executes, regardless of success or failure
    cleanup();
  });

Async/Await Error Handling

async function processData() {
  try {
    const data = await fetchData();
    const processed = await processIt(data);
    await saveData(processed);
    return processed;
  } catch (err) {
    if (err instanceof NetworkError) {
      console.error('Network error:', err.message);
      // Retry logic
    } else if (err instanceof ValidationError) {
      console.error('Validation error:', err.message);
      // Handle validation
    } else {
      console.error('Unexpected error:', err);
      throw err; // Re-throw unknown errors
    }
  } finally {
    // Cleanup code
    await closeConnection();
  }
}

Always use try/catch with async/await. Unhandled promise rejections in async functions won’t be caught by surrounding try/catch blocks unless you await them.

Unhandled Promise Rejections

// Listen for unhandled rejections
process.on('unhandledRejection', (reason, promise) => {
  console.error('Unhandled Rejection at:', promise, 'reason:', reason);
  // Application specific logging, throwing an error, or other logic
});

// Example of unhandled rejection
Promise.reject(new Error('Oops!')); // No .catch() handler

// This will trigger the unhandledRejection event

In future Node.js versions, unhandled promise rejections will terminate the process. Always handle promise rejections with .catch() or try/catch.

Advanced Patterns

Retry Logic

async function retryOperation(operation, maxRetries = 3, delay = 1000) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await operation();
    } catch (err) {
      if (i === maxRetries - 1) throw err;
      
      console.log(`Attempt ${i + 1} failed, retrying...`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

// Usage
retryOperation(() => fetch('https://api.example.com/data'))
  .then(data => console.log(data))
  .catch(err => console.error('All retries failed:', err));

Timeout Pattern

function timeout(ms) {
  return new Promise((_, reject) => {
    setTimeout(() => reject(new Error('Operation timed out')), ms);
  });
}

async function fetchWithTimeout(url, timeoutMs = 5000) {
  try {
    const result = await Promise.race([
      fetch(url),
      timeout(timeoutMs)
    ]);
    return result;
  } catch (err) {
    if (err.message === 'Operation timed out') {
      console.error('Request timed out');
    }
    throw err;
  }
}

Queue Processing

class AsyncQueue {
  constructor(concurrency = 1) {
    this.concurrency = concurrency;
    this.running = 0;
    this.queue = [];
  }
  
  async push(task) {
    this.queue.push(task);
    return this.run();
  }
  
  async run() {
    while (this.running < this.concurrency && this.queue.length > 0) {
      const task = this.queue.shift();
      this.running++;
      
      try {
        await task();
      } finally {
        this.running--;
        if (this.queue.length > 0) {
          this.run();
        }
      }
    }
  }
}

// Usage
const queue = new AsyncQueue(3); // Max 3 concurrent tasks

for (let i = 0; i < 10; i++) {
  queue.push(async () => {
    console.log(`Starting task ${i}`);
    await delay(1000);
    console.log(`Completed task ${i}`);
  });
}

Event Emitter to Promise

const EventEmitter = require('events');

function eventToPromise(emitter, successEvent, errorEvent) {
  return new Promise((resolve, reject) => {
    emitter.once(successEvent, resolve);
    emitter.once(errorEvent, reject);
  });
}

// Usage
const server = require('http').createServer();

async function startServer() {
  server.listen(3000);
  await eventToPromise(server, 'listening', 'error');
  console.log('Server started on port 3000');
}

Async Iteration

Async Iterators

const fs = require('fs');
const readline = require('readline');

async function processLineByLine(filePath) {
  const fileStream = fs.createReadStream(filePath);
  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity
  });
  
  // Using for await...of
  for await (const line of rl) {
    console.log(`Line: ${line}`);
    // Process line asynchronously
    await processLine(line);
  }
}

async function processLine(line) {
  // Async processing
  return new Promise(resolve => {
    setTimeout(() => resolve(), 100);
  });
}

Custom Async Iterator

class AsyncRange {
  constructor(start, end, delay = 100) {
    this.start = start;
    this.end = end;
    this.delay = delay;
  }
  
  async *[Symbol.asyncIterator]() {
    for (let i = this.start; i <= this.end; i++) {
      await new Promise(resolve => setTimeout(resolve, this.delay));
      yield i;
    }
  }
}

// Usage
async function example() {
  for await (const num of new AsyncRange(1, 5)) {
    console.log(num);
  }
}

Async iterators enable processing streams of data asynchronously. They’re particularly useful for reading large files, processing database result sets, or consuming API pagination.

Best Practices

Prefer async/await over callbacks

Modern code should use async/await for better readability and error handling.

Handle all errors

Always use try/catch with async/await and .catch() with Promises.

Use Promise.all() for parallel operations

Don’t await in loops if operations can run concurrently.

Set timeouts for network operations

Prevent operations from hanging indefinitely.

Avoid mixing patterns

Don’t mix callbacks, Promises, and async/await unnecessarily.

Event Loop - Understanding async execution
Architecture - How async fits in Node.js architecture
Modules - Async module loading

Getting Started

Core Concepts

Guides

Advanced

Asynchronous Programming

Why Asynchronous?

Callback Pattern

Error-First Callbacks

Real Example from Node.js Core

Callback Hell

Promises

Promise Basics

Promise States

Promise Chaining

Promise Combinators

Promisification

Async/Await

Async Functions

Await Expressions

Parallel Execution with Async/Await

Real-World Example

Error Handling

Callback Error Handling

Promise Error Handling

Async/Await Error Handling

Unhandled Promise Rejections

Advanced Patterns

Retry Logic

Timeout Pattern

Queue Processing

Event Emitter to Promise

Async Iteration

Async Iterators

Custom Async Iterator

Best Practices

Getting Started

Core Concepts

Guides

Advanced

​Why Asynchronous?

​Callback Pattern

​Error-First Callbacks

​Real Example from Node.js Core

​Callback Hell

​Promises

​Promise Basics

​Promise States

​Promise Chaining

​Promise Combinators

​Promisification

​Async/Await

​Async Functions

​Await Expressions

​Parallel Execution with Async/Await

​Real-World Example

​Error Handling

​Callback Error Handling

​Promise Error Handling

​Async/Await Error Handling

​Unhandled Promise Rejections

​Advanced Patterns

​Retry Logic

​Timeout Pattern

​Queue Processing

​Event Emitter to Promise

​Async Iteration

​Async Iterators

​Custom Async Iterator

​Best Practices

​Related Topics

Why Asynchronous?

Callback Pattern

Error-First Callbacks

Real Example from Node.js Core

Callback Hell

Promises

Promise Basics

Promise States

Promise Chaining

Promise Combinators

Promisification

Async/Await

Async Functions

Await Expressions

Parallel Execution with Async/Await

Real-World Example

Error Handling

Callback Error Handling

Promise Error Handling

Async/Await Error Handling

Unhandled Promise Rejections

Advanced Patterns

Retry Logic

Timeout Pattern

Queue Processing

Event Emitter to Promise

Async Iteration

Async Iterators

Custom Async Iterator

Best Practices

Related Topics