Broadcasting Semantics

Broadcasting is a powerful mechanism that allows operations on tensors with different shapes without explicit replication. Deepbox follows NumPy-style broadcasting rules for efficient memory usage and clean code.

What is Broadcasting?

Broadcasting automatically expands smaller tensors to match larger ones during element-wise operations:

import { tensor, add } from 'deepbox/ndarray';

// Matrix + vector (broadcasts)
const matrix = tensor([
  [1, 2, 3],
  [4, 5, 6],
]);
const vector = tensor([10, 20, 30]);

const result = add(matrix, vector);
// [[11, 22, 33],
//  [14, 25, 36]]

Without broadcasting, you’d need to manually replicate the vector:

// Manual approach (inefficient)
const repeated = tensor([
  [10, 20, 30],
  [10, 20, 30],
]);
const result = add(matrix, repeated);

Broadcasting avoids memory allocation for intermediate arrays, making operations faster and more memory-efficient.

Broadcasting Rules

Two tensors are broadcastable if, for each dimension (starting from the trailing dimensions):

The dimensions are equal, OR
One dimension is 1, OR
One dimension doesn’t exist (can be prepended)

Rule 1: Equal Dimensions

import { add, tensor } from 'deepbox/ndarray';

const a = tensor([[1, 2], [3, 4]]);  // shape: [2, 2]
const b = tensor([[5, 6], [7, 8]]);  // shape: [2, 2]

const c = add(a, b);
// shape: [2, 2]
// Both shapes match exactly

Rule 2: One Dimension is 1

import { add, tensor } from 'deepbox/ndarray';

const a = tensor([[1, 2, 3]]);        // shape: [1, 3]
const b = tensor([[10], [20], [30]]); // shape: [3, 1]

const c = add(a, b);
// shape: [3, 3]
// a broadcasts along axis 0
// b broadcasts along axis 1
// Result:
// [[11, 12, 13],
//  [21, 22, 23],
//  [31, 32, 33]]

Rule 3: Dimension Doesn’t Exist

import { add, tensor } from 'deepbox/ndarray';

const matrix = tensor([[1, 2], [3, 4]]);  // shape: [2, 2]
const vector = tensor([10, 20]);          // shape: [2]

const result = add(matrix, vector);
// shape: [2, 2]
// vector is treated as shape [1, 2] (prepend 1)
// Then broadcasts to [2, 2]

Broadcasting Examples

Scalar Broadcasting

Scalars (rank-0 tensors) broadcast to any shape:

import { add, tensor } from 'deepbox/ndarray';

const matrix = tensor([
  [1, 2, 3],
  [4, 5, 6],
]);
const scalar = tensor(10);  // shape: []

const result = add(matrix, scalar);
// [[11, 12, 13],
//  [14, 15, 16]]

Helper functions like addScalar(matrix, 10) are provided for common scalar operations, but they internally use broadcasting.

Vector + Matrix

import { add, tensor } from 'deepbox/ndarray';

// Row vector broadcasts along rows
const matrix = tensor([
  [1, 2, 3],
  [4, 5, 6],
]);  // [2, 3]

const row = tensor([10, 20, 30]);  // [3]

const result = add(matrix, row);
// [[11, 22, 33],
//  [14, 25, 36]]

Column Vector + Matrix

import { add, tensor } from 'deepbox/ndarray';

const matrix = tensor([
  [1, 2, 3],
  [4, 5, 6],
]);  // [2, 3]

const column = tensor([[100], [200]]);  // [2, 1]

const result = add(matrix, column);
// [[101, 102, 103],
//  [204, 205, 206]]

3D Broadcasting

import { mul, tensor } from 'deepbox/ndarray';

const batch = tensor([
  [[1, 2], [3, 4]],
  [[5, 6], [7, 8]],
]);  // [2, 2, 2]

const weights = tensor([[0.1, 0.2]]);  // [1, 2]

const result = mul(batch, weights);
// shape: [2, 2, 2]
// weights broadcasts to [2, 2, 2]

How Broadcasting Works

Alignment

Shapes are right-aligned and compared from right to left:

// Example: [3, 1, 5] + [2, 5]
//          [3, 1, 5]
//             [2, 5]  ← right-aligned
//          ---------
//          [3, 2, 5]  ← result shape

Dimension Expansion

Missing dimensions are treated as 1:

// [2, 3] is expanded to [1, 2, 3]
// [5] is expanded to [1, 1, 5]

Step-by-Step Example

import { add, tensor } from 'deepbox/ndarray';

const a = tensor([[[1, 2]]]);  // shape: [1, 1, 2]
const b = tensor([[3], [4]]);  // shape: [2, 1]

// Step 1: Align shapes
// a: [1, 1, 2]
// b:    [2, 1]  → prepend 1 → [1, 2, 1]

// Step 2: Compare dimensions
// dim 0: 1 vs 1 ✓ (equal)
// dim 1: 1 vs 2 ✓ (a has 1, can broadcast)
// dim 2: 2 vs 1 ✓ (b has 1, can broadcast)

// Step 3: Result shape
// [1, 2, 2]

const result = add(a, b);
console.log(result.shape);  // [1, 2, 2]

Broadcast Errors

Broadcasting fails when dimensions are incompatible:

import { add, tensor } from 'deepbox/ndarray';

try {
  const a = tensor([[1, 2, 3]]);  // shape: [1, 3]
  const b = tensor([[1, 2]]);     // shape: [1, 2]

  add(a, b);
  // ShapeError: Cannot broadcast [1, 3] with [1, 2]
  // dim 1: 3 vs 2 (neither is 1)
} catch (err) {
  console.error(err.message);
}

Common Incompatibilities

// ❌ Cannot broadcast
[3, 4] + [4, 3]  // Neither dimension matches
[2, 3] + [2]     // Would need [1, 2] or [2, 1]
[3, 5] + [3, 4]  // Last dim: 5 vs 4

// ✅ Can broadcast
[3, 4] + [4]     // [4] → [1, 4] → broadcasts
[3, 4] + [1, 4]  // Direct match
[3, 4] + [3, 1]  // Last dim broadcasts

Operations Supporting Broadcasting

All element-wise operations support broadcasting:

Arithmetic

import { add, sub, mul, div, tensor } from 'deepbox/ndarray';

const a = tensor([[1, 2], [3, 4]]);
const b = tensor([10, 20]);

add(a, b)  // Addition
sub(a, b)  // Subtraction
mul(a, b)  // Multiplication
div(a, b)  // Division

Comparison

import { equal, greater, less, tensor } from 'deepbox/ndarray';

const x = tensor([[1, 2], [3, 4]]);
const threshold = tensor([2]);

greater(x, threshold)  // [[false, false], [true, true]]
less(x, threshold)     // [[true, false], [false, false]]
equal(x, threshold)    // [[false, true], [false, false]]

Logical

import { logicalAnd, logicalOr, tensor } from 'deepbox/ndarray';

const a = tensor([[1, 0], [1, 1]], { dtype: 'bool' });
const b = tensor([1, 0], { dtype: 'bool' });

logicalAnd(a, b)  // [[true, false], [true, false]]
logicalOr(a, b)   // [[true, false], [true, true]]

Mathematical

import { pow, maximum, minimum, tensor } from 'deepbox/ndarray';

const x = tensor([[1, 2], [3, 4]]);
const exponent = tensor([2]);

pow(x, exponent)        // Element-wise power
maximum(x, tensor([2])) // Element-wise max
minimum(x, tensor([5])) // Element-wise min

Broadcasting in Autograd

Gradients automatically handle broadcasting:

import { parameter, GradTensor, tensor } from 'deepbox/ndarray';

const a = parameter([[1], [2]]);  // shape: [2, 1]
const b = GradTensor.fromTensor(
  tensor([[1, 2, 3]]),
  { requiresGrad: false }
);  // shape: [1, 3]

// Broadcasting: [2, 1] + [1, 3] → [2, 3]
const c = a.add(b);
const loss = c.sum();

loss.backward();

// Gradient is reduced back to original shape
console.log(a.shape);       // [2, 1]
console.log(a.grad?.shape); // [2, 1] (not [2, 3])

Gradients are summed over broadcasted dimensions to match the original parameter shape. This ensures correct gradient flow.

Performance Considerations

Memory Efficiency

Broadcasting avoids allocating intermediate arrays:

import { add, tensor } from 'deepbox/ndarray';

const large = tensor([[...], [...]])  // [1000, 1000]
const small = tensor([1, 2, 3, ...])  // [1000]

// Efficient: No memory allocation for broadcasting
const result = add(large, small);

// Inefficient: Would allocate [1000, 1000] for repeated small
// const repeated = tile(small, [1000, 1]);
// const result = add(large, repeated);

Fast Path Optimization

Deepbox uses optimized code paths for common patterns:

// Fast path: contiguous same-shape arrays
const a = tensor([[1, 2], [3, 4]]);
const b = tensor([[5, 6], [7, 8]]);
add(a, b);  // Optimized loop

// General path: broadcasting logic
const c = tensor([[1, 2]]);
add(a, c);  // Slower, handles broadcasting

When possible, pre-reshape tensors to matching shapes for best performance in tight loops.

Advanced Broadcasting

Outer Product

Broadcasting enables outer products:

import { mul, tensor } from 'deepbox/ndarray';

const a = tensor([[1], [2], [3]]);  // [3, 1]
const b = tensor([[4, 5, 6]]);      // [1, 3]

const outer = mul(a, b);
// [[4, 5, 6],
//  [8, 10, 12],
//  [12, 15, 18]]

Batch Operations

Broadcast over batch dimension:

import { add, tensor } from 'deepbox/ndarray';

const batched = tensor([
  [[1, 2], [3, 4]],
  [[5, 6], [7, 8]],
]);  // [2, 2, 2] (batch_size=2)

const bias = tensor([10, 20]);  // [2]

// Broadcast bias to all samples in batch
const result = add(batched, bias);
// [[[11, 22], [13, 24]],
//  [[15, 26], [17, 28]]]

Dimension Expansion

Use expandDims to prepare for broadcasting:

import { expandDims, mul, tensor } from 'deepbox/ndarray';

const a = tensor([1, 2, 3]);  // [3]

// Add dimension at axis 0
const b = expandDims(a, 0);  // [1, 3]

// Add dimension at axis 1
const c = expandDims(a, 1);  // [3, 1]

// Now can broadcast
const result = mul(b, c);
// [[1, 2, 3],
//  [2, 4, 6],
//  [3, 6, 9]]

Debugging Broadcasting

Check broadcast compatibility:

import { getBroadcastShape, canBroadcast } from 'deepbox/ndarray/ops/broadcast';

// Internal utilities (not exported, for reference)
const shapeA = [3, 1, 5];
const shapeB = [2, 5];

if (canBroadcast(shapeA, shapeB)) {
  const resultShape = getBroadcastShape(shapeA, shapeB);
  console.log(resultShape);  // [3, 2, 5]
} else {
  console.error('Cannot broadcast');
}

Visualization

Understand broadcasting visually:

// Matrix [2, 3]
// [ 1  2  3 ]
// [ 4  5  6 ]

// Vector [3]
// [ 10  20  30 ]

// Broadcasting vector to [2, 3]:
// [ 10  20  30 ]  ← repeat
// [ 10  20  30 ]

// Result:
// [ 11  22  33 ]
// [ 14  25  36 ]

Get Started

Core Concepts

Modules

​Broadcasting Semantics

​What is Broadcasting?

​Broadcasting Rules

​Rule 1: Equal Dimensions

​Rule 2: One Dimension is 1

​Rule 3: Dimension Doesn’t Exist

​Broadcasting Examples

​Scalar Broadcasting

​Vector + Matrix

​Column Vector + Matrix

​3D Broadcasting

​How Broadcasting Works

​Alignment

​Dimension Expansion

​Step-by-Step Example

​Broadcast Errors

​Common Incompatibilities

​Operations Supporting Broadcasting

​Arithmetic

​Comparison

​Logical

​Mathematical

​Broadcasting in Autograd

​Performance Considerations

​Memory Efficiency

​Fast Path Optimization

​Advanced Broadcasting

​Outer Product

​Batch Operations

​Dimension Expansion

​Debugging Broadcasting

​Visualization

​Next Steps

Tensors

Autograd

​API Reference

Operations

Build docs developers (and LLMs) love

Broadcasting Semantics

What is Broadcasting?

Broadcasting Rules

Rule 1: Equal Dimensions

Rule 2: One Dimension is 1

Rule 3: Dimension Doesn’t Exist

Broadcasting Examples

Scalar Broadcasting

Vector + Matrix

Column Vector + Matrix

3D Broadcasting

How Broadcasting Works

Alignment

Dimension Expansion

Step-by-Step Example

Broadcast Errors

Common Incompatibilities

Operations Supporting Broadcasting

Arithmetic

Comparison

Logical

Mathematical

Broadcasting in Autograd

Performance Considerations

Memory Efficiency

Fast Path Optimization

Advanced Broadcasting

Outer Product

Batch Operations

Dimension Expansion

Debugging Broadcasting

Visualization

Next Steps

API Reference