Skip to main content
The PromptCache class provides an in-memory LRU cache with background refresh functionality, specifically designed for caching prompts from the LangSmith Hub.

​
PromptCache

​
Constructor

import { PromptCache } from "langsmith";

const cache = new PromptCache({
  maxSize: 100,
  ttlSeconds: 3600,
  refreshIntervalSeconds: 60,
});
​
config
CacheConfig
Configuration options for the cache
​
maxSize
number
default:"100"
Maximum entries in cache. LRU eviction when exceeded. Set to 0 to disable caching.
​
ttlSeconds
number | null
default:"3600"
Time in seconds before entry is stale. Set to null for infinite TTL.
​
refreshIntervalSeconds
number
default:"60"
How often to check for stale entries in seconds.

​
Methods

​
get()

Get a value from cache. 
const prompt = cache.get(
  "my-prompt:latest",
  async () => await client.getPrompt("my-prompt:latest")
);
​
key
string
required
The cache key.
​
refreshFunc
() => Promise<PromptCommit>
required
Function to refresh the value when stale.
​
PromptCommit | undefined
PromptCommit | undefined
The cached value or undefined if not found. Stale entries are still returned (background refresh handles updates).

​
set()

Set a value in the cache. 
cache.set(
  "my-prompt:latest",
  promptCommit,
  async () => await client.getPrompt("my-prompt:latest")
);
​
key
string
required
The cache key.
​
value
PromptCommit
required
The value to cache.
​
refreshFunc
() => Promise<PromptCommit>
required
Function to refresh the value when stale.

​
invalidate()

Remove a specific entry from cache. 
cache.invalidate("my-prompt:latest");

​
clear()

Clear all cache entries. 
cache.clear();

​
stop()

Stop background refresh. Should be called when the cache is being cleaned up. 
cache.stop();

​
dump()

Dump cache contents to a JSON file for offline use. 
cache.dump("/path/to/cache.json");
​
filePath
string
required
Path to save the cache file.

​
load()

Load cache contents from a JSON file. Loaded entries get a fresh TTL starting from load time. 
const numLoaded = cache.load("/path/to/cache.json");
​
filePath
string
required
Path to the cache file.
​
number
number
Number of entries loaded.

​
configure()

Reconfigure the cache. 
cache.configure({
  maxSize: 200,
  ttlSeconds: 7200,
});

​
Properties

​
size
number
The number of entries in the cache.
​
metrics
CacheMetrics
Cache performance metrics.
interface CacheMetrics {
  hits: number;
  misses: number;
  refreshes: number;
  refreshErrors: number;
}
​
totalRequests
number
Total cache requests (hits + misses).
​
hitRate
number
Cache hit rate (0.0 to 1.0).

​
resetMetrics()

Reset all metrics to zero. 
cache.resetMetrics();

​
Global prompt cache

The SDK provides a global singleton cache for prompts.

​
configureGlobalPromptCache()

Configure the global prompt cache. 
import { configureGlobalPromptCache } from "langsmith";

configureGlobalPromptCache({
  maxSize: 200,
  ttlSeconds: 7200,
  refreshIntervalSeconds: 120,
});
​
config
CacheConfig
required
Cache configuration options.

​
Client integration

By default, the LangSmith client uses the global prompt cache. You can disable it per-client: 
import { Client } from "langsmith";

const client = new Client({
  disablePromptCache: true,
});

​
Features

​
LRU eviction

When the cache reaches maxSize, the least recently used entry is evicted: 
const cache = new PromptCache({ maxSize: 3 });

cache.set("key1", value1, refresh1);
cache.set("key2", value2, refresh2);
cache.set("key3", value3, refresh3);

// This will evict key1
cache.set("key4", value4, refresh4);

​
Stale-while-revalidate

Stale entries are still returned while being refreshed in the background: 
const cache = new PromptCache({
  ttlSeconds: 60,
  refreshIntervalSeconds: 10,
});

// Initial fetch
const prompt1 = cache.get("my-prompt", fetchPrompt);

// After 70 seconds, entry is stale but still returned
const prompt2 = cache.get("my-prompt", fetchPrompt);

// Background refresh happens automatically

​
Background refresh

The cache automatically refreshes stale entries in the background: 
const cache = new PromptCache({
  ttlSeconds: 300, // 5 minutes
  refreshIntervalSeconds: 60, // Check every minute
});

// Cache will check for stale entries every 60 seconds
// and refresh them if TTL has expired

​
Persistence

Dump and load cache for offline use: 
const cache = new PromptCache();

// Populate cache
cache.set("prompt1", value1, refresh1);
cache.set("prompt2", value2, refresh2);

// Save to disk
cache.dump("./prompts.cache.json");

// Later, in another process
const newCache = new PromptCache();
const numLoaded = newCache.load("./prompts.cache.json");
console.log(`Loaded ${numLoaded} prompts`);

​
Example usage

​
Basic usage

import { PromptCache, Client } from "langsmith";

const cache = new PromptCache({
  maxSize: 100,
  ttlSeconds: 3600,
});

const client = new Client();

const getPrompt = async (key: string) => {
  const cached = cache.get(key, async () => {
    return await client.getPrompt(key);
  });
  
  if (cached) {
    console.log("Cache hit!");
    return cached;
  }
  
  console.log("Cache miss, fetching...");
  const prompt = await client.getPrompt(key);
  cache.set(key, prompt, async () => await client.getPrompt(key));
  return prompt;
};

const prompt = await getPrompt("my-prompt:latest");

​
With metrics

const cache = new PromptCache();

// Use the cache
for (let i = 0; i < 100; i++) {
  cache.get(`prompt-${i % 10}`, fetchPrompt);
}

// Check metrics
console.log(`Hit rate: ${(cache.hitRate * 100).toFixed(1)}%`);
console.log(`Total requests: ${cache.totalRequests}`);
console.log(`Metrics:`, cache.metrics);

​
Disable caching

Set maxSize to 0 to disable caching: 
const cache = new PromptCache({ maxSize: 0 });

// This will always return undefined
const result = cache.get("key", fetchFunc);

​
Cache (deprecated)

The Cache class is deprecated. Use PromptCache instead.
import { Cache } from "langsmith";

// This will log a deprecation warning
const cache = new Cache();

Build docs developers (and LLMs) love

Get started for freeTalk to us