Memory Graph

REMem organizes information as a hybrid memory graph that combines structured knowledge (entities, facts) with episodic traces (conversation history, gist summaries) and semantic similarity.

Graph Philosophy

Why a graph? Traditional RAG systems retrieve passages using dense search alone. REMem enhances this by:

Structured memory: Extracting entities and facts creates explicit knowledge representation
Associative recall: Graph edges enable multi-hop reasoning across related concepts
Episodic memory: Gist summaries provide compressed, context-aware representations
Semantic clustering: Synonymy edges connect similar entities for better coverage

This hybrid approach mirrors human memory: we remember both specific facts and general impressions.

Node Types

The graph contains different node types depending on the extraction method:

Standard OpenIE Nodes

Entities (prefix: entity-):

Named entities extracted from triples (subjects and objects)
Example: “Alan Turing”, “Turing Test”, “1950”
Embedded for semantic search

Facts (prefix: fact-):

Subject-predicate-object triples
Example: “(Alan Turing, proposed, Turing Test)”
Stored as strings for embedding: “Alan Turing proposed Turing Test”

Passages (prefix: chunk hash):

Original document chunks
Connected to entities mentioned within them
Used for final answer generation

Paraphrases (prefix: paraphrase-, only for episodic_gist):

Chunk-level summaries
Connected to their source chunks

Episodic Gist Nodes

The episodic_gist extraction method creates a richer graph structure: Verbatim (prefix: verbatim-):

Original conversation turns or document chunks
Can be split per message if split_verbatim_per_chunk=True
Contains full context with speaker roles and timestamps

Gists (prefix: gists-):

Paraphrased summaries of verbatim content
Multiple gists per chunk for different aspects
Can be concatenated if concatenate_gists_per_chunk=True
Example: “User asked about implementing a search feature”

Facts (prefix: facts-):

Structured knowledge extracted from conversations
Includes qualifiers like time, location, emotion
Format: {"subject": "User", "predicate": "wants to implement", "object": "search feature", "qualifiers": {"time": "2024-01-15"}}

Entities (prefix: entity-):

Named entities from facts (subjects and objects)
Used as connection points in the graph

From episodic_gist_strategy.py:389:

entries = {"verbatim", "gists", "entity", "facts"}
all_nodes = {}
for entry in entries:
    if entry in self.remem.episodic_embedding_stores:
        all_nodes.update(self.remem.episodic_embedding_stores[entry].hash_id_to_row)

Temporal Nodes

The temporal extraction method focuses on time-aware facts:

Verbatim: Original text chunks
Facts: Triples with temporal qualifiers
Entities: Named entities from facts

Temporal facts include time anchors in their qualifiers for chronological reasoning.

Edge Types

Edges connect nodes based on different relationships:

Fact Edges (Entity → Entity)

Connect subject and object entities from extracted triples:

# From remem.py:924-932
subject_key = compute_mdhash_id(content=triple[0], prefix=("entity-"))
object_key = compute_mdhash_id(content=triple[2], prefix=("entity-"))

self.node_to_node_count[(subject_key, object_key)] = \
    self.node_to_node_count.get((subject_key, object_key), 0.0) + 1

Edge weight increases when the same relationship appears multiple times.

Context Edges (Passage → Entity)

Connect passages to entities mentioned within them:

# From remem.py:978
self.node_to_node_count[(chunk_key, node_key)] = 1.0

This allows retrieval to start from relevant entities and navigate to source passages.

Synonymy Edges (Entity ↔ Entity)

Connect semantically similar entities based on embedding similarity:

# From remem.py:1027-1035
query_node_key2knn_node_keys = retrieve_knn(
    query_ids=node_keys,
    key_ids=node_keys,
    query_vecs=node_embeddings,
    key_vecs=node_embeddings,
    k=self.global_config.synonymy_edge_topk,
)

Only pairs with similarity > synonymy_edge_sim_threshold (default 0.8) are connected. Why synonymy edges? They help with:

Spelling variations: “Alan Turing” ↔ “A. Turing”
Abbreviations: “USA” ↔ “United States”
Related concepts: “search” ↔ “retrieval”

Episodic Gist Edges

For episodic_gist extraction, additional edge types connect the graph: Verbatim → Gist (episodic_gist_strategy.py:267-271):

for verbatim_key in verbatim_keys:
    for gist_key in gist_keys:
        self.remem.node_to_node_count[(verbatim_key, gist_key)] = \
            self.remem.node_to_node_count.get((verbatim_key, gist_key), 0) + 1

Verbatim → Entity (episodic_gist_strategy.py:309-311):

for verbatim_key in verbatim_keys:
    self.remem.node_to_node_count[(verbatim_key, subject_key)] = 1.0
    self.remem.node_to_node_count[(verbatim_key, object_key)] = 1.0

Gist → Entity (episodic_gist_strategy.py:314-320):

for gist_key in gist_keys:
    self.remem.node_to_node_count[(gist_key, object_key)] = \
        self.remem.node_to_node_count.get((gist_key, object_key), 0) + 1
    self.remem.node_to_node_count[(gist_key, subject_key)] = \
        self.remem.node_to_node_count.get((gist_key, subject_key), 0) + 1

Gist → Fact (episodic_gist_strategy.py:323-325):

self.remem.node_to_node_count[(gist_key, fact_key)] = \
    self.remem.node_to_node_count.get((gist_key, fact_key), 0) + 1

This creates a rich structure:

Verbatim ──→ Gist ──→ Entity
           │         ↗  ↓
           └──→ Fact → Entity

Graph Construction

The graph is built incrementally during indexing:

1. Extract Structure

Depending on the extraction method, different units are extracted:

# For openie (remem.py:354-359)
ie_results = self.openie.batch_openie(new_openie_rows)
ner_results_dict, triple_results_dict = ie_results[0], ie_results[1]

# For episodic_gist (episodic_gist_strategy.py:78-79)
ie_results = self.remem.openie.batch_openie(new_openie_rows)
self.merge_gist_extraction_results(all_openie_info, chunk_keys_to_process, ie_results)

2. Create Node Embeddings

Each node type gets embedded for semantic search:

# Entities
self.phrase_embedding_store.insert_strings(phrase_nodes)

# Facts
self.triple_embedding_store.insert_strings([str(fact) for fact in flattened_triples])

# Gists (episodic_gist only)
self.remem.episodic_embedding_stores["gists"].insert_strings(element_to_encode["gists"])

3. Add Nodes to Graph

New nodes are added with attributes (episodic_gist_strategy.py:437-438):

for node_hash_id, node in all_nodes.items():
    node["name"] = node_hash_id
    node["label"] = node.get("content", node_hash_id)  # Human-readable label

self.remem.graph.add_vertices(n=len(next(iter(new_nodes.values()))), attributes=new_nodes)

4. Add Edges

Edges are created from the node_to_node_count statistics (episodic_gist_strategy.py:460-478):

for (source_node, target_node), weight in self.remem.node_to_node_count.items():
    source_idx = self.remem.node_name_to_vertex_idx[source_node]
    target_idx = self.remem.node_name_to_vertex_idx[target_node]
    edges_to_add.append((source_idx, target_idx))
    weights_to_add.append(weight)

self.remem.graph.add_edges(edges_to_add, {"weight": weights_to_add})

5. Augment with Synonymy

Synonymy edges are added using KNN search:

# From episodic_gist_strategy.py:356-362
for entry in ["gists"]:  # Can include other node types
    node_keys = self.remem.episodic_embedding_stores[entry].get_all_ids()
    embeddings = list(self.remem.episodic_embedding_stores[entry].get_embeddings(node_keys))
    self.remem.add_synonymy_edges(
        embeddings, node_keys, self.remem.episodic_embedding_stores[entry].hash_id_to_row
    )

6. Persist

The graph is saved to disk for reuse:

self.remem.save_igraph()  # Saves to working_dir/graph.pkl

Graph Statistics

You can inspect the graph structure:

graph_info = rag_strategy.get_graph_info()
print(graph_info)
# {
#   "num_phrase_nodes": 1247,
#   "num_passage_nodes": 150,
#   "num_paraphrase_nodes": 150,  # If using episodic_gist
#   "num_total_nodes": 1397,
#   "num_extracted_edges": 892,
#   "num_context_edges": 1845,
#   "num_paraphrase_edges": 150,
#   "num_synonymy_edges": 3421,
#   "num_total_edges": 6308
# }

Configuration Options

Control graph construction with these config parameters:

config = BaseConfig(
    # Synonymy edge tuning
    synonymy_edge_topk=2047,  # Max neighbors per node
    synonymy_edge_sim_threshold=0.8,  # Min similarity to connect
    
    # Graph structure
    is_directed_graph=False,  # Undirected by default
    
    # Episodic gist options
    concatenate_gists_per_chunk=False,  # Single vs multiple gist nodes
    split_verbatim_per_chunk=True,  # Split conversations by message
)

Higher synonymy_edge_topk creates denser graphs but increases memory usage and retrieval time.

Why This Structure?

The hybrid graph addresses key RAG challenges: Multi-hop reasoning: Navigate entity→entity→passage to answer “Who proposed the test that Turing created?” Semantic clustering: Synonymy edges connect “AI” and “artificial intelligence” even if embeddings differ slightly Episodic context: Gist nodes provide compressed summaries for long conversations without losing the verbatim details Temporal awareness: Temporal qualifiers enable “What happened after X?” queries Efficient retrieval: Graph structure reduces search space compared to brute-force passage retrieval

Next Steps

Learn about Extraction Methods that populate the graph
Understand Retrieval Strategies that navigate the graph
Review Architecture for the full pipeline

Get Started

Core Concepts

Guides

Customization

Benchmarks

Memory Graph

Graph Philosophy

Node Types

Standard OpenIE Nodes

Episodic Gist Nodes

Temporal Nodes

Edge Types

Fact Edges (Entity → Entity)

Context Edges (Passage → Entity)

Synonymy Edges (Entity ↔ Entity)

Episodic Gist Edges

Graph Construction

1. Extract Structure

2. Create Node Embeddings

3. Add Nodes to Graph

4. Add Edges

5. Augment with Synonymy

6. Persist

Graph Statistics

Configuration Options

Why This Structure?

Next Steps

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Customization

Benchmarks

​Graph Philosophy

​Node Types

​Standard OpenIE Nodes

​Episodic Gist Nodes

​Temporal Nodes

​Edge Types

​Fact Edges (Entity → Entity)

​Context Edges (Passage → Entity)

​Synonymy Edges (Entity ↔ Entity)

​Episodic Gist Edges

​Graph Construction

​1. Extract Structure

​2. Create Node Embeddings

​3. Add Nodes to Graph

​4. Add Edges

​5. Augment with Synonymy

​6. Persist

​Graph Statistics

​Configuration Options

​Why This Structure?

​Next Steps

Build docs developers (and LLMs) love

Graph Philosophy

Node Types

Standard OpenIE Nodes

Episodic Gist Nodes

Temporal Nodes

Edge Types

Fact Edges (Entity → Entity)

Context Edges (Passage → Entity)

Synonymy Edges (Entity ↔ Entity)

Episodic Gist Edges

Graph Construction

1. Extract Structure

2. Create Node Embeddings

3. Add Nodes to Graph

4. Add Edges

5. Augment with Synonymy

6. Persist

Graph Statistics

Configuration Options

Why This Structure?

Next Steps