Benchmark Datasets

​

Dataset Overview

​

MuSiQue

​

Dataset Structure

• Location: reproduce/dataset/musique/
• Files:
  • musique.json - Questions and answers
  • musique_corpus.json - Document corpus
• Format: Each sample contains:
  • question: The question text
  • answer: Gold answer
  • paragraphs: Supporting paragraphs with is_supporting flag

​

Running MuSiQue

python main.py \
    --dataset musique \
    --llm_name gpt-4o-mini \
    --embedding_name nvidia/NV-Embed-v2 \
    --extract_method openie

​

2WikiMultiHopQA

​

Dataset Structure

• Location: reproduce/dataset/2wikimultihopqa/
• Files:
  • 2wikimultihopqa.json - Questions with supporting facts
  • 2wikimultihopqa_corpus.json - Wikipedia passages
• Format: Includes supporting_facts field with document titles and sentence indices

​

Running 2WikiMultiHopQA

python main.py \
    --dataset 2wikimultihopqa \
    --llm_name gpt-4o-mini \
    --embedding_name nvidia/NV-Embed-v2

​

LoCoMo (Long Conversation Memory)

​

Dataset Structure

• Location: reproduce/dataset/locomo/
• Files:
  • locomo_episodic.json - Episodic QA samples
  • locomo_temporal.json - Temporal QA samples
  • locomo10.json - 10-sample subset
  • msc_personas_all.json - Persona data

​

Data Format

• conversation: Multi-session conversation data
  • session_{idx}: Dialogue turns with speaker and text
  • session_{idx}_date_time: Timestamp for each session
• qa: List of question-answer pairs with:
  • question: Question text
  • answer: Gold answer
  • evidence: Dialogue IDs containing the answer
  • category: Question type (e.g., “factual”, “opinion”)
  • temporal_category: Temporal reasoning type (e.g., “none”, “before”, “after”)

​

Running LoCoMo

python examples/locomo.py \
    --dataset locomo_episodic \
    --llm_name gpt-4o-mini \
    --embedding_name nvidia/NV-Embed-v2 \
    --extract_method episodic_gist

​

LongMemEval

​

Dataset Structure

• Location: reproduce/dataset/longmemeval/
• Files:
  • longmemeval_s.json - Small version (500+ samples)
  • longmemeval_m.json - Medium version
  • longmemeval_oracle.json - Oracle version with gold context

​

Data Format

• haystack_sessions: List of conversation sessions with:
  • id: Session identifier
  • date: Session timestamp
  • messages: List of messages with role, content, and has_answer flag
• question: Question to answer
• question_date: When the question was asked
• question_type: Type of question
• answer: Gold answer
• answer_session_ids: IDs of sessions containing the answer

​

Running LongMemEval

python examples/longmemeval.py \
    --dataset longmemeval_s.json \
    --llm_name gpt-4o-mini \
    --embedding_name nvidia/NV-Embed-v2 \
    --extract_method episodic_gist

​

Complex TR (Temporal Reasoning)

​

Dataset Structure

• Location: reproduce/dataset/complex-tr/
• Files:
  • complex_tr_1000.json - 1000 temporal reasoning questions
  • complex_tr_1000_corpus.json - Temporal facts corpus
  • complex_tr_3993.json - Full dataset (3993 questions)

​

Data Format

• question: Temporal reasoning question
• answers: List of valid answers
• fact_context: Relevant temporal facts for the question
• Corpus contains temporal statements like “Event A happened before Event B”

​

Running Complex TR

python examples/complex_tr.py \
    --llm_name gpt-4o-mini \
    --embedding_name nvidia/NV-Embed-v2 \
    --extract_method episodic_gist

​

TimeQA

​

Dataset Structure

• Location: reproduce/dataset/timeqa/
• Files: dev.easy.json, dev.hard.json
• Format: Questions with temporal context and multiple target answers

​

Running TimeQA

python examples/timeqa.py \
    --llm_name gpt-4o-mini \
    --dataset_file reproduce/dataset/timeqa/dev.easy.json \
    --embedding_name nvidia/NV-Embed-v2 \
    --extract_method temporal \
    --qa_top_k 5

​

RealTalk

​

Dataset Structure

• Location: reproduce/dataset/realtalk/
• Files: Multiple chat files (Chat_1_*.json, Chat_2_*.json, etc.)
• Format: Real conversation sessions with participants and messages

​

Running RealTalk

python examples/realtalk.py \
    --llm_name gpt-4o-mini \
    --embedding_name nvidia/NV-Embed-v2 \
    --extract_method episodic_gist

​

Semantic QA

​

Running Semantic QA

python examples/semantic_qa.py \
    --llm_name gpt-4o-mini \
    --embedding_name nvidia/NV-Embed-v2

​

Dataset Comparison

​

Common Dataset Arguments