Skip to main content

​
Welcome to the X For You Feed Algorithm

The X For You feed algorithm is a sophisticated recommendation system that combines machine learning with real-time data processing to deliver personalized content to users. This documentation provides a comprehensive guide to understanding, deploying, and customizing the system.
The transformer implementation is ported from the Grok-1 open source release by xAI, adapted for recommendation system use cases.

​
What Makes This System Unique

The For You feed represents a modern approach to content recommendation that eliminates traditional complexity:

No Hand-Engineered Features

The Grok-based transformer learns relevance entirely from user engagement sequences, eliminating manual feature engineering

Real-Time Processing

Thunder provides sub-millisecond in-network content lookups through an in-memory architecture

Multi-Action Prediction

Predicts 14 different engagement types simultaneously, from likes to blocks, for nuanced relevance scoring

Candidate Isolation

Special attention masking ensures consistent, cacheable scores independent of batch composition

​
Key Components

The system is built on four major components working together:
1

Home Mixer

The orchestration layer that assembles your For You feed using a composable pipeline architectureLocation: home-mixer/
2

Thunder

In-memory post store with realtime Kafka ingestion for lightning-fast in-network content retrievalLocation: thunder/
3

Phoenix

ML component handling both retrieval (two-tower model) and ranking (Grok-based transformer)Location: phoenix/
4

Candidate Pipeline

Reusable framework providing composable traits for building recommendation pipelinesLocation: candidate-pipeline/

​
How Content Flows Through The System

The For You feed retrieves, ranks, and filters posts from two distinct sources:

​
Thunder: In-Network Posts

Posts from accounts you follow are retrieved from Thunder, an in-memory store that:
  • Consumes post create/delete events from Kafka in real-time
  • Maintains per-user stores for original posts, replies/reposts, and videos
  • Serves candidates in sub-millisecond timeframes
  • Automatically trims posts older than the retention period
This ensures you never miss content from people you’ve chosen to follow.

​
The Ranking Model

Both in-network and out-of-network candidates are ranked together using Phoenix, a Grok-based transformer that:
Phoenix analyzes your engagement history (what you liked, replied to, shared, etc.) to understand your interests and predict which content is most relevant to you.

​
Multi-Action Prediction

Rather than predicting a single “relevance” score, Phoenix predicts probabilities for 14 different actions: 
Predictions:
├── P(favorite)        # Positive signal
├── P(reply)           # Strong positive signal  
├── P(repost)          # Strong positive signal
├── P(quote)           # Positive signal
├── P(click)           # Engagement signal
├── P(profile_click)   # Interest signal
├── P(video_view)      # Engagement signal
├── P(photo_expand)    # Engagement signal
├── P(share)           # Strong positive signal
├── P(dwell)           # Time spent signal
├── P(follow_author)   # Very strong positive signal
├── P(not_interested)  # Negative signal
├── P(block_author)    # Strong negative signal
├── P(mute_author)     # Strong negative signal
└── P(report)          # Strong negative signal
These predictions are combined into a weighted final score: 
Final Score = Σ (weight_i × P(action_i))
Negative actions (block, mute, report) have negative weights, actively pushing down content you would likely dislike.

​
Technology Stack

Backend: Rust

Home Mixer, Thunder, and Candidate Pipeline are built in Rust for performance and reliability

ML: JAX + Python

Phoenix models are implemented in JAX for high-performance tensor operations

Serving: gRPC

The system exposes a ScoredPostsService gRPC endpoint for efficient client communication

Data: Kafka

Real-time post ingestion through Kafka ensures fresh content availability

​
What You’ll Learn

This documentation will guide you through:
  • Understanding the system architecture and design decisions
  • Deploying the components in your infrastructure
  • Customizing scoring weights and filtering logic
  • Monitoring pipeline performance and debugging issues
  • Extending the framework with new sources, filters, and scorers
Start with the Overview to understand how all components work together, then explore the Architecture for technical details.

​
Open Source

This project is licensed under the Apache License 2.0, making it freely available for use, modification, and distribution. 
# Clone the repository
git clone https://github.com/xai-org/x-algorithm
cd x-algorithm

# Explore the components
ls -l
# candidate-pipeline/
# home-mixer/
# phoenix/
# thunder/

​
Installation

​
Phoenix ML Models

The Phoenix models require Python 3.11+ and the uv package manager:
1

Install uv

Follow the uv installation guide
curl -LsSf https://astral.sh/uv/install.sh | sh
2

Navigate to Phoenix directory

cd phoenix/
3

Run the models

uv automatically manages dependencies and Python versions
uv run run_ranker.py
Dependencies are specified in pyproject.toml and include JAX 0.8.1, dm-haiku, and numpy. uv handles all package installation automatically.

​
Rust Components

The Home Mixer, Thunder, and Candidate Pipeline components are written in Rust: 
# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Build all Rust components
cargo build --release

​
Next Steps

System Overview

Learn how the pipeline stages work together

Architecture Deep Dive

Explore the technical architecture and design patterns

Phoenix ML Models

Understand the retrieval and ranking models

Deployment Guide

Deploy the system in production

Build docs developers (and LLMs) love

Get started for freeTalk to us