Skip to main content

Overview

RAPTOR includes a comprehensive test suite to validate all commands, workflows, and user scenarios. Tests cover command structure, security analysis modes, package architecture, and real vulnerability detection.
Tests use real vulnerable code samples to ensure RAPTOR’s detection capabilities work in practice.

Quick Start

cd raptor
bash test/comprehensive_test.sh
Expected duration: 2-3 minutes for full suite

Test Structure

RAPTOR’s test suite is organized into multiple categories:

Test Categories

Validates that all RAPTOR commands are properly registered and accessible.Tests:
  • Main launcher recognizes all modes (scan, fuzz, web, agentic, codeql)
  • Core execution scripts exist (raptor_agentic.py, raptor_fuzzing.py, etc.)
  • Main scripts have valid Python syntax
  • Help information is accessible
Example:
python3 raptor.py --help
# Should list: scan, fuzz, web, agentic, codeql
Validates static analysis capabilities.Tests:
  • Scan mode help available
  • Scan requires —repo argument
  • Scan supports policy groups
Example:
python3 raptor.py scan --repo /test/data
Validates autonomous workflow capabilities.Tests:
  • Agentic script exists
  • Agentic accepts arguments
  • CodeQL integration available
  • Exploit/patch generation controls present
Example:
python3 raptor.py agentic --repo /test/data --skip-exploits
Validates binary fuzzing capabilities.Tests:
  • Fuzzing script exists
  • Requires —binary argument
  • Supports —duration option
  • Supports —parallel option
  • Supports autonomous mode
Example:
python3 raptor.py fuzz --binary ./target --duration 3600
Validates deep semantic analysis.Tests:
  • CodeQL script exists
  • CodeQL accepts arguments
  • Language detection supported
Example:
python3 raptor.py codeql --repo /test/data --language python
Validates web application security testing (alpha).Tests:
  • Web mode script exists
  • Web mode listed in help
Note: Web mode is in alpha stage.
Validates internal package structure.Tests:
  • Core modules directory exists
  • Packages directory exists
  • LLM analysis package exists
  • Static analysis package exists
  • Fuzzing utilities exist
Validates that test data samples are present and contain real vulnerabilities.Tests:
  • Test data directory exists
  • Sample Python vulnerable code present
  • Sample JavaScript vulnerable code present
  • Code samples contain vulnerability patterns
Validates that all user-facing workflows are accessible.Tests:
  • scan —help accessible
  • agentic —help accessible
  • fuzz —help accessible
  • codeql —help accessible

Test Data Examples

RAPTOR includes realistic vulnerable code samples for testing:

Python SQL Injection

import sqlite3
from flask import Flask, request

app = Flask(__name__)

# VULNERABLE: SQL injection in database query
@app.route('/user/<user_id>')
def get_user(user_id):
    db = sqlite3.connect(':memory:')
    cursor = db.cursor()
    
    # Direct string concatenation - SQL injection vulnerability
    query = "SELECT * FROM users WHERE id = " + user_id
    cursor.execute(query)
    
    return cursor.fetchone()

# VULNERABLE: Hardcoded credentials
DATABASE_PASSWORD = "admin123!SuperSecret"

# VULNERABLE: Weak cryptography (MD5)
import hashlib
def hash_password(password):
    return hashlib.md5(password.encode()).hexdigest()

# VULNERABLE: Command injection
import subprocess
@app.route('/convert', methods=['POST'])
def convert_file():
    filename = request.form.get('filename')
    result = subprocess.run(f"convert {filename} output.jpg", shell=True)
    return "Conversion complete"

# VULNERABLE: Path traversal
@app.route('/download/<file_path>')
def download_file(file_path):
    with open(f"/var/www/files/{file_path}", "rb") as f:
        return f.read()
Vulnerabilities in this file:
  • SQL injection (string concatenation)
  • Hardcoded credentials
  • Weak cryptography (MD5)
  • Command injection (shell=True)
  • Path traversal (unsanitized file path)

JavaScript XSS

// VULNERABLE: DOM-based XSS
function displayUserInput() {
    const userInput = document.getElementById('user_input').value;
    // Direct innerHTML assignment - XSS vulnerability
    document.getElementById('output').innerHTML = userInput;
}

// VULNERABLE: Reflected XSS via query parameter
function handleSearchQuery() {
    const query = new URLSearchParams(window.location.search).get('q');
    // Unsanitized query in HTML - reflected XSS
    document.write('<p>Search results for: ' + query + '</p>');
}

// VULNERABLE: Eval usage
function executeUserCode(code) {
    // eval() is extremely dangerous - arbitrary code execution
    eval(code);
}

// VULNERABLE: Hardcoded API key
const API_KEY = "sk-1234567890abcdef";
const SECRET_TOKEN = "super_secret_token_12345";

// VULNERABLE: Insecure localStorage usage
function storeUserCredentials(username, password) {
    // Storing sensitive data in localStorage - insecure
    localStorage.setItem('username', username);
    localStorage.setItem('password', password);
}

// VULNERABLE: Insecure random number generation
function generateSessionToken() {
    // Math.random() is not cryptographically secure
    return Math.random().toString(36).substring(7);
}

// VULNERABLE: Prototype pollution
function mergeObjects(target, source) {
    for (const key in source) {
        // No validation - allows prototype pollution
        target[key] = source[key];
    }
    return target;
}

// VULNERABLE: Regular expression DoS
function validateEmail(email) {
    // Inefficient regex - ReDoS vulnerability
    const pattern = /^([a-zA-Z0-9]+)*@([a-zA-Z0-9]+)*\.([a-zA-Z0-9]+)*$/;
    return pattern.test(email);
}
Vulnerabilities in this file:
  • DOM-based XSS (innerHTML)
  • Reflected XSS (document.write)
  • Arbitrary code execution (eval)
  • Hardcoded secrets (API keys)
  • Insecure storage (localStorage)
  • Weak random generation (Math.random)
  • Prototype pollution
  • ReDoS (inefficient regex)

Running Tests

Comprehensive Test Suite

Run all tests with detailed output:
cd raptor
bash test/comprehensive_test.sh
Output format:
╔═══════════════════════════════════════════════════════════╗
║          RAPTOR Comprehensive Test Suite                  ║
║     Testing actual workflows, commands, and scenarios     ║
╚═══════════════════════════════════════════════════════════╝

1. COMMAND STRUCTURE TESTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ Main launcher lists all modes
✓ Core execution scripts exist
✓ Main scripts have valid Python syntax
✓ Help information accessible

2. SCAN MODE TESTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ Scan mode help available
✓ Scan requires --repo argument
...

╔═══════════════════════════════════════════════════════════╗
║ SUMMARY
╚═══════════════════════════════════════════════════════════╝

Passed:  42
Failed:  0
Skipped: 3
Total:   45

Compliance: 93% (42/45 tests passed)

✓ All tests passed!

Test Result Indicators

  • PASS - Test succeeded
  • FAIL - Test failed (with error message)
  • SKIP - Test skipped (with reason)

Test Workflows (Claude Code)

RAPTOR includes a /test-workflows command for Claude Code:
claude
# Then in Claude:
/test-workflows
What it tests:
  1. Basic scan (findings only, no exploits)
  2. Full agentic workflow (scan + exploit + patch)
  3. Binary fuzzing
  4. Manual crash validation
  5. Tool routing sanity checks
Output: Pass/Fail/Skip status with summary counts

Integration Tests

Integration tests verify RAPTOR works with external tools:
bash test/integration_tests.sh
Requirements:
  • Semgrep installed
  • CodeQL installed (optional)
  • AFL++ installed (optional)
  • Test data present
Tests:
  • Semgrep scanning works
  • CodeQL database creation (if available)
  • AFL++ fuzzing setup (if available)
  • Python package imports

User Stories

Tests are designed around real user scenarios:
1

User Story 1: Quick Scan

As a developer, I want to quickly scan my code for vulnerabilities.
python3 raptor.py scan --repo ./myapp
Tested by: Scan mode tests, fixture tests
2

User Story 2: Autonomous Analysis

As a security researcher, I want RAPTOR to autonomously analyze, generate exploits, and propose patches.
python3 raptor.py agentic --repo ./myapp
Tested by: Agentic mode tests, workflow tests
3

User Story 3: Binary Fuzzing

As a bug hunter, I want to fuzz a binary and analyze crashes.
python3 raptor.py fuzz --binary ./target --duration 3600
Tested by: Fuzzing mode tests, integration tests
4

User Story 4: Deep Analysis

As an analyst, I want to use CodeQL for deep semantic analysis.
python3 raptor.py codeql --repo ./myapp
Tested by: CodeQL mode tests, integration tests
5

User Story 5: Custom Workflows

As a power user, I want to verify RAPTOR works after I make changes.
bash test/comprehensive_test.sh
Tested by: All test suites

Writing New Tests

To add tests to the suite:
1

Choose Test Category

Determine which test file to modify:
  • comprehensive_test.sh - Core functionality
  • integration_tests.sh - External tool integration
  • test_workflows.sh - End-to-end workflows
2

Add Test Function

# Add test in appropriate section
if [condition]; then
    test_result "Your test name" "PASS"
else
    test_result "Your test name" "FAIL" "Error message"
fi
3

Add Test Data (if needed)

Create new vulnerable code samples in test/data/:
# test/data/your_vuln.py
# VULNERABLE: Description
def vulnerable_function():
    # ... vulnerable code ...
4

Run and Verify

bash test/comprehensive_test.sh
Ensure your new test passes and doesn’t break existing tests.

Continuous Integration

RAPTOR uses GitHub Actions for CI. Tests run automatically on pull requests.
CI Workflow:
  1. Install dependencies
  2. Run syntax validation
  3. Run comprehensive test suite
  4. Run integration tests (if tools available)
  5. Report results
Badge: Tests

Troubleshooting Tests

Problem: Required tools not installed.Solution:
  1. Use the devcontainer (all tools pre-installed)
  2. Or install missing tools:
pip install semgrep
# Install other tools as needed
Problem: Test requires actual execution, skipped for safety.Solution: This is normal. Skipped tests indicate optional features or safety guards.
Problem: External tools not working.Solution:
  1. Verify tool installation:
semgrep --version
codeql version
  1. Check test/data directory exists
  2. Ensure PYTHONPATH is set correctly
Problem: Test data files modified or missing.Solution:
  1. Restore test/data files from git:
git checkout test/data/
  1. Ensure files contain documented vulnerabilities

Test Coverage

RAPTOR’s test suite covers:

Command Coverage

All major commands:
  • /scan
  • /agentic
  • /fuzz
  • /codeql
  • /web
  • /analyze

Vulnerability Coverage

Multiple vulnerability types:
  • SQL injection
  • XSS (DOM, reflected)
  • Command injection
  • Path traversal
  • Secrets exposure
  • Weak cryptography

Language Coverage

Multiple languages:
  • Python
  • JavaScript
  • C/C++ (via binaries)

Package Coverage

All packages:
  • llm_analysis
  • static-analysis
  • fuzzing
  • codeql
  • web

Next Steps

DevContainer

Set up pre-configured environment

Contributing

Add your own tests

Quick Start

Start using RAPTOR

FAQ

Common questions

Build docs developers (and LLMs) love