Custom Generators

Overview

The syscalls-cpp framework allows you to implement custom stub generation policies. This enables advanced techniques like encryption, obfuscation, polymorphism, and metamorphism at the syscall level. This guide walks through the complete custom-generator.cpp example, which implements an encrypted shellcode generator with junk code injection.

Custom Generator Requirements

To create a custom generator, you must implement a policy with:

struct YourCustomGenerator
{
    // Does this generator require syscall gadgets?
    static constexpr bool bRequiresGadget = false;
    
    // Maximum size of generated stub
    static constexpr size_t kMaxStubSize = 128;
    
    // Generate the stub
    static void generate(uint8_t* pBuffer, uint32_t uSyscallNumber, void* pGadgetAddress);
    
    // Return stub size
    static constexpr size_t getStubSize() { return kMaxStubSize; }
};

Complete Custom Generator Example

This example implements an encrypted syscall number generator with junk code injection:

The EncryptedShellGenerator

#include <syscalls-cpp/syscall.hpp>
#include <iostream>
#include <cstdint>
#include <array>
#include <vector>
#include <random>

struct EncryptedShellGenerator
{
    static constexpr bool bRequiresGadget = false;
    static constexpr size_t kMaxStubSize = 128;

private:
    enum class EOperationType { ADD, SUB, XOR, NOT, NEG, ROL, ROR };

    struct ObfuscationLayer_t
    {
        EOperationType m_eType;
        uint32_t m_uKey;
    };

    inline static std::mt19937 randEngine{ 
        static_cast<unsigned int>(
            std::chrono::high_resolution_clock::now()
                .time_since_epoch().count()
        ) 
    };

public:
    static void generate(uint8_t* pBuffer, uint32_t uSyscallNumber, void* /*pGadgetAddress*/);
    static constexpr size_t getStubSize() { return kMaxStubSize; }
    
private:
    static void emitJunk(CBufferWriter& writer);
};

Encryption Layer System

The generator applies multiple random transformations to the syscall number:

static void generate(uint8_t* pBuffer, uint32_t uSyscallNumber, void* /*pGadgetAddress*/)
{
    CBufferWriter writer(pBuffer, kMaxStubSize);

    std::uniform_int_distribution<> operationDist(0, 6);
    std::uniform_int_distribution<uint32_t> keyDist(1, 0xFFFFFFFF);
    std::uniform_int_distribution<> layersDist(4, 8);

    const int iAmountOfLayers = layersDist(randEngine);
    std::vector<ObfuscationLayer_t> vecLayers;
    vecLayers.reserve(iAmountOfLayers);

    // Generate random obfuscation layers
    for (int i = 0; i < iAmountOfLayers; ++i)
    {
        EOperationType operationType = static_cast<EOperationType>(operationDist(randEngine));
        uint32_t uKey = 0;

        if (operationType == EOperationType::ROL || operationType == EOperationType::ROR)
            uKey = keyDist(randEngine) % 31 + 1;
        else if (operationType != EOperationType::NOT && operationType != EOperationType::NEG)
        {
            uKey = keyDist(randEngine);
            if (uKey == 0) uKey = 1;
        }

        vecLayers.push_back({ operationType, uKey });
    }
    
    // Apply layers in reverse to encrypt the syscall number
    uint32_t uEncryptedValue = uSyscallNumber;
    for (auto it = vecLayers.rbegin(); it != vecLayers.rend(); ++it)
    {
        const auto& layer = *it;
        switch (layer.m_eType)
        {
            case EOperationType::ADD: uEncryptedValue -= layer.m_uKey; break;
            case EOperationType::SUB: uEncryptedValue += layer.m_uKey; break;
            case EOperationType::XOR: uEncryptedValue ^= layer.m_uKey; break;
            case EOperationType::NOT: uEncryptedValue = ~uEncryptedValue; break;
            case EOperationType::NEG: uEncryptedValue = -uEncryptedValue; break;
            case EOperationType::ROL: 
                uEncryptedValue = (uEncryptedValue >> layer.m_uKey) | 
                                 (uEncryptedValue << (32 - layer.m_uKey)); 
                break;
            case EOperationType::ROR: 
                uEncryptedValue = (uEncryptedValue << layer.m_uKey) | 
                                 (uEncryptedValue >> (32 - layer.m_uKey)); 
                break;
        }
    }

    // Emit stub prologue
    writer.writeBytes({ 0x51, 0x41, 0x5A });     // push rcx; pop r10
    writer.write<uint8_t>(0xB8);                 // mov eax, ...
    writer.write<uint32_t>(uEncryptedValue);     // ... encrypted_value

    emitJunk(writer);

    // Emit decryption layers in forward order
    for (const auto& layer : vecLayers)
    {
        switch (layer.m_eType) {
        case EOperationType::ADD:
            writer.write<uint8_t>(0x05);          // add eax, key
            writer.write<uint32_t>(layer.m_uKey);
            break;
        case EOperationType::SUB:
            writer.write<uint8_t>(0x2D);          // sub eax, key
            writer.write<uint32_t>(layer.m_uKey);
            break;
        case EOperationType::XOR:
            writer.write<uint8_t>(0x35);          // xor eax, key
            writer.write<uint32_t>(layer.m_uKey);
            break;
        case EOperationType::NOT:
            writer.writeBytes({ 0xF7, 0xD0 });    // not eax
            break;
        case EOperationType::NEG:
            writer.writeBytes({ 0xF7, 0xD8 });    // neg eax
            break;
        case EOperationType::ROL:
            writer.writeBytes({ 0xC1, 0xC0 });    // rol eax, key
            writer.write<uint8_t>(static_cast<uint8_t>(layer.m_uKey));
            break;
        case EOperationType::ROR:
            writer.writeBytes({ 0xC1, 0xC8 });    // ror eax, key
            writer.write<uint8_t>(static_cast<uint8_t>(layer.m_uKey));
            break;
        }

        emitJunk(writer);  // Random junk between operations
    }

    // Emit syscall and return
    writer.writeBytes({ 0x0F, 0x05 });                   // syscall
    writer.writeBytes({ 0x48, 0x83, 0xC4, 0x08 });      // add rsp, 8
    writer.writeBytes({ 0xFF, 0x64, 0x24, 0xF8 });      // jmp qword ptr [rsp-8]

    writer.fillRest(0xCC);  // Fill rest with int3
}

Junk Code Injection

The generator injects random “junk” instructions that don’t affect execution:

static void emitJunk(CBufferWriter& writer)
{
    std::uniform_int_distribution<> junkDist(0, 5);
    int iJunkType = junkDist(randEngine);

    const uint8_t uREXW = 0x48;
    const std::array<uint8_t, 6> arrPushOpcodes = { 0x50, 0x51, 0x52, 0x53, 0x56, 0x57 };
    const std::array<uint8_t, 6> arrPopOpcodes = { 0x58, 0x59, 0x5A, 0x5B, 0x5E, 0x5F };

    std::uniform_int_distribution<> regDist(0, arrPushOpcodes.size() - 1);
    int iRegIdx = regDist(randEngine);

    switch (iJunkType)
    {
        case 0:
            // nop
            writer.write<uint8_t>(0x90);
            break;

        case 1:
            // push rXX; pop rXX
            writer.write<uint8_t>(arrPushOpcodes[iRegIdx]);
            writer.write<uint8_t>(arrPopOpcodes[iRegIdx]);
            break;

        case 2:
            // pushfq; inc rXX; dec rXX; popfq
            writer.write<uint8_t>(0x9C);
            writer.writeBytes({ uREXW, 0xFF, arrIncDecNegModRM[iRegIdx] });
            writer.writeBytes({ uREXW, 0xFF, (uint8_t)(arrIncDecNegModRM[iRegIdx] + 0x08) });
            writer.write<uint8_t>(0x9D);
            break;

        case 3:
            // lea rXX, [rXX + 0x00]
            writer.writeBytes({ uREXW, 0x8D, arrLeaModRM[iRegIdx], 0x00 });
            break;

        case 4:
            // pushfq; xor rXX, imm32; xor rXX, imm32; popfq
            writer.write<uint8_t>(0x9C);
            std::uniform_int_distribution<uint32_t> valDist;
            uint32_t uRandomVal = valDist(randEngine);
            writer.writeBytes({ uREXW, 0x81, (uint8_t)(0xF0 + iRegIdx) });
            writer.write<uint32_t>(uRandomVal);
            writer.writeBytes({ uREXW, 0x81, (uint8_t)(0xF0 + iRegIdx) });
            writer.write<uint32_t>(uRandomVal);
            writer.write<uint8_t>(0x9D);
            break;

        case 5:
            // No junk
            break;
    }
}

Buffer Writer Helper

A simple helper class for writing bytes to the stub buffer:

class CBufferWriter 
{
public:
    CBufferWriter(uint8_t* buffer, size_t size) 
        : m_pStart(buffer), m_pCurrent(buffer), m_pEnd(buffer + size) {}

    template<typename T>
    void write(T value) 
    {
        if (m_pCurrent + sizeof(T) <= m_pEnd)
        {
            *reinterpret_cast<T*>(m_pCurrent) = value;
            m_pCurrent += sizeof(T);
        }
    }

    void writeBytes(std::initializer_list<uint8_t> listBytes) {
        if (m_pCurrent + listBytes.size() <= m_pEnd) 
        {
            memcpy(m_pCurrent, listBytes.begin(), listBytes.size());
            m_pCurrent += listBytes.size();
        }
    }

    void fillRest(uint8_t uValue) 
    {
        if (m_pCurrent < m_pEnd) {
            memset(m_pCurrent, uValue, m_pEnd - m_pCurrent);
        }
    }

private:
    uint8_t* m_pStart;
    uint8_t* m_pCurrent;
    const uint8_t* m_pEnd;
};

Using the Custom Generator

int main() 
{
    // Use custom generator instead of built-in policies
    syscall::Manager<
        syscall::policies::allocator::section, 
        EncryptedShellGenerator
    > syscallManager;
    
    if (!syscallManager.initialize()) 
    {
        std::cerr << "failed to initialize syscall manager" << std::endl;
        return 1;
    }

    std::cout << "syscall manager initialized successfully" << std::endl;

    NTSTATUS status;
    PVOID pBaseAddress = nullptr;
    SIZE_T uSize = 0x1000;

    status = syscallManager.invoke<NTSTATUS>(
        SYSCALL_ID("NtAllocateVirtualMemory"),
        syscall::native::getCurrentProcess(),
        &pBaseAddress,
        0,
        &uSize,
        MEM_COMMIT | MEM_RESERVE,
        PAGE_READWRITE
    );

    if(pBaseAddress)
        std::cout << "memory allocated: " << pBaseAddress;
    else
        std::cout << "failed to allocate, status:" << status;

    return 0;
}

How the Encryption Works

Example Flow

Let’s say the syscall number for NtAllocateVirtualMemory is 0x18:

Generate Random Layers: [ADD 0x1234, XOR 0xABCD, ROL 5]
Encrypt (Apply Reverse):
- Start: 0x18
- Apply ROL 5 reverse (ROR 5): 0x0C000000
- Apply XOR 0xABCD: 0x0C00ABCD
- Apply ADD reverse (SUB): 0x0C009999
- Result: Encrypted value 0x0C009999

Generated Stub:

push rcx
pop r10
mov eax, 0x0C009999    ; Load encrypted value
add eax, 0x1234        ; Decrypt layer 1
xor eax, 0xABCD        ; Decrypt layer 2
rol eax, 5             ; Decrypt layer 3
; eax now contains 0x18
syscall

Benefits of Custom Generators

Polymorphism

Each stub is unique due to random encryption layers and junk code. The same syscall generates different bytecode on each run.

Signature Evasion

No static patterns to signature. Security products can’t create a single signature that matches all variants.

Analysis Resistance

The encrypted syscall number and junk code make static and dynamic analysis significantly harder.

Extensibility

Easy to add new obfuscation techniques: more operations, different junk patterns, or metamorphic engines.

Use Cases

Research: Studying polymorphic code generation
Advanced Evasion: Maximum stealth with unique per-stub bytecode
Testing: Validating security product effectiveness against polymorphic code
Education: Learning x64 assembly and obfuscation techniques

Performance Considerations

Generation Time: ~1-5μs per stub (during initialization)
Execution Time: Slower than direct due to decryption layers
Size: Larger stubs (up to 128 bytes) due to junk and decryption

Expected Output

$ ./custom-generator.exe
syscall manager initialized successfully
memory allocated: 0x000001C2D3E4F000

Advanced Customization Ideas

Metamorphic Engine

Implement code reordering, equivalent instruction substitution, and register reallocation for maximum polymorphism.

Virtualization

Create a virtual machine interpreter that executes custom bytecode to perform syscalls.

Steganography

Hide syscall numbers in seemingly legitimate data structures or code patterns.

Control Flow Flattening

Use state machines and indirect jumps to obscure the execution flow.

Testing Your Custom Generator

When developing a custom generator:

Start Simple: Begin with a basic implementation without obfuscation
Test Core Functionality: Verify syscalls work correctly
Add Features Incrementally: Add encryption, then junk code, then advanced features
Validate Output: Disassemble generated stubs to verify correctness
Benchmark: Measure performance impact of your obfuscation

Debugging Tips

// Add debug output to see generated stubs
void dumpStub(const uint8_t* pBuffer, size_t size)
{
    std::cout << "Generated stub:" << std::endl;
    for (size_t i = 0; i < size; ++i)
    {
        std::cout << std::hex << std::setw(2) << std::setfill('0') 
                  << static_cast<int>(pBuffer[i]) << " ";
        if ((i + 1) % 16 == 0) std::cout << std::endl;
    }
    std::cout << std::dec << std::endl;
}

Get Started

Core Concepts

Guides

Examples

Overview

Custom Generator Requirements

Complete Custom Generator Example

The EncryptedShellGenerator

Encryption Layer System

Junk Code Injection

Buffer Writer Helper

Using the Custom Generator

How the Encryption Works

Example Flow

Benefits of Custom Generators

Use Cases

Performance Considerations

Expected Output

Advanced Customization Ideas

Metamorphic Engine

Virtualization

Steganography

Control Flow Flattening

Testing Your Custom Generator

Debugging Tips

See Also

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Examples

​Overview

​Custom Generator Requirements

​Complete Custom Generator Example

​The EncryptedShellGenerator

​Encryption Layer System

​Junk Code Injection

​Buffer Writer Helper

​Using the Custom Generator

​How the Encryption Works

​Example Flow

​Benefits of Custom Generators

​Use Cases

​Performance Considerations

​Expected Output

​Advanced Customization Ideas

Metamorphic Engine

Virtualization

Steganography

Control Flow Flattening

​Testing Your Custom Generator

​Debugging Tips

​See Also

Build docs developers (and LLMs) love

Overview

Custom Generator Requirements

Complete Custom Generator Example

The EncryptedShellGenerator

Encryption Layer System

Junk Code Injection

Buffer Writer Helper

Using the Custom Generator

How the Encryption Works

Example Flow

Benefits of Custom Generators

Use Cases

Performance Considerations

Expected Output

Advanced Customization Ideas

Testing Your Custom Generator

Debugging Tips

See Also