System Calls Overview

Introduction

Aurora OS implements system calls using the x86_64 SYSCALL instruction (with int $0x80 as a fallback). All syscalls follow a consistent calling convention and return negative values to indicate errors.

Calling Convention

Syscalls use the following register layout:

RAX

uint64_t

required

Syscall number (0-39)

RDI

uint64_t

First argument (arg1)

RSI

uint64_t

Second argument (arg2)

RDX

uint64_t

Third argument (arg3)

R10

uint64_t

Fourth argument (use R10 instead of RCX, which is clobbered by SYSCALL)

Return Value

RAX

int64_t

Return value. Negative values indicate errors:

-1: Generic error (EINVAL, ENOENT, ENOMEM)
-2: Permission denied (EPERM) — syscall filter blocked the call

ABI Notes

The SYSCALL instruction clobbers RCX (saved RIP) and R11 (saved RFLAGS). Use R10 for the 4th argument instead of RCX.

The kernel stack is loaded from TSS rsp0 on ring transition (privilege level change from user to kernel mode).

Syscall Categories

Aurora OS provides 40 system calls organized into the following categories:

Process Management

SYS_EXIT — Terminate current process
SYS_FORK — Create child process
SYS_EXECVE — Execute new program
SYS_WAITPID — Wait for child process
SYS_YIELD — Yield CPU to scheduler
SYS_GETPID — Get process ID
SYS_KILL — Send signal to process

File System

SYS_OPEN — Open file or device
SYS_CLOSE — Close file descriptor
SYS_READ — Read from file
SYS_WRITE — Write to file
SYS_STAT — Get file status
SYS_FSTAT — Get file status by fd
SYS_LSEEK — Reposition file offset
SYS_GETDENTS — Read directory entries
SYS_MKDIR — Create directory
SYS_UNLINK — Remove file

Memory Management

SYS_MMAP — Map memory region
SYS_MUNMAP — Unmap memory region
SYS_BRK — Change data segment size

Inter-Process Communication

SYS_PIPE — Create pipe
SYS_DUP2 — Duplicate file descriptor
SYS_MSGSND — Send message
SYS_MSGRCV — Receive message
SYS_SHMGET — Create shared memory segment
SYS_SHMAT — Attach shared memory
SYS_SHMDT — Detach shared memory

Network

SYS_SOCKET — Create socket
SYS_BIND — Bind socket to address
SYS_CONNECT — Connect to remote address
SYS_LISTEN — Listen for connections
SYS_ACCEPT — Accept incoming connection
SYS_SEND — Send data on socket
SYS_RECV — Receive data from socket

Miscellaneous

SYS_GETCWD — Get current working directory
SYS_CLOCK — Get timestamp (TSC ticks)
SYS_NANOSLEEP — Sleep for specified duration
SYS_IOCTL — Device control operations
SYS_SIGACTION — Set signal handler
SYS_SIGRETURN — Return from signal handler

Syscall Filtering

Per-process syscall filter bitmaps can block specific syscalls for sandboxed agents. When a syscall is blocked, it returns -2 (EPERM).

The kernel maintains a 256-bit filter bitmap (32 bytes) per process. When filter_enabled is set, each syscall number is checked against the bitmap before execution.

struct process {
    uint8_t syscall_filter[32];  // 256 bits (one per syscall)
    int filter_enabled;
};

Signals

Aurora OS supports the following signals:

Signal	Number	Default Action	Catchable
SIGKILL	9	Terminate	No
SIGSEGV	11	Terminate	Yes
SIGPIPE	13	Terminate	Yes
SIGCHLD	17	Ignore	Yes

SIGKILL cannot be caught or ignored. It always terminates the target process immediately.

Implementation Details

The syscall interface is implemented in /kernel/src/proc/syscall.c with the following components:

MSR Configuration

void syscall_init(void) {
    // Enable SYSCALL/SYSRET (SCE bit in EFER)
    wrmsr(IA32_EFER, rdmsr(IA32_EFER) | 1);
    
    // Set up segment selectors
    wrmsr(IA32_STAR, ((uint64_t)0x0008 << 32) | ((uint64_t)0x0018 << 48));
    
    // Set syscall entry point
    wrmsr(IA32_LSTAR, (uint64_t)syscall_entry);
    
    // Mask interrupts on syscall entry
    wrmsr(IA32_FMASK, (1 << 9));
}

Entry Point

The assembly entry point is defined in /kernel/src/proc/syscall_entry.S and saves user context before dispatching to the C handler.

Dispatcher

uint64_t syscall_handler(uint64_t num, uint64_t arg1, 
                         uint64_t arg2, uint64_t arg3);

The handler performs syscall filtering checks and dispatches to the appropriate implementation based on the syscall number.

System Calls

Kernel API

Package Manager

System Calls Overview

Introduction

Calling Convention

Return Value

ABI Notes

Syscall Categories

Process Management

File System

Memory Management

Inter-Process Communication

Network

Miscellaneous

Syscall Filtering

Signals

Implementation Details

MSR Configuration

Entry Point

Dispatcher

Build docs developers (and LLMs) love

System Calls

Kernel API

Package Manager

​Introduction

​Calling Convention

​Return Value

​ABI Notes

​Syscall Categories

​Process Management

​File System

​Memory Management

​Inter-Process Communication

​Network

​Miscellaneous

​Syscall Filtering

​Signals

​Implementation Details

​MSR Configuration

​Entry Point

​Dispatcher

Build docs developers (and LLMs) love

Introduction

Calling Convention

Return Value

ABI Notes

Syscall Categories

Process Management

File System

Memory Management

Inter-Process Communication

Network

Miscellaneous

Syscall Filtering

Signals

Implementation Details

MSR Configuration

Entry Point

Dispatcher