System Overview

Architecture Overview

The robotic arm system consists of three main hardware components working together to create an autonomous pick-and-place robot with computer vision capabilities:

Raspberry Pi

High-level control, AI inference, and camera processing

VEX Brain

Real-time motor control and sensor management

Camera System

Object detection using YOLO models

System Architecture Diagram

Main Modules

Communication Module

The communication system enables real-time data exchange between the Raspberry Pi and VEX Brain using serial communication with JSON messaging.

Raspberry Pi Communication Manager

Location: arm_system/communication/serial_manager.py:16Key Features:

Serial port configuration (default: /dev/ttyACM1 @ 115200 baud)
JSON message encoding/decoding
Asynchronous message reading with threading
Callback system for handling responses
Event-based synchronization for movements

Message Types:

check_service — Verify hardware status
safety_service — Execute safety protocols
scan_service — Perform environment scanning
pick_service / place_service — Object manipulation
get_angles — Query current joint positions

VEX Brain Communication Manager

Location: arm_system/vex_brain/src/main.py:26Key Features:

Serial communication via /dev/serial1
JSON message parsing and sending
Message buffering with newline delimiters
Real-time response to Raspberry Pi commands

Communication Protocol:

{
  "type": "scan_service",
  "data": {
    "speed": 20,
    "angle": 90,
    "distance": 150
  }
}

Perception Module

The perception system combines camera hardware, computer vision, and AI models to detect and classify objects.

The system uses YOLO11s models optimized with NCNN for efficient inference on Raspberry Pi.

Components:

Camera Manager (arm_system/perception/vision/camera/main.py:5)
- Captures images using OpenCV (cv2.VideoCapture)
- Default resolution: 1280x720
- Frame grabbing technique for image stability
- Automatic image saving with timestamps
Image Processor (arm_system/perception/vision/image_processing.py:8)
- YOLO-based object detection inference
- Configurable confidence threshold (default: 0.45)
- Detects: apple, orange, bottle
- Returns best detection with bounding box coordinates
- Draws detection results on images
Detection Model (arm_system/perception/vision/detection/main.py:15)
- YOLO11s model with NCNN backend
- Half-precision inference for speed
- Input image size: 640x640
- Confidence threshold: 0.55 during inference

Detection Workflow:

Image Capture

Camera captures frame when VEX Brain detects an object via distance sensor

Object Detection

YOLO model processes the image and returns detection results

Classification

System filters for target classes (apple, orange, bottle)

Best Detection Selection

Highest confidence detection is selected for the object

Result Storage

Detection data (class, confidence, position) is sent back to main robot controller

Control Module

The control system manages all motors and actuators on the VEX Brain. VEX Control Module (arm_system/vex_brain/src/main.py:169) Hardware Configuration:

Motor	Port	Function	Torque Limit
Base Motor	PORT1	Rotates the arm base	100%
Shoulder Motor	PORT2	Controls shoulder joint	95%
Elbow Motor	PORT3	Controls elbow joint	95%
Gripper Motor	PORT4	Opens/closes gripper	100%

Key Features:

Inertial sensor-based base rotation (accurate angle control)
Torque-limited movement for safety
Current monitoring for grip detection
Synchronized multi-joint movements

Movement Control:

# Base rotation uses inertial sensor for precision
self.control.move_motor_to_angle(
    motor=self.control.base_motor,
    target=90,  # degrees
    speed=20    # RPM
)

Mapping Module

The mapping system builds a probabilistic occupancy grid of the environment. Occupancy Grid (arm_system/mapping/occupancy_grid.py:5)

Grid Representation

Grid size: 100x100 cells (configurable)
Resolution: 0.5 units per cell (configurable)
Cell values: -1 (unknown), 0-100 (occupancy probability)
Origin: Center of grid (50, 50)

Probabilistic Updates

Uses Bayesian inference to update cell occupancy:

Prior: Current cell probability
Sensor Model: 90% accuracy (configurable)
Posterior: Updated probability based on observation

Free cells in the sensor ray path are also updated to reduce false positives.

Scan Integration

Converts polar coordinates (angle, distance) from sensors to Cartesian grid coordinates:

world_x = robot_x + distance * cos(robot_theta + angle)
world_y = robot_y + distance * sin(robot_theta + angle)

Data Flow

The system follows this operational flow for autonomous pick-and-place:

1. System Initialization

Hardware Check

VEX Brain verifies all motors and sensors are installed

Serial Connection

Raspberry Pi establishes serial connection to VEX Brain

Camera Setup

Camera manager initializes video capture device

Model Loading

YOLO detection model is loaded into memory

2. Environment Scanning

3. Pick and Place Operation

Object Selection

User selects object from scanned list on Raspberry Pi

Base Rotation

VEX rotates base to object’s angle using inertial sensor

Arm Extension

Shoulder and elbow motors extend toward object based on distance

Grip Detection

Gripper closes while monitoring motor current (threshold: 0.5A)

Retraction

Arm lifts object using bumper switch for safety limit

Transport

Base rotates to placement zone (predefined angle per object class)

Release

Gripper opens to release object

Return Home

Arm returns to safe home position

4. Safety Protocol

The safety system continuously monitors for collision and overload conditions:

Bumper Switch: Triggers immediate stop and retraction if pressed during operation

Safety Features:

Torque limiting on shoulder and elbow (95%)
Current monitoring for grip force
Bumper switch collision detection
Timeout mechanisms (30s for scan, 20s for pick)
Emergency stop and safe retraction sequence

Hardware-Software Integration

Sensor Integration

Sensor	VEX Port	Purpose	Range
Inertial Sensor	Default	Base angle measurement	0-360°
Base Distance	PORT9	Object detection during scan	50-345mm
Gripper Distance	PORT7	Pick operation guidance	0-40mm
Bumper Switch	PORT10	Collision detection	Boolean
Touchled	PORT8	Visual status indicator	RGB

Message Protocol

All communication uses JSON over serial with newline delimiters:

{
  "type": "pick_service",
  "data": {
    "joint": "base",
    "angle": 90,
    "speed": 30
  }
}

Object Placement Zones

The system uses predefined placement zones based on object class:

placement_zones = {
    'apple': {'angle': 90, 'distance': 200},
    'orange': {'angle': 180, 'distance': 200},
    'bottle': {'angle': 45, 'distance': 200},
    'default': {'angle': 270, 'distance': 200}
}

Each detected object is automatically assigned to its corresponding zone during the scan phase.

Next Steps

Installation Guide

Set up the software environment on your Raspberry Pi

Hardware Setup

Learn about the physical components and assembly

Communication Details

Deep dive into the serial communication protocol

Quick Start Tutorial

Run your first autonomous operation

Getting Started

Hardware Setup

Software Architecture

Computer Vision

Tutorials

Architecture Overview

Raspberry Pi

VEX Brain

Camera System

System Architecture Diagram

Main Modules

Communication Module

Perception Module

Control Module

Mapping Module

Data Flow

1. System Initialization

2. Environment Scanning

3. Pick and Place Operation

4. Safety Protocol

Hardware-Software Integration

Sensor Integration

Message Protocol

Object Placement Zones

Next Steps

Installation Guide

Hardware Setup

Communication Details

Quick Start Tutorial

Build docs developers (and LLMs) love

Getting Started

Hardware Setup

Software Architecture

Computer Vision

Tutorials

​Architecture Overview

Raspberry Pi

VEX Brain

Camera System

​System Architecture Diagram

​Main Modules

​Communication Module

​Perception Module

​Control Module

​Mapping Module

​Data Flow

​1. System Initialization

​2. Environment Scanning

​3. Pick and Place Operation

​4. Safety Protocol

​Hardware-Software Integration

​Sensor Integration

​Message Protocol

​Object Placement Zones

​Next Steps

Installation Guide

Hardware Setup

Communication Details

Quick Start Tutorial

Build docs developers (and LLMs) love

Architecture Overview

System Architecture Diagram

Main Modules

Communication Module

Perception Module

Control Module

Mapping Module

Data Flow

1. System Initialization

2. Environment Scanning

3. Pick and Place Operation

4. Safety Protocol

Hardware-Software Integration

Sensor Integration

Message Protocol

Object Placement Zones

Next Steps