Skip to main content

​
Classification Types

​
ClassificationProps

Props for the useClassification hook.
​
model
object
required
Model configuration object
​
preventLoad
boolean
Boolean that can prevent automatic model loading (and downloading the data if you load it for the first time) after running the hook.

​
ClassificationType

Return type for the useClassification hook. Manages the state and operations for Computer Vision image classification.
​
error
RnExecutorchError | null
Contains the error object if the model failed to load, download, or encountered a runtime error during classification.
​
isReady
boolean
Indicates whether the classification model is loaded and ready to process images.
​
isGenerating
boolean
Indicates whether the model is currently processing an image.
​
downloadProgress
number
Represents the download progress of the model binary as a value between 0 and 1.
​
forward
(imageSource: string) => Promise<{ [category: string]: number }>
Executes the model’s forward pass to classify the provided image.Parameters:
  • imageSource (string) - A string representing the image source (e.g., a file path, URI, or base64 string) to be classified.
Returns: A Promise that resolves to the classification result (typically containing labels and confidence scores).Throws: RnExecutorchError if the model is not loaded or is currently processing another image.

​
Object Detection Types

​
Bbox

Represents a bounding box for a detected object in an image.
​
x1
number
required
The x-coordinate of the bottom-left corner of the bounding box.
​
y1
number
required
The y-coordinate of the bottom-left corner of the bounding box.
​
x2
number
required
The x-coordinate of the top-right corner of the bounding box.
​
y2
number
required
The y-coordinate of the top-right corner of the bounding box.

​
Detection

Represents a detected object within an image, including its bounding box, label, and confidence score.
​
bbox
Bbox
required
The bounding box of the detected object, defined by its top-left (x1, y1) and bottom-right (x2, y2) coordinates.
​
label
keyof L
required
The class label of the detected object.
​
score
number
required
The confidence score of the detection, typically ranging from 0 to 1.

​
ObjectDetectionModelSources

Per-model config for ObjectDetectionModule.fromModelName. Each model name maps to its required fields. 
type ObjectDetectionModelSources =
  | { modelName: 'ssdlite-320-mobilenet-v3-large'; modelSource: ResourceSource }
  | { modelName: 'rf-detr-nano'; modelSource: ResourceSource };

​
ObjectDetectionModelName

Union of all built-in object detection model names. 
type ObjectDetectionModelName = ObjectDetectionModelSources['modelName'];

​
ObjectDetectionConfig

Configuration for a custom object detection model.
​
labelMap
T extends LabelEnum
required
The label map for the model.
​
preprocessorConfig
object
Optional preprocessor configuration

​
ObjectDetectionProps

Props for the useObjectDetection hook.
​
model
C extends ObjectDetectionModelSources
required
The model config containing modelName and modelSource.
​
preventLoad
boolean
Boolean that can prevent automatic model loading (and downloading the data if you load it for the first time) after running the hook.

​
ObjectDetectionType

Return type for the useObjectDetection hook. Manages the state and operations for Computer Vision object detection tasks.
​
error
RnExecutorchError | null
Contains the error object if the model failed to load, download, or encountered a runtime error during detection.
​
isReady
boolean
Indicates whether the object detection model is loaded and ready to process images.
​
isGenerating
boolean
Indicates whether the model is currently processing an image.
​
downloadProgress
number
Represents the download progress of the model binary as a value between 0 and 1.
​
forward
(input: string | PixelData, detectionThreshold?: number) => Promise<Detection<L>[]>
Executes the model’s forward pass with automatic input type detection.Parameters:
  • input (string | PixelData) - Image source (string path/URI or PixelData object)
  • detectionThreshold (number, optional) - An optional number between 0 and 1 representing the minimum confidence score. Default is 0.7.
Returns: A Promise that resolves to an array of Detection objects.Throws: RnExecutorchError if the model is not loaded or is currently processing another image.
​
runOnFrame
((frame: Frame, detectionThreshold: number) => Detection<L>[]) | null
Synchronous worklet function for real-time VisionCamera frame processing. Automatically handles native buffer extraction and cleanup.Use this for VisionCamera frame processing in worklets. For async processing, use forward() instead.Available after model is loaded (isReady: true).Parameters:
  • frame (Frame) - VisionCamera Frame object
  • detectionThreshold (number) - The threshold for detection sensitivity.
Returns: Array of Detection objects representing detected items in the frame.

​
Semantic Segmentation Types

​
SemanticSegmentationConfig

Configuration for a custom semantic segmentation model.
​
labelMap
T extends LabelEnum
required
The enum-like object mapping class names to indices.
​
preprocessorConfig
object
Optional preprocessing parameters.

​
SemanticSegmentationModelSources

Per-model config for SemanticSegmentationModule.fromModelName. Each model name maps to its required fields. 
type SemanticSegmentationModelSources =
  | { modelName: 'deeplab-v3-resnet50'; modelSource: ResourceSource }
  | { modelName: 'deeplab-v3-resnet101'; modelSource: ResourceSource }
  | { modelName: 'deeplab-v3-mobilenet-v3-large'; modelSource: ResourceSource }
  | { modelName: 'lraspp-mobilenet-v3-large'; modelSource: ResourceSource }
  | { modelName: 'fcn-resnet50'; modelSource: ResourceSource }
  | { modelName: 'fcn-resnet101'; modelSource: ResourceSource }
  | { modelName: 'deeplab-v3-resnet50-quantized'; modelSource: ResourceSource }
  | { modelName: 'deeplab-v3-resnet101-quantized'; modelSource: ResourceSource }
  | { modelName: 'deeplab-v3-mobilenet-v3-large-quantized'; modelSource: ResourceSource }
  | { modelName: 'lraspp-mobilenet-v3-large-quantized'; modelSource: ResourceSource }
  | { modelName: 'fcn-resnet50-quantized'; modelSource: ResourceSource }
  | { modelName: 'fcn-resnet101-quantized'; modelSource: ResourceSource }
  | { modelName: 'selfie-segmentation'; modelSource: ResourceSource };

​
SemanticSegmentationModelName

Union of all built-in semantic segmentation model names. 
type SemanticSegmentationModelName = SemanticSegmentationModelSources['modelName'];

​
DeeplabLabel

Labels used in the DeepLab semantic segmentation model. 
enum DeeplabLabel {
  BACKGROUND,
  AEROPLANE,
  BICYCLE,
  BIRD,
  BOAT,
  BOTTLE,
  BUS,
  CAR,
  CAT,
  CHAIR,
  COW,
  DININGTABLE,
  DOG,
  HORSE,
  MOTORBIKE,
  PERSON,
  POTTEDPLANT,
  SHEEP,
  SOFA,
  TRAIN,
  TVMONITOR,
}

​
SelfieSegmentationLabel

Labels used in the selfie semantic segmentation model. 
enum SelfieSegmentationLabel {
  SELFIE,
  BACKGROUND,
}

​
SemanticSegmentationProps

Props for the useSemanticSegmentation hook.
​
model
C extends SemanticSegmentationModelSources
required
The model config containing modelName and modelSource.
​
preventLoad
boolean
Boolean that can prevent automatic model loading (and downloading the data if you load it for the first time) after running the hook.

​
SemanticSegmentationType

Return type for the useSemanticSegmentation hook. Manages the state and operations for semantic segmentation models.
​
error
RnExecutorchError | null
Contains the error object if the model failed to load, download, or encountered a runtime error during segmentation.
​
isReady
boolean
Indicates whether the segmentation model is loaded and ready to process images.
​
isGenerating
boolean
Indicates whether the model is currently processing an image.
​
downloadProgress
number
Represents the download progress of the model binary as a value between 0 and 1.
​
forward
function
Executes the model’s forward pass to perform semantic segmentation on the provided image.Parameters:
  • imageSource (string) - A string representing the image source (e.g., a file path, URI, or base64 string) to be processed.
  • classesOfInterest (K[], optional) - An optional array of label keys indicating which per-class probability masks to include in the output. ARGMAX is always returned regardless.
  • resizeToInput (boolean, optional) - Whether to resize the output masks to the original input image dimensions. If false, returns the raw model output dimensions. Defaults to true.
Returns: A Promise resolving to an object with an 'ARGMAX' Int32Array of per-pixel class indices, and each requested class label mapped to a Float32Array of per-pixel probabilities.Throws: RnExecutorchError if the model is not loaded or is currently processing another image.

Build docs developers (and LLMs) love

Get started for freeTalk to us