Skip to main content
Consensus tools help you measure inter-annotator agreement and identify areas where labelers disagree, enabling you to improve label quality and identify ambiguous data.

​
Methods

​
get_summary

Get a summary of consensus metrics for a project. 
client.consensus.get_summary(project_uid="proj_123")
​
project_uid
str
required
UID of the project
​
project_uid
str
UID of the project
​
overall_agreement
float
Overall agreement score across all tasks (0.0 to 1.0)
​
tasks_analyzed
int
Number of tasks included in consensus analysis
​
annotators_count
int
Number of annotators contributing to the project
​
avg_annotations_per_task
float
Average number of annotations per task
​
low_consensus_tasks
int
Number of tasks with low annotator agreement

​
list_scores

List consensus scores for individual tasks. 
client.consensus.list_scores(
    project_uid="proj_123",
    limit=100,
    cursor="next_page_token"
)
​
project_uid
str
required
UID of the project
​
limit
int
Maximum number of scores to return per page
​
cursor
str
Pagination cursor for fetching the next page
​
data
list[ConsensusScore]
List of consensus score objects
​
data[].task_uid
str
UID of the task
​
data[].consensus_score
float
Agreement score for this task (0.0 to 1.0)
​
data[].annotation_count
int
Number of annotations for this task
​
data[].annotator_count
int
Number of unique annotators
​
next_cursor
str
Cursor for the next page of results

​
compute

Compute consensus scores for a project. 
client.consensus.compute(project_uid="proj_123")
​
project_uid
str
required
UID of the project to compute consensus for
​
status
str
Status of the computation (e.g., “completed”, “in_progress”)
​
tasks_processed
int
Number of tasks processed
​
computation_time
float
Time taken to compute consensus (in seconds)
​
overall_agreement
float
Overall agreement score

​
get_config

Get consensus configuration for a project. 
client.consensus.get_config(project_uid="proj_123")
​
project_uid
str
required
UID of the project
​
iou_threshold
float
Intersection-over-union threshold for bounding box agreement (0.0 to 1.0)
​
min_agreement_ratio
float
Minimum agreement ratio to consider consensus achieved (0.0 to 1.0)
​
min_annotations
int
Minimum number of annotations required per task for consensus calculation

​
update_config

Update consensus configuration for a project. 
client.consensus.update_config(
    project_uid="proj_123",
    iou_threshold=0.5,
    min_agreement_ratio=0.7,
    min_annotations=3
)
​
project_uid
str
required
UID of the project
​
iou_threshold
float
Intersection-over-union threshold for bounding box agreement (0.0 to 1.0). Higher values require stricter overlap.
​
min_agreement_ratio
float
Minimum agreement ratio to consider consensus achieved (0.0 to 1.0)
​
min_annotations
int
Minimum number of annotations required per task for consensus calculation
​
iou_threshold
float
Updated IoU threshold
​
min_agreement_ratio
float
Updated minimum agreement ratio
​
min_annotations
int
Updated minimum annotations requirement

​
Async Methods

All methods are available in async form through AsyncAvala: 
await client.consensus.get_summary(project_uid="proj_123")
await client.consensus.list_scores(project_uid="proj_123")
await client.consensus.compute(project_uid="proj_123")
await client.consensus.get_config(project_uid="proj_123")
await client.consensus.update_config(
    project_uid="proj_123",
    iou_threshold=0.5
)

Build docs developers (and LLMs) love

Get started for freeTalk to us