Grading System

Overview

Wecode uses an automatic grading system that compares submission outputs against expected outputs, applies late penalties, and calculates final scores.

Grading Methods

Output Comparison

Compare program output directly with expected output using diff commands

Tester Code

Use custom tester programs to validate complex outputs or special formats

Output Comparison Method

Diff Commands

The most common grading method uses diff commands to compare outputs:

Problem.php:14

protected $fillable = [
    'diff_cmd',  // e.g., 'diff', 'diff.rb', 'checker.exe'
    'diff_arg',  // e.g., '-bB', '--ignore-all-space'
    // ...
];

Common Diff Configurations

Exact Match
Ignore Whitespace
Custom Checker

$problem->diff_cmd = 'diff';
$problem->diff_arg = '';

Requires exact character-by-character match including whitespace.

$problem->diff_cmd = 'diff';
$problem->diff_arg = '-bB';

-b: Ignore changes in whitespace amount
-B: Ignore blank lines

$problem->diff_cmd = 'checker.rb';
$problem->diff_arg = '--tolerance=0.001';

Use custom scripts for floating-point comparison or special validation.

Test Case Scoring

Pre-Score Calculation

Submissions receive a pre_score from 0 to 10000 (representing 0% to 100%):

// Example: 8 out of 10 test cases passed
$passed_tests = 8;
$total_tests = 10;
$pre_score = ($passed_tests / $total_tests) * 10000; // = 8000

Problem Score Weight

Each problem in an assignment has a maximum score:

Assignment.php:46-51

public function problems()
{
    return $this->belongsToMany("App\Models\Problem")
        ->withPivot("score", "ordering", "problem_name")
        ->withTimestamps();
}

The weighted score is calculated:

Scoreboard.php:58-64

$pre_score = ceil(
    $submission->pre_score *
    ($problems[$submission->problem_id]->pivot->score ?? 0) / 10000
);

if ($submission['coefficient'] === 'error') 
    $final_score = 0;
else 
    $final_score = ceil($pre_score * $submission['coefficient'] / 100);

Score Example

Raw Score

Student passes 9/10 test cases: pre_score = 9000

Problem Weight

Problem worth 50 points: weighted = 9000 * 50 / 10000 = 45

Late Penalty

Submitted 1 hour late with coefficient 80: final = 45 * 80 / 100 = 36

Late Submission Penalties

Coefficient Calculation

Late penalties use the late_rule formula:

Assignment.php:181-207

private function _eval_coefficient($delay = 0, $submit_time = 0)
{
    try {
        $extra_time = $this->extra_time;
        $expressionLanguage = new ExpressionLanguage();
        $expressionLanguage->registerProvider(new my_expression_language_provider());

        $coefficient = $expressionLanguage->evaluate($this->late_rule, [
            'delay' => $delay,
            'extra_time' => $extra_time,
        ]);

        $coefficient = round($coefficient, 1);
        if ($coefficient < 0)
            $coefficient = max(-10000, $coefficient);
        else
            $coefficient = min(10000, $coefficient);

    } catch (\Throwable $e) {
        $coefficient = "error";
    }
    
    if (!isset($coefficient) || !is_numeric($coefficient)) {
        $coefficient = "error";
    }
    return $coefficient;
}

Available Math Functions

Assignment.php:16

$func_list = [
    'abs', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh',
    'base_convert', 'bindec', 'ceil', 'cos', 'cosh', 'decbin', 'dechex',
    'decoct', 'deg2rad', 'exp', 'expm1', 'fdiv', 'floor', 'fmod',
    'hexdec', 'hypot', 'intdiv', 'is_finite', 'is_infinite', 'is_nan',
    'log', 'log10', 'log1p', 'max', 'min', 'octdec', 'pi', 'pow',
    'rad2deg', 'round', 'sin', 'sinh', 'sqrt', 'tan', 'tanh'
];

Late Rule Examples

No Penalty
Grace Period
Linear Decay
Exponential Decay
Step Function

Full credit regardless of submission time.

delay <= extra_time ? 100 : 0

Full credit during extra time, 0 after.

max(0, 100 - (delay / 3600) * 10)

Lose 10% per hour late, minimum 0%.

100 * exp(-delay / 7200)

Exponential decay with 2-hour half-life.

delay <= 1800 ? 100 : delay <= 3600 ? 80 : delay <= 7200 ? 60 : 0

0-30 min: 100%
30-60 min: 80%
1-2 hours: 60%
After 2 hours: 0%

Update All Coefficients

When late_rule changes, update existing submissions:

Assignment.php:209-218

public function update_submissions_coefficient()
{
    foreach ($this->submissions as $sub) {
        $delay = $this->finish_time->diffInSeconds($sub->created_at);
        $submit_time = $this->start_time->diffInSeconds($sub->created_at);

        $sub->coefficient = $this->_eval_coefficient($delay, $submit_time);
        $sub->save();
    }
}

Coefficient Storage

Coefficient is stored with each submission:

Submission.php:26-27

protected $fillable = [
    'coefficient',  // Percentage (0-100) or "error"
    'pre_score',    // Raw score (0-10000)
    // ...
];

Current Coefficient

Check current coefficient before submitting:

Assignment.php:219-224

public function eval_coefficient()
{
    $delay = $this->finish_time->diffInSeconds(Carbon::now());
    $submit_time = $this->start_time->diffInSeconds(Carbon::now());
    return $this->_eval_coefficient($delay, $submit_time);
}

Automated Grading Workflow

Submission Flow

Submit Code

User uploads code file or pastes code

submission_controller.php:326-364

$submission = new Submission ([
    'assignment_id' => $assignment->id,
    'problem_id' => $problem->id,
    'user_id' => Auth::user()->id,
    'is_final' => 0,
    'status' => 'pending',
    'pre_score' => 0,
    'coefficient' => $coefficient,
    'language_id' => $language->id,
]);

Queue for Judging

Submission added to judging queue

submission_controller.php:204-212

private function add_to_queue($submission, $assignment, $file_name)
{
    $assignment->increment('total_submits');
    $submission->file_name = $file_name;
    $submission->save();

    Queue_item::add_and_process($submission->id, 'judge');
}

Execute and Test

Judge compiles/runs code against each test case, records:

Execution time
Memory usage
Verdict (AC, WA, TLE, MLE, RE, CE)

Calculate Score

Count passed test cases and calculate pre_score

$pre_score = ($passed_tests / $total_tests) * 10000;

Apply Penalties

Calculate final score with coefficient

$final_score = ceil($pre_score * $coefficient / 100);

Update Scoreboard

If is_final, update assignment scoreboard

submission_controller.php:393

Scoreboard::update_scoreboard($submission_curr->assignment_id);

Judgement Details

Detailed results stored in JSON:

Submission.php:28-30

protected $casts = [
    'judgement' => 'object'
];

Parsing Result HTML

Submission.php:66-86

public function get_judgement_from_result_html(){
    $result = mb_convert_encoding(
        file_get_contents($this->directory() . "/result-" . $this->id . ".html"), 
        'UTF-8'
    );
    $results = explode("</span>\n", $result);
    
    // Extract execution times and memory
    $times_and_mem = Arr::flatten(array_filter($results, function($i){
        return str_contains($i, 'text-muted');
    }));
    $times = array_map(function($i){ 
        return floatval(Str::before(Str::after($i, "<small>"), " s and"));
    }, $times_and_mem);
    $mems = array_map(function($i){ 
        return floatval(Str::before(Str::after($i, "s and "), " KiB"));
    }, $times_and_mem);
    
    // Extract verdicts
    $testcase_verdict = array_filter($results, function($s){
        return $s != '' && !Str::contains($s, ['text-muted', 'text-primary', 'text-success']);
    });
    $testcase_verdict = array_map(function($s) {
        return Str::before(Str::after($s, '>'), "<" );
    }, $testcase_verdict);
    
    // Count verdicts
    $verdicts = [];
    foreach ($testcase_verdict as $key => $value) {
        $verdicts[$value] = ($verdicts[$value] ?? 0) + 1;
    }

    return ["times" => $times, "mems" => $mems, "verdicts" => $verdicts];
}

Final Score Calculation

Complete scoring process:

submission_controller.php:438-449

private function _status(&$submission, $all_problems = null)
{
    if ($all_problems == null) 
        $all_problems = $submission->assignment->problems->keyBy('id');
    
    // Weighted score
    $score = (
        $submission->pre_score *
        ($all_problems[$submission->problem_id]->pivot->score ?? 100) /
        10000
    );
    
    // Apply coefficient
    if ($submission->coefficient == 'error')
        $submission->final_score = $score;
    else
        $submission->final_score = round($score * $submission->coefficient / 100, 0);
}

Get Final Submissions

Retrieve final submissions for grading/export:

Submission.php:88-94

public static function get_final_submissions($assignment_id)
{
    if (in_array(Auth::user()->role->name, ['admin', 'head_instructor', 'instructor']))
        return DB::table('users')
            ->join('submissions', 'users.id', '=', 'submissions.user_id')
            ->select('users.username', 'submissions.*')
            ->where(['assignment_id' => $assignment_id, 'is_final' => 1])
            ->orderBy('username','asc')
            ->orderBy('problem_id','asc')
            ->get();
    else
        return DB::table('users')
            ->join('submissions', 'users.id', '=', 'submissions.user_id')
            ->select('users.username', 'submissions.*')
            ->where([
                'assignment_id' => $assignment_id, 
                'is_final' => 1, 
                'username' => Auth::user()->username
            ])
            ->orderBy('username','asc')
            ->orderBy('problem_id','asc')
            ->get();
}

Verdict Types

Common submission verdicts:

AC

Accepted: All test cases passed

WA

Wrong Answer: Output doesn’t match expected

TLE

Time Limit Exceeded: Took too long

MLE

Memory Limit Exceeded: Used too much memory

RE

Runtime Error: Crashed during execution

CE

Compilation Error: Code didn’t compile

Best Practices

Test Cases

Include edge cases
Test boundary conditions
Verify expected outputs
Use diverse test data

Diff Configuration

Use -bB for most problems
Exact match for formatted output
Custom checkers for floats
Test diff commands manually

Late Penalties

Set fair penalty formulas
Test with example delays
Communicate policy clearly
Consider grace periods

Score Weights

Weight by difficulty
Ensure total makes sense
Balance problem values
Document scoring scheme

Score Range: All scores use integer math with pre_score from 0-10000 to avoid floating point errors.

Getting Started

Core Features

Advanced Features

Configuration

User Guides

​Overview

​Grading Methods

Output Comparison

Tester Code

​Output Comparison Method

​Diff Commands

​Common Diff Configurations

​Test Case Scoring

​Pre-Score Calculation

​Problem Score Weight

​Score Example

​Late Submission Penalties

​Coefficient Calculation

​Available Math Functions

​Late Rule Examples

​Update All Coefficients

​Coefficient Storage

​Current Coefficient

​Automated Grading Workflow

​Submission Flow

​Judgement Details

​Parsing Result HTML

​Final Score Calculation

​Get Final Submissions

​Verdict Types

AC

WA

TLE

MLE

RE

CE

​Best Practices

Test Cases

Diff Configuration

Late Penalties

Score Weights

Build docs developers (and LLMs) love

Overview

Grading Methods

Output Comparison Method

Diff Commands

Common Diff Configurations

Test Case Scoring

Pre-Score Calculation

Problem Score Weight

Score Example

Late Submission Penalties

Coefficient Calculation

Available Math Functions

Late Rule Examples

Update All Coefficients

Coefficient Storage

Current Coefficient

Automated Grading Workflow

Submission Flow

Judgement Details

Parsing Result HTML

Final Score Calculation

Get Final Submissions

Verdict Types

Best Practices