v0.7.0¶

Release Date: 2026-06-07

Highlights¶

Report Validation: Comprehensive validation for rubric evaluation reports
Enum Constraints: JSON Schema enhanced with enum values for all fields
Lint Command: New sevaluation lint command for validating report correctness

Overview¶

v0.7.0 introduces report validation capabilities to catch common errors in evaluation reports before they cause issues in downstream processing. The validation checks enum values (score, severity, decision status), verifies count accuracy, and ensures semantic consistency between decisions and findings.

New Features¶

Report Validation¶

Validate rubric reports for correctness:

import "github.com/plexusone/structured-evaluation/rubric"

result := rubric.ValidateReport(&report)

if !result.Valid {
    fmt.Printf("Invalid report: %d errors, %d warnings\n",
        result.ErrorCount, result.WarningCount)
    for _, issue := range result.Issues {
        fmt.Printf("[%s] %s: %s\n", issue.Severity, issue.Path, issue.Message)
    }
}

Validation Checks¶

Check	Level	Description
Enum Values	Error	Score, severity, decision status must be valid enum values
Required Fields	Error	metadata.document and reviewType are required
Finding Title	Error	Each finding must have a title
Count Accuracy	Warning	Reported category/finding counts must match actual data
Decision Consistency	Warning	Decision should align with blocking findings and failed categories
OverallDecision Match	Warning	overallDecision should match decision.status

Enum Helper Functions¶

Get valid enum values for cross-language tooling:

// Get all valid values
scores := rubric.ValidScoreValues()         // ["pass", "partial", "fail"]
severities := rubric.ValidSeverityValues()  // ["critical", "high", "medium", "low", "info"]
statuses := rubric.ValidDecisionStatusValues() // ["pass", "conditional", "fail", "human_review"]

CLI Lint Command¶

Validate reports from the command line:

# Basic validation
sevaluation lint report.json

# Strict mode (warnings are errors)
sevaluation lint report.json --strict

# JSON output
sevaluation lint report.json --format=json

Exit codes:

0: Valid (no errors, warnings allowed unless --strict)
1: Invalid (has errors, or has warnings with --strict)

Enum Constraints in Schema¶

The rubric.schema.json now includes enum constraints:

{
  "score": {
    "type": "string",
    "enum": ["pass", "partial", "fail"],
    "description": "Category score: pass, partial, or fail"
  },
  "severity": {
    "type": "string",
    "enum": ["critical", "high", "medium", "low", "info"],
    "description": "Finding severity level"
  }
}

Cross-Language Enums¶

A new schema/enums.json file provides enum definitions for non-Go tooling:

{
  "enums": {
    "ScoreValue": ["pass", "partial", "fail"],
    "Severity": ["critical", "high", "medium", "low", "info"],
    "DecisionStatus": ["pass", "conditional", "fail", "human_review"]
  },
  "validation": {
    "severity": {
      "blocking": ["critical", "high"]
    },
    "decisionStatus": {
      "passing": ["pass", "conditional"]
    }
  }
}

Use Cases¶

CI/CD Pipeline Validation¶

Add report validation to your CI pipeline:

#!/bin/bash
# Validate all evaluation reports before processing
for report in reports/*.json; do
    if ! sevaluation lint "$report" --strict; then
        echo "Invalid report: $report"
        exit 1
    fi
done

Pre-Submission Validation¶

Validate reports before submission in your agent workflow:

report := generateEvaluationReport()

result := rubric.ValidateReport(report)
if result.HasErrors() {
    return fmt.Errorf("invalid report: %s", result.String())
}
if result.HasWarnings() {
    log.Warn("Report has warnings", "count", result.WarningCount)
}

// Submit report
submitReport(report)

Schema Validation in Other Languages¶

Use enums.json to validate reports in Python, JavaScript, etc.:

import json

with open('schema/enums.json') as f:
    enums = json.load(f)

def validate_score(score):
    return score in enums['enums']['ScoreValue']

def is_blocking_severity(severity):
    return severity in enums['validation']['severity']['blocking']

Migration Guide¶

Additive Changes¶

All changes in v0.7.0 are additive:

New ValidateReport() function
New helper functions (ValidScoreValues(), etc.)
New lint command
Updated schema with enum constraints (stricter, but backwards compatible)

Recommended Updates¶

Add validation to your report generation:

report := generateReport()
result := rubric.ValidateReport(report)
if !result.Valid {
    // Handle validation errors
}

Update CI to use lint command:

- name: Validate reports
  run: sevaluation lint output/*.json --strict

Regenerate schema files to get enum constraints:

sevaluation schema -o schema/

Documentation¶

Updated CLI reference with lint command
Schema documentation updated with enum constraints

Full Changelog¶

See CHANGELOG.md for the complete list of changes.