Skip to content

Agentic Development Maturity Assessment

This document assesses how "closed loop" DSS is for autonomous agent development, identifying what's automated vs. what requires human intervention.

Maturity Rating Scale

Level Description Human Role
L0 Manual Human does everything
L1 Assisted Agent helps, human decides
L2 Partial Agent does routine, human handles exceptions
L3 Conditional Agent autonomous, human approves critical actions
L4 High Agent autonomous, human monitors
L5 Full Agent autonomous, human only for policy changes

Phase-by-Phase Assessment

Phase 1: Design Spec Creation

Capability Level Status Gap
Create initial spec L1 Partial No dss init scaffolding
Add components L2 Partial No schema validation during edit
Add LLM context L2 ✅ Done Agent can use lint_spec to verify
Configure validators L2 ✅ Done validators-configured lint rule
Spec versioning L0 Missing No version bump automation

Overall: L1-L2 - Agent can assist but human creates/edits spec files

Human-in-the-Loop: Required for design decisions (which components, what variants, brand guidelines)

Phase 2: Code Generation

Capability Level Status Gap
Generate LLM context L4 ✅ Done generate_prompt tool
Query components L4 ✅ Done get_component, list_components
Get implementation guidance L4 ✅ Done get_variants, get_props, get_anti_patterns
Generate code from spec L3 External Depends on coding agent (Claude, Copilot)

Overall: L3-L4 - Agent can generate code autonomously using spec context

Human-in-the-Loop: Review generated code for business logic, UX decisions

Phase 3: Validation

Capability Level Status Gap
Static code validation L4 ✅ Done validate_file, validate_directory
Color token compliance L4 ✅ Done check_colors
Spacing compliance L4 ✅ Done check_spacing
Accessibility (static) L3 ✅ Done img-alt-required, button-accessible-name
Accessibility (runtime) L1 Delegated Requires agent-a11y integration
Visual regression L0 Missing No visual comparison
Cross-browser testing L0 Missing Requires w3pilot integration
API style compliance L1 Delegated Requires spectral integration

Overall: L3 - Static validation automated, runtime/visual needs integration

Human-in-the-Loop: - Visual correctness ("does it look right?") - Semantic appropriateness ("is this the right component?") - Runtime behavior validation

Phase 4: Fix

Capability Level Status Gap
Auto-fix colors L4 ✅ Done fix_colors
Auto-fix spacing L4 ✅ Done fix_spacing
Auto-fix accessibility L3 ✅ Done fix_accessibility (adds placeholders)
Verify fixes worked L4 ✅ Done fix_and_verify
Fix loop until convergence L4 ✅ Done run_fix_loop
Fix semantic issues L1 Missing Requires LLM reasoning
Fix anti-pattern violations L1 Missing Requires code refactoring

Overall: L3-L4 - Token fixes automated, semantic fixes need human

Human-in-the-Loop: - Review placeholder text (alt, aria-label) - Fix anti-patterns (requires code restructuring) - Approve fixes that change behavior

Phase 5: Release Gate

Capability Level Status Gap
Compliance scoring L4 ✅ Done generate_compliance_report
Pass/fail decision L4 ✅ Done check_release_gate
Blocking issue detection L4 ✅ Done Categories marked as blocking
Certificate generation L4 ✅ Done get_compliance_certificate
Version detection L1 Missing No semantic version analysis
Changelog generation L1 External Requires schangelog integration
Git tagging L2 Missing No tag automation
CI integration L2 Missing No GitHub Action

Overall: L3 - Gate decision automated, release execution manual

Human-in-the-Loop: - Final approval for release - Version number decision - Release notes review

Phase 6: External Validation

Capability Level Status Gap
Validator discovery L4 ✅ Done list_validators
Validator invocation info L4 ✅ Done get_validator_invocation
Automatic invocation L1 Missing Agent must manually invoke
Result aggregation L1 Missing No unified result format
Cross-validator orchestration L1 Missing Agent must coordinate

Overall: L2 - Discovery automated, orchestration manual

Human-in-the-Loop: - Configure which validators to use - Interpret validator results - Resolve conflicts between validators

Overall Maturity Score

Phase Current Target Gap
Spec Creation L1.5 L3 dss init, schema validation
Code Generation L3.5 L4 External coding agent
Validation L3 L4 Visual/runtime validation
Fix L3.5 L4 Semantic fixes
Release Gate L3 L4 CI/CD integration
External Validation L2 L4 Auto-invocation, aggregation
Average L2.8 L4

Critical Human-in-the-Loop Points

Always Required (Policy)

Decision Why Human Needed
Design system principles Brand identity, creative direction
Component intent/purpose Business requirements
Accessibility level target Legal/compliance decision
Release approval Accountability

Currently Required (Gaps)

Decision Why Human Needed Path to Automation
Visual correctness No visual comparison Integrate visual regression testing
Runtime behavior No browser testing Integrate w3pilot fully
Semantic fixes Requires reasoning LLM-powered fix suggestions
Version bumping No commit analysis Integrate conventional commits
Changelog No release notes Integrate schangelog

Exception Handling

Scenario Current Target
Unfixable violation Human reviews Agent escalates with context
Conflicting validators Human decides Agent proposes resolution
Score below threshold Human investigates Agent provides remediation plan
Regression detected Loop stops Agent rolls back and reports

Recommendations to Reach L4

High Priority

  1. Visual Validation Integration
  2. Integrate Chromatic/Percy for visual regression
  3. Add screenshot comparison to compliance report

  4. Blocks: Visual correctness is currently L0

  5. External Validator Auto-Invocation

  6. Add invoke_validator tool
  7. Aggregate results into compliance report

  8. Blocks: Agent must manually coordinate validators

  9. CI/CD GitHub Action

  10. Create dss-action for PR validation
  11. Auto-comment compliance report on PRs
  12. Blocks: No automated PR feedback

Medium Priority

  1. Semantic Fix Suggestions
  2. LLM-powered fix reasoning for anti-patterns
  3. Generate refactoring suggestions

  4. Blocks: Anti-pattern fixes are L1

  5. Release Automation

  6. Version detection from commits
  7. Changelog generation
  8. Git tag creation

  9. Blocks: Release execution is manual

  10. dss init Scaffolding

  11. Interactive spec creation
  12. Template selection
  13. Blocks: New projects start from scratch

Lower Priority

  1. Schema Validation During Edit
  2. Real-time validation in IDE

  3. JSON Schema LSP integration

  4. Cross-Validator Orchestration

  5. Unified validator result format
  6. Conflict resolution rules

Closed Loop Score Card

┌─────────────────────────────────────────────────────────────────┐
│                    DSS AGENTIC MATURITY                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Spec Creation      [████████░░░░░░░░░░░░] L1.5                │
│  Code Generation    [██████████████░░░░░░] L3.5                │
│  Validation         [████████████░░░░░░░░] L3.0                │
│  Fix                [██████████████░░░░░░] L3.5                │
│  Release Gate       [████████████░░░░░░░░] L3.0                │
│  External Valid.    [████████░░░░░░░░░░░░] L2.0                │
│                                                                 │
│  ─────────────────────────────────────────────────────────────  │
│  Overall            [███████████░░░░░░░░░] L2.8 / L5           │
│                                                                 │
│  Target for Full Autonomy: L4+                                  │
│  Current Gap: 1.2 levels                                        │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Human-in-the-Loop Summary

Category Count Examples
Always Human 4 Design principles, component intent, a11y level, release approval
Currently Human (fixable) 5 Visual validation, runtime testing, semantic fixes, versioning, changelog
Exception Handling 4 Unfixable violations, validator conflicts, low scores, regressions

Total Human Touchpoints: 13 (4 permanent + 5 fixable + 4 exceptions)

Path to L4: Address the 5 fixable gaps to reduce routine human touchpoints from 9 to 4.