Skip to main content

​
Overview

Geval uses standard exit codes that CI/CD systems can interpret to make automated decisions. This allows seamless integration with any CI/CD platform.

​
Exit Code Reference

CodeStatusDescriptionCI/CD Behavior
0PASSAll evals passed contract requirements✅ Pipeline continues
1BLOCKContract violated, release blocked❌ Pipeline fails, merge blocked
2REQUIRES_APPROVALApproval needed to proceed⚠️ Pipeline fails, requires manual review
3ERRORExecution error❌ Pipeline fails, indicates system error

​
Usage in CI/CD

​
GitHub Actions

GitHub Actions automatically interprets exit codes: 
- name: Check eval results
  run: geval check --contract contract.yaml --eval results.csv
  # Exit code 0 = step passes
  # Exit code 1+ = step fails, PR blocked

​
GitLab CI

GitLab CI uses exit codes to determine job status: 
check-evals:
  script:
    - geval check --contract contract.yaml --eval results.csv
  # Exit code 0 = job passes
  # Exit code 1+ = job fails, MR blocked

​
Custom Handling

You can handle exit codes explicitly: 
#!/bin/bash
geval check --contract contract.yaml --eval results.csv
EXIT_CODE=$?

case $EXIT_CODE in
  0)
    echo "✅ All checks passed"
    exit 0
    ;;
  1)
    echo "❌ Contract violated - blocking merge"
    exit 1
    ;;
  2)
    echo "⚠️ Approval required"
    # Send notification, create ticket, etc.
    exit 2
    ;;
  3)
    echo "💥 Error occurred"
    exit 3
    ;;
esac

​
Status Meanings

​
PASS (0)

All evaluation rules passed. Safe to merge and deploy. 
$ geval check --contract contract.yaml --eval results.csv
✓ PASS

Contract:    quality-gate
Version:     1

All 1 eval(s) passed contract requirements
CI/CD Action: Pipeline continues, merge allowed.

​
BLOCK (1)

One or more rules failed. Do not merge or deploy. 
$ geval check --contract contract.yaml --eval results.csv
✗ BLOCK

Contract:    quality-gate
Version:     1

Blocked: 2 violation(s) in 1 eval

Violations
  1. quality-metrics → accuracy
     accuracy = 0.72, expected >= 0.85
CI/CD Action: Pipeline fails, merge blocked.

​
REQUIRES_APPROVAL (2)

Rules failed but configured to require approval instead of blocking. 
on_violation:
  action: require_approval
  message: "Manual review required"
CI/CD Action: Pipeline fails, but can be overridden with approval.

​
ERROR (3)

System error occurred (file not found, invalid contract, etc.). 
$ geval check --contract missing.yaml --eval results.csv
Error: Contract file not found: missing.yaml
CI/CD Action: Pipeline fails, indicates configuration issue.

​
JSON Output

When using --json flag, exit codes still apply, but you also get structured output: 
geval check --contract contract.yaml --eval results.csv --json

{
  "status": "PASS",
  "contract": {
    "name": "quality-gate",
    "version": 1
  },
  "evals": [
    {
      "name": "quality-metrics",
      "status": "PASS",
      "rules": []
    }
  ],
  "violations": []
}

​
Best Practices

  1. Always check exit codes - Don’t ignore them in scripts
  2. Use JSON for automation - Parse structured output for custom handling
  3. Set appropriate actions - Use block for critical, warn for non-critical
  4. Handle errors gracefully - Exit code 3 indicates configuration issues
  5. Document expectations - Make it clear what each exit code means in your CI/CD

​
Integration Examples

​
GitHub Actions with Custom Handling

- name: Check evals
  id: eval-check
  run: |
    geval check --contract contract.yaml --eval results.csv --json > decision.json
    echo "status=$(jq -r '.status' decision.json)" >> $GITHUB_OUTPUT
  continue-on-error: true

- name: Comment on PR
  if: always() && steps.eval-check.outcome != 'success'
  uses: actions/github-script@v7
  with:
    script: |
      const decision = JSON.parse(fs.readFileSync('decision.json', 'utf8'));
      // Custom handling based on status

​
GitLab CI with Artifacts

check-evals:
  script:
    - geval check --contract contract.yaml --eval results.csv --json > decision.json
  artifacts:
    when: always
    paths:
      - decision.json
    reports:
      junit: decision.json