Wrong CLI / wrong geval
Symptoms: Commands like init, validate-contract, or demo are missing; or --eval is required.
Fix: You are not running the Geval release binary. Download it, use ./geval from that directory, or fix PATH so the release binary wins over a namesake from npm or another tool.
geval check expects --signals, not --eval
The open-source Geval CLI uses:
--eval flag in this product. Your eval pipeline should produce signals.json (or whatever name you pass) and point --signals at it.
Policy paths not found
Paths incontract.yaml are resolved relative to the contract file’s directory. Check:
- Working directory when you run
geval check - Typos in
policies/...paths - Files not committed in CI
geval validate-contract before check to fail fast.
Duplicate rule priorities
Symptom: Policy load error about duplicate priorities. Fix: Each rule in a policy must have a uniquepriority value.
Numeric vs presence rules
Symptom: A threshold rule never matches. Fix: Comparison operators (>, <, …) need a numeric value on the signal for that metric (and component, if used). Use operator: presence when you only care that a metric exists.
GitHub Actions: exit code not captured
Symptom:steps.<id>.outputs.exitcode is empty, or enforcement logic is wrong after geval check fails.
Cause: The default shell uses errexit. When ./geval check exits 1 or 2, the script stops before echo "exitcode=$?" >> $GITHUB_OUTPUT runs.
Fix: Use set +e, run check, save code=$?, then set -e and write $code to GITHUB_OUTPUT. See Optional: REQUIRE_APPROVAL does not fail the job in GitHub Actions.
Related: You do not have to generate signals.json in CI—committed files work after checkout. Confusion often comes from mixing “LangSmith export in CI” with “GitHub can’t see the repo.”
REQUIRE_APPROVAL in CI
Symptom: You want PRs to stay “green” when Geval returns REQUIRE_APPROVAL (exit 1) but red on BLOCK (exit 2).
Fix: You must capture the exit code with set +e and branch in a later step or job. Patterns for labels, needs: jobs, and geval approve are documented under Handling REQUIRE_APPROVAL: extra logic on GitHub Actions.