Skip to main content

​
Overview

Source configuration tells Geval how to parse and aggregate metrics from CSV and JSON files. This is especially useful when working with exports from tools like LangSmith, Braintrust, or custom eval scripts.

​
CSV Source Configuration

​
Basic Configuration

sources:
  csv:
    metrics:
      - column: accuracy
        aggregate: avg
      - column: latency_ms
        aggregate: p95
        as: latency_p95
    evalName:
      fixed: my-eval

​
Advanced Configuration

sources:
  csv:
    metrics:
      - column: accuracy          # Column name in CSV
        aggregate: avg            # Aggregation method
        as: accuracy_score        # Optional: rename metric
      - column: latency
        aggregate: p95
        as: latency_p95
      - column: status
        aggregate: pass_rate
    evalName:
      fixed: my-eval             # Fixed eval name
      # OR
      # column: eval_name        # Extract from column
    runId:
      fixed: run-123             # Fixed run ID
      # OR
      # column: run_id           # Extract from column
    timestamp: created_at         # Optional: column for timestamp

​
Example: LangSmith Export

sources:
  csv:
    metrics:
      - column: accuracy
        aggregate: avg
      - column: latency_ms
        aggregate: p95
        as: latency_p95
      - column: status
        aggregate: pass_rate
        as: success_rate
    evalName:
      fixed: langsmith-eval

​
Example: Multiple Metrics

sources:
  csv:
    metrics:
      - column: accuracy
        aggregate: avg
      - column: quality_score
        aggregate: avg
      - column: latency
        aggregate: p95
        as: latency_p95
      - column: cost
        aggregate: sum
        as: total_cost
      - column: passed
        aggregate: pass_rate
    evalName:
      fixed: comprehensive-eval

​
JSON Source Configuration

​
Basic JSON Config

sources:
  json:
    metrics:
      - column: accuracy
        aggregate: first
    evalName:
      fixed: my-eval

​
JSON with Nested Paths

sources:
  json:
    metrics:
      - column: score
        aggregate: avg
    json:
      resultsPath: data.results   # Path to array in JSON
    evalName:
      fixed: nested-eval

​
Auto-Detection

JSON files in normalized format are auto-detected and don’t require source config: 
{
  "evalName": "my-eval",
  "runId": "run-123",
  "metrics": {
    "accuracy": 0.95,
    "latency_p95": 200
  }
}

​
Working with Different Eval Tools

​
Promptfoo

Promptfoo outputs are auto-detected. No source config needed for standard format.

​
LangSmith

Use CSV source config for LangSmith exports: 
sources:
  csv:
    metrics:
      - column: accuracy
        aggregate: avg
      - column: latency_ms
        aggregate: p95
    evalName:
      fixed: langsmith-export

​
Braintrust

Similar to LangSmith, use CSV source config: 
sources:
  csv:
    metrics:
      - column: score
        aggregate: avg
      - column: passed
        aggregate: pass_rate
    evalName:
      fixed: braintrust-eval

​
Custom JSON Formats

For custom JSON formats, use JSON source config: 
sources:
  json:
    metrics:
      - column: my_metric
        aggregate: avg
    json:
      resultsPath: results.data  # Custom path
    evalName:
      fixed: custom-eval

​
Best Practices

​
1. Use Descriptive Eval Names

evalName:
  fixed: production-quality-gate

​
2. Rename Metrics for Clarity

metrics:
  - column: latency_ms
    aggregate: p95
    as: latency_p95_ms  # Clear naming

​
3. Extract from Columns When Possible

evalName:
  column: eval_name  # Extract from CSV column

​
4. Combine Multiple Aggregations

metrics:
  - column: latency
    aggregate: p50
    as: latency_median
  - column: latency
    aggregate: p95
    as: latency_p95
  - column: latency
    aggregate: p99
    as: latency_p99

​
Troubleshooting

​
CSV Not Parsing

  • Verify column names match exactly (case-sensitive)
  • Check for extra spaces in column names
  • Ensure CSV has headers

​
Metrics Not Found

  • Verify column names in source config match CSV columns
  • Check that aggregation method is valid
  • Ensure data types are compatible

​
JSON Path Issues

  • Verify resultsPath points to an array
  • Check JSON structure matches expected format
  • Use dot notation for nested paths