Agent as Judge

Lytix supports evaluating agentic workflows. This means not only do we evaluate the input/output of the flow, but also can pass in a data source (e.g. repository) to further evaluate the output.

​

Create a Test Suite

The first step is to create a test suite. This is a group of tests that will be run together.

lytix agent-test create-suite --suiteName "suite0"

​

Update config.json

After creating the test suite, you’ll need to update the config.json file to define what repository you’d like to evaluate.

{
  "repository": {
    "remote": "github",
    "branch": "main",
    "repository": "Lytix-Labs/optimodel"
  }
}

​

Create a Test in a Suite

lytix agent-test add-test --testName "test0" --suiteName "suite0"

​

Update start.py

To collect test data, we want to allow full flexibility. Thus, we create a start.py file in the folder of the test. This file will be executed to collect the data.

The only requirement is that the start.py file prints the following JSON object to stdout:

{
  "messages": ["..."],
  "output": "...",
  "sources": ["..."]
}

Where messages is an array of {role: "user" | "assistant", content: "..."} and sources is an array of file paths that contain the data we want to evaluate the output against.

​

Run a Test Suite

lytix agent-test run --suiteName "suite0"