Skip to main content
Create your own evaluator to match your specific needs. You can start right away with custom criteria for full flexibility, or use one of our recommended formats as a starting point.

​
Do It Yourself

This option lets you write the evaluator prompt from scratch by adding the desired messages (System, Assistant, User, or Developer) and configuring the model along with its settings.

​
Generate Evaluator

The evaluator prompt can be automatically configured by Traceloop by clicking on the Generate Evaluator button. To enable the button, map the column you want to evaluate (such as an LLM response) and add any additional data columns required for prompt creation. Describe the evaluator’s purpose and reference the relevant data columns in the description. The system generates a prompt template that you can edit and customize as needed.

​
Test Evaluator

Before creating an evaluator, you can test it on existing Playground data. This allows you to refine and correct the evaluator prompt before saving the final version.

​
Execute Evaluator

Evaluators can be executed in playground columns and in experiments through the SDK.

​
Create via API

You can programmatically create custom LLM-as-a-judge evaluators using the API.
See the full API reference for detailed parameter documentation.