# Set Up Scorecards

**How to Set Up and Use Evaluation Scorecards** 

Evaluation scorecards let you define what good performance looks like for a digital worker and then measure every interaction against those criteria automatically. This guide walks through creating a scorecard, attaching rubrics, and reviewing results. 

 

**Step 1: Create a Scorecard** 

1. Navigate to the evaluation section for the digital worker you want to measure. 
2. Create a new scorecard and give it a descriptive name that reflects its purpose, for example Dispatch Triage Quality or Support Resolution Quality. 
3. Add score categories. Categories are high-level measurement dimensions. Common examples include Policy and Process Adherence, Accuracy and Resolution Quality, Communication Quality, and Safety and Compliance. 
4. Assign a weight to each category so the scores roll up into a meaningful overall score. For example: 40% accuracy, 30% process adherence, 20% communication, 10% compliance. Weights must total 100%. 
5. Save and move on to creating rubric criteria for each category. 

 

**Step 2: Create Rubric Criteria** 

Rubrics are the specific checks used to score each interaction. For each category in your scorecard, create one or more rubric criteria following these principles: 

* Write each criterion as a clear, observable question or check, something a reviewer can answer consistently without ambiguity. 
* Choose a scoring scale for each criterion: Pass/Fail, 1 to 5, or 0 to 2. 
* Add reviewer guidance: describe what a good response looks like, list common failure modes, and note what evidence to look for in the conversation transcript. 
* Assign each criterion to its relevant scorecard category. 

 

**Step 3: Publish the Scorecard** 

1. Review the scorecard structure, including categories, weights, and criteria, to confirm they reflect your quality goals. 
2. Publish the scorecard to the relevant digital worker(s). 
3. Start in UAT to validate that the scoring is working as intended before publishing to Production. 

 

**Step 4: Review Interaction-Level Results** 

After a conversation closes, scores are calculated automatically. To review results: 

1. Open any agent interaction from the monitoring or conversation view. 
2. Review the rubric panel on the right side of the screen. It shows criterion-level scores, category rollups, and the overall score. 
3. Click into any failed criterion to see what evidence in the conversation drove the lower score. 

 

**Step 5: Monitor Trends in the Dashboard** 

1. Open the Evals dashboard for the digital worker. 
2. Review overall score trends over time (daily or weekly). 
3. Use filters to slice by digital worker, workspace, environment, channel, score category, or individual rubric criterion. 
4. Pay attention to distributions, specifically the percentage of interactions below threshold, not just average scores. 
5. After releasing a new version, use the dashboard to validate whether the change improved quality. 

*<mark style="color:$info;">Tip: Track rubric versions over time. If you change a criterion's definition or scoring scale, historical scores will no longer be directly comparable. Version your rubrics to maintain a reliable audit trail.</mark>*&#x20;


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://kb.theloops.io/agenticstudio/resource-library/how-to-guides/set-up-scorecards.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.