Summary

Add a compliance dashboard to the Studio interface that provides a single roll-up view of evaluation status, compliance posture, and usage across all Rovo agents in the organisation.

Problem Statement

There is currently no way to see the compliance status of all agents at a glance. Administrators must navigate to each agent individually to check evaluation results, determine whether evaluations are current, or identify regressions. For organisations operating multiple agents, this does not scale.

When reporting to governance stakeholders (e.g. an AI governance panel), administrators must manually compile evaluation results across agents into external reports. There is no built-in mechanism to generate a compliance summary or export evaluation data for audit purposes.

Additionally, there is no visibility into agent usage at the portfolio level. Administrators cannot see which agents are heavily used, which are dormant, or how usage correlates with evaluation compliance.

Proposed Solution

Compliance tab in Studio

Add a "Compliance" tab to the Studio landing page (the "All agents" view), alongside the existing agent list. This provides a portfolio-level view before drilling into any specific agent.

Summary metrics

Four summary cards at the top of the dashboard:

• Total agents - Count of all agents in the organisation

• Fully compliant - Agents with a passing evaluation within the site policy's minimum frequency

• Evaluation overdue - Agents that have not been evaluated within the site policy's minimum frequency

• Failed evaluation - Agents whose most recent evaluation did not meet pass criteria

Agent compliance table

A table listing all agents with the following columns:

• Agent - Name, icon, and pattern classification (e.g. "Pattern 1 - Knowledge Q&A")

• Status - Compliant (green), Overdue (yellow), or Failed (red)

• Last evaluation - Date and relative time since last evaluation. Overdue agents are highlighted.

• AUP score - Pass rate on the AUP compliance dataset with visual score bar

• Boundary score - Pass rate on the boundary dataset with visual score bar

• Accuracy score - Pass rate on the accuracy dataset with visual score bar

• Trend - 8-week sparkline showing evaluation score trajectory (improving, stable, or declining)

• Usage (30d) - Chat count and unique user count over the last 30 days

The table supports:

• Filtering by status (all, compliant, overdue, failed) and by pattern

• Sorting by any column

• Export to CSV for external reporting (e.g. AIROC governance submissions)

Row highlighting

• Overdue agents are highlighted with a yellow background

• Failed agents are highlighted with a red background

• Compliant agents have no background highlight

Design Reference

See attached mockup (mockup-compliance-dashboard.html) showing:

• Studio landing page with the "Compliance" tab active

• Summary metric cards (8 total agents, 5 compliant, 2 overdue, 1 failed)

• Agent compliance table with 8 example agents across Patterns 1, 2, and 3

• Visual score bars, 8-week trend sparklines, and usage metrics

• Filter controls for status and pattern

• Export CSV button

Use Cases

Governance reporting: A platform team prepares a monthly compliance report for the AI governance panel. Rather than compiling results manually, they open the compliance dashboard, confirm all agents are compliant, and export the table to CSV as an auditable artefact.

Proactive issue detection: An administrator opens the compliance dashboard and sees that two agents are overdue for evaluation. They trigger evaluations immediately rather than waiting for the scheduled run, catching a regression on one agent before users are affected.

Usage-informed governance: The dashboard shows an agent with 23 chats and 11 unique users in the last 30 days alongside another with 2,106 chats and 89 unique users. The low-usage agent may be a candidate for decommission. The high-usage agent warrants more frequent evaluation.

Trend monitoring: An agent's 8-week trend sparkline shows a declining trajectory despite still passing evaluations. This early warning signal prompts the platform team to investigate before the agent drops below the pass threshold.

Considerations

• Performance. The dashboard should load efficiently even for organisations with a large number of agents. Consider paginating the table or lazy-loading trend data.

• Permissions. The compliance tab should be visible to organisation administrators and agent administrators. Agent administrators should only see agents they have administrative access to.

• Real-time vs cached. Evaluation scores and usage metrics can be cached and refreshed periodically (e.g. hourly) rather than queried in real time. The "Last evaluation" date should be accurate.

• Export format. CSV export should include all columns visible in the table, plus the agent's knowledge source list and dataset configuration for audit completeness.