Automate Deployment Gates with Critiqor Policy Checks

Automate release gates with critiqor check — define trust score and hallucination risk thresholds in a policy file and block deploys that fall short.

Policy checks let you enforce reliability standards programmatically — before code ships. Instead of manually reviewing dashboards before every release, you define thresholds once and let critiqor check tell you whether your agent meets them. Use policy checks as a deployment gate in your CI pipeline, a pre-release verification step, or a local sanity check after a session.

What is a policy check?

critiqor check reads your saved evaluation history and verifies that recent runs meet the reliability thresholds you’ve configured. If every checked run meets the thresholds, the command exits successfully. If any run falls short, the check fails and reports which thresholds were not met.

This makes it straightforward to integrate Critiqor into automated workflows: block a deployment if the trust score is too low, fail a CI job if hallucination risk is above your limit, or enforce consistent standards across a team.

Basic usage

critiqor check

By default, critiqor check reads evaluations from critiqor_evaluations.jsonl in your current working directory. Run this after finalizing one or more sessions to check them against your configured thresholds. To read from a different file, pass the --evaluations flag:

critiqor check --evaluations path/to/critiqor_evaluations.jsonl

Setting thresholds

Pass threshold flags to define the reliability bar your runs must meet:

critiqor check --minimum-trust-score 75 --maximum-hallucination-risk 20

Flag	Description
--minimum-trust-score	The minimum trust score (0–100) a run must achieve. Any run below this score causes the check to fail.
--maximum-hallucination-risk	The maximum hallucination risk score (0–100) allowed. Any run above this threshold causes the check to fail.

You can use either flag independently or combine them. All specified thresholds must be met for the check to pass.

Filtering by agent

If your evaluation file contains runs from multiple agents, use --agent-id to scope the check to a specific agent:

critiqor check --agent-id my_agent --minimum-trust-score 80

Only runs matching the specified agent ID are evaluated. Runs from other agents in the same file are ignored.

Using a policy file

Instead of passing flags on every invocation, define your thresholds in a policy file and reference it with --policy. This is the recommended approach for teams and CI environments — thresholds live in version control alongside your agent configuration.

critiqor check --policy policy.json

Create a policy.json file in your project with your thresholds:

{
  "minimum_trust_score": 75,
  "maximum_hallucination_risk": 25
}

Critiqor accepts both JSON and YAML policy files. Any threshold you can pass as a flag can also be defined in the policy file. Flags passed at the command line take precedence over values in the policy file if both are present.

Exit codes

critiqor check uses standard exit codes, making it straightforward to integrate into any CI or scripting environment:

Exit code	Meaning
0	All checks passed. Every evaluated run meets the configured thresholds.
Non-zero	One or more checks failed. At least one run did not meet a threshold.

When a check fails, the command prints a summary of which runs failed and which thresholds were not met. Use this output to identify which session to investigate in the dashboard.

Example CI integration (GitHub Actions):

- name: Run Critiqor policy check
  run: critiqor check --policy policy.json

If the step exits non-zero, the workflow fails and the deployment is blocked.

Policy checks evaluate your saved evaluation history. Make sure you have finalized all recent sessions with critiqor finalize before running checks — any session that hasn’t been finalized won’t appear in the evaluation file and won’t be included in the check.