DeepEval Evaluation Workbench

Evaluate assistant responses on Ubuntu 24.04 with DeepEval 4.2.4, a local CLI, and saved JSON reports. The supplied example tests a synthetic returns policy: 30 days with a receipt. A correct response passes and a response claiming 90 days without a receipt fails. The local example uses DeepEval's JSON correctness metric with exact policy constraints. It does not call or download a model.

This is paid repackaged open-source software supplied by Software Sushi. DeepEval core uses Apache License 2.0. Dependencies retain their own licenses. Confident AI's hosted dashboard, commercial platform and subscriptions are not included. No Confident AI account is required for the supplied workflows. No model weights or provider credits are included. This image is not affiliated with or endorsed by Confident AI.

Connect and run

Use a t3.medium, x86_64, with at least 24 GiB of root storage. The supplied examples need no GPU. Provider-based evaluations also need outbound HTTPS and access to the provider you choose. EC2 storage and external model usage incur separate charges.

Connect through AWS Systems Manager Session Manager, using an instance role with Session Manager permissions and outbound access to the regional SSM endpoints. Alternatively use SSH with your own launch key and tightly restricted inbound access. No application web UI, HTTP API listener, or application port is opened. Access is controlled by the Ubuntu operating-system account.

Run:

sudo -iu ubuntu bash /opt/sushi/docs/demo.sh

The command saves a new report under /home/ubuntu/deepeval-results, prints its path and SHA-256, independently reads the saved scores, and confirms that the incorrect answer failed. Each run keeps its own report rather than overwriting an earlier result. Reports and provider settings are readable by ubuntu only.

To evaluate and inspect separately:

sudo -iu ubuntu
sushi-deepeval evaluate
ls -lt ~/deepeval-results

Pass a report's complete path to sushi-deepeval read. This reader checks the supplied good/bad example, including its actual answer values, thresholds and saved success flags. It is specific to this example, not a general report viewer.

Add a model judge

The application being tested and the judge are separate. The supplied assistant answers are fixed synthetic fixtures; the optional judge evaluates them with DeepEval GEval against the expected facts. This tests judging, not live assistant generation. It sends the question, actual answer and reference answer to the configured provider. Use synthetic data until you have reviewed provider privacy and retention terms. Model scores can vary and need human review.

As ubuntu, run:

sushi-deepeval configure

Enter an OpenAI-compatible base URL including its /v1 path, a model ID supported by that endpoint, and your provider key at the hidden prompt. For example, the OpenAI endpoint is https://api.openai.com/v1; choose a model available to your account that supports JSON-object chat responses. HTTPS is required for remote providers. A localhost OpenAI-compatible server may use HTTP. AWS Bedrock's native API is not called by this example; it requires a separately configured compatible gateway with your own permissions and model access.

The command creates ~/.config/deepeval/judge.json with mode 0600. It refuses to overwrite an existing file. To change providers, edit that file locally with an editor. Its exact keys are base_url, model, and api_key. Do not paste keys into shell commands, logs, source files, screenshots or support tickets.

Then run:

bash /opt/sushi/docs/demo.sh --judge

This adds a GEval factual-correctness score and reason for each answer. The example succeeds only if the correct answer passes and the incorrect answer fails both metrics. This makes billable provider requests. No platform signup is needed. Telemetry and automatic dotenv loading are disabled in the supplied commands; reports stay on the instance. Explicit judge requests go to your configured endpoint.

Use your own evaluations

The unmodified upstream deepeval CLI is on PATH. Run deepeval --help for its commands. Use /opt/deepeval/venv/bin/python in your own workspace to import deepeval, create LLMTestCase instances and select metrics. The supplied /opt/deepeval/evaluate.py shows the two example criteria. Keep the installed runtime intact and use a separate environment when adding dependencies. DeepEval features beyond these examples can have additional provider, optional package or account requirements; hosted Confident AI features need their own service agreement.

Persistence and recovery

Reports live in ~/deepeval-results; judge settings live in ~/.config/deepeval/judge.json. These survive a reboot on the same root volume. Back them up before replacing or terminating the instance; EC2 root-volume termination settings determine whether that volume survives termination.

Initialization runs once at boot as sushi-deepeval-firstboot.service, using the Ubuntu account and restrictive file modes. Check it with:

sudo systemctl status sushi-deepeval-firstboot.service
sudo journalctl -u sushi-deepeval-firstboot.service --no-pager

If a judge request fails, verify the base URL, provider entitlement, model ID, JSON response support, key, outbound access and file ownership. No accepted report is created for a failed request. Fix the configuration and rerun; a failed request does not disable later runs. chmod 600 ~/.config/deepeval/judge.json repairs overly broad permissions. Preserve original reports during diagnosis. Do not delete the initialized state in /var/lib/deepeval. Restarting the initialization service preserves the existing identity, settings and reports.

Original wheels, corresponding source distributions and license inventories are retained under /opt/deepeval. Read LICENSE-NOTICE.md and licenses/DeepEval-LICENSE.md there for attribution and source locations.