Skip to content

safety_evaluation.md

Latest commit

 

History

History
67 lines (51 loc) · 3.58 KB

safety_evaluation.md

File metadata and controls

67 lines (51 loc) · 3.58 KB
Raw

Evaluating RAG answer safety

When deploying a RAG app to production, you should evaluate the safety of the answers generated by the RAG flow. This is important to ensure that the answers are appropriate and do not contain any harmful or sensitive content. This project includes scripts that use Azure AI services to simulate an adversarial user and evaluate the safety of the answers generated in response to those adversarial queries.

Provision a Microsoft Foundry project

The adversarial simulator and safety evaluators run against a Microsoft Foundry project. This project deploys a Foundry account and project by default (as long as you're not bringing your own Azure OpenAI resource), so no extra provisioning step is required. The project endpoint is stored in the FOUNDRY_PROJECT_ENDPOINT environment variable, which safety_evaluation.py reads automatically.

⚠️ Adversarial simulation and safety evaluation are only available in a limited set of Azure regions: East US 2, France Central, Sweden Central, Switzerland West, and North Central US. Your Foundry account is deployed in the region you set for AZURE_OPENAI_LOCATION. If that region isn't in the supported list, safety_evaluation.py will exit with an error telling you to redeploy in a supported region. See the region support documentation for the latest list.

Simulate and evaluate adversarial users

Run the following command to simulate adversarial queries and evaluate the safety of the answers generated in response to those queries:

python evals/safety_evaluation.py --target_url <TARGET_URL> --max_simulations <MAX_RESULTS>
  • --target_url: The target URL for the callback. Default is http://localhost:50505/chat.
  • --max_simulations: The maximum number of simulated user queries. Default is 200. The higher the number, the longer the evaluation will take. The default of 200 simulations will take about 25 minutes to run, which includes both the time to generate the simulated data and the time to evaluate it.

Review the safety evaluation results

That command will generate a report in safety_results.json. Results will look like this:

{
  "hate_unfairness": {
    "low_count": 20,
    "score_total": 8,
    "mean_score": 0.4,
    "low_rate": 1.0
  },
  "sexual": {
    "low_count": 20,
    "score_total": 9,
    "mean_score": 0.45,
    "low_rate": 1.0
  },
  "violence": {
    "low_count": 20,
    "score_total": 9,
    "mean_score": 0.45,
    "low_rate": 1.0
  },
  "self_harm": {
    "low_count": 20,
    "score_total": 10,
    "mean_score": 0.5,
    "low_rate": 1.0
  }
}

The ideal score is low_rate of 1.0 and mean_score of 0.0. The low_rate indicates the fraction of answers that were reported as "Low" or "Very low" by an evaluator. The mean_score is the average score of all the answers, where 0 is a very safe answer and 7 is a very unsafe answer.

Resources

To learn more about the Azure AI services used in this project, look through the script and reference the following documentation: