RAI Audit Kit

RAI = Responsible AI. A Python package suite for evidence-grade audits of responsible, secure, and trustworthy AI systems.

Run fairness, data quality, robustness, compliance, image, medical imaging, LLM safety, RAG security, and agent trace checks. Export HTML, Markdown, or JSON reports and gate CI pipelines on risk thresholds.

Author: Sai Teja Erukude | License: MIT

Why this exists

AI teams often run fairness, robustness, RAG, and agent security checks separately. RAI Audit Kit brings them into one evidence and reporting workflow, so teams can review findings consistently, preserve audit artifacts, and apply the same CI gates across model types.

What it looks like

HTML audit report	Model card export
LLM and RAG audit output	Agent trace finding

Packages

Package	Purpose
rai-audit-core	Audit engine, findings, reports, history, CI gates
rai-audit-ml	Tabular ML - fairness, drift, data quality, robustness
rai-audit-dl	Image, medical imaging, and scientific AI audits
rai-audit-llm	LLM and RAG safety, faithfulness, citation, and security audits
rai-audit-agents	Agent tool-use, memory, permission, and injection audits
rai-audit-kit	Meta-package - installs core + ml, unified CLI

Install

pip install rai-audit-kit          # core + tabular ML
pip install "rai-audit-kit[all]"   # all modules (dl, llm, agents)

Quick start

rai-audit ml run --help

For repeatable audit workflows, generate and run a YAML configuration:

rai-audit init --project loan-model
rai-audit run --config audit.yaml

Configured runs write report artifacts and an evidence manifest with input, environment, source-revision, and artifact hashes.

from rai_audit.ml import ClassificationAudit

report = ClassificationAudit(
    y_true=y_true,
    y_pred=y_pred,
    sensitive_features=sensitive_df,
).run()

report.to_html("audit_report.html")

Examples

• Fairness audit walkthrough
• Batch drift monitoring
• MLflow and Airflow templates
• Captured-response LLM and RAG audit suite
• Scientific image robustness audit
• Agent trace with a webpage prompt-injection attempt

Development

pip install uv
uv sync
uv run pytest

See CONTRIBUTING.md for monorepo layout and release workflow.