The Weekly AI Audit Playbook

AI estates change constantly, and almost never through a change-management process. An engineer tweaks a prompt to fix an edge case. Someone swaps a model to save money. A contractor's credential stays live after the contract ends. Each change is small and reasonable. Compounded over a quarter with no review, they add up to a system nobody fully understands.

The antidote is boring and effective: a weekly audit. Thirty minutes, same time every week, one owner, one checklist. Here is the playbook we recommend — the same five scans Obsivara runs automatically every Monday.

Scan one: what changed?

Start with the change log. Every workflow modification since last week: prompts edited, models swapped, nodes added or removed, schedules altered, connections rewired. For each change, two questions: was it intentional, and did anything downstream degrade after it shipped? Most AI incidents trace back to a change nobody flagged. A weekly diff review turns 'we have no idea when this broke' into 'it broke Tuesday at 3pm when the prompt changed' — which shrinks root-cause time from days to minutes.

Scan two: prompts and models

Review the model inventory: which models are in use, where, and at what cost tier. Look for silent upgrades and downgrades — a workflow moved to a cheaper model whose output quality was never re-validated, or an expensive frontier model doing classification a small model handles fine. Then sample prompts on your critical workflows: are they accumulating patches and contradictory instructions? Prompt rot is real, and it degrades output long before anyone connects the symptoms to the cause.

Scan three: cost and tokens

Pull cost per workflow, week over week. You are looking for two patterns: step changes (usually a config change or a new consumer) and steady creep (usually growing context, growing retries, or growing input volume). Flag any workflow whose cost per execution rose more than 20% with no corresponding change in what it does. Tokens are the leading indicator here — token-per-execution drift shows up before the invoice does.

Scan four: credentials and access

This is the scan most teams skip and regret. List every credential your workflows use: API keys, OAuth tokens, service accounts. For each: is it still needed, who owns it, when did it last rotate, and how many workflows share it? Shared, stale, over-privileged credentials are the single biggest security liability in automation estates — one leaked key with access to twelve workflows is twelve incidents waiting to happen. Retire what is unused, rotate what is old, split what is shared.

Scan five: failures and near-misses

Finally, review every failure and retry cluster from the week — including the ones that 'recovered on their own.' Auto-recovered failures are near-misses, and near-misses are how systems tell you where the next real incident will be. A workflow that needed three retry storms this week is not fine; it is early.

Write down what you found, assign owners, and close last week's items. That is the whole playbook. Teams that run it consistently report the same thing: after a month, the audit gets boring — because the surprises stopped. If thirty minutes is still too much, this is exactly the audit Obsivara compiles and prioritizes for you every Monday morning.