Evaluating and Approving a New AI Tool

Evaluating and Approving a New AI Tool

No AI tool enters the workflow without scrutiny — and every approval comes with settings.

4 min read

The fastest way to lose control of your AI posture is to let tools enter the workflow one convenient signup at a time. This play is a standard intake that puts every new AI tool through the same scrutiny, so the result is a deliberate, documented approved-tool list rather than a sprawl of whatever individuals happened to try.

When to use this play#

Run it whenever someone wants to bring a new AI tool into the workflow, before they start relying on it. It is intentionally the same process for a small utility and a major platform, because the lightweight ones are exactly where unreviewed data-sharing tends to creep in. Re-run it periodically for tools already approved, since their terms and behavior change.

The evaluation dimensions#

Assess every new AI tool across these dimensions. None is optional; a tool that aces five and fails the sixth still fails.

  • Security review — how it handles data, what it encrypts, and what access controls it enforces.
  • Privacy impact — what data is shared with it, how long the provider retains it, and how you can delete it.
  • Integration testing — whether it is compatible with the tools you already use.
  • Impact analysis — its effect on delivery and on client privacy.
  • Cost-benefit and ROI — whether the value justifies the cost and effort.
  • Legal review — what the terms of service actually commit you to.

How to run it#

1. Notify operations at the start. The moment you begin evaluating a tool, tell operations. This keeps the org's view of "what are we assessing" current and prevents parallel, duplicated evaluations.

2. Run the assessment across every dimension. Work through security, privacy, integration, impact, cost, and legal. Treat each as a gate, not a box to tick.

3. Document the results. Record a clear decision: keep using it or not, and if yes, exactly which privacy and security settings to apply. The settings are part of the approval; "approved" with no configuration guidance is an incomplete answer.

4. Route external tools through an approved gateway. Where possible, external tools connect through your approved gateway rather than directly, so access is centralized and observable.

5. Publish and maintain a visible approved-tool list. The list only works if people can see it. Keep it current and accessible so the answer to "can I use this" is always findable.

6. Re-assess periodically. Terms of service, data-handling practices, and pricing models all change. A one-time approval is a snapshot, not a permanent verdict.

Extra criteria worth weighing#

Beyond the core dimensions, these often separate two tools that both pass the gates:

  • Ease of use — whether the team will actually adopt it.
  • Ease of integration — how cleanly it fits your existing stack.
  • Pricing model — how cost scales as usage grows.
  • Lock-in risk — how hard it would be to leave.
  • Quality and consistency — whether its output holds up over time.
  • Data privacy and security — its posture beyond the minimum review.
  • Environmental policy — the provider's stance, where that matters to you.

Common traps#

  • Approving without settings. "Yes, use it" with no guidance on retention, sharing, or access controls hands people an unconfigured tool and hopes for the best.
  • Skipping the legal review. The terms of service are where surprising data and ownership commitments hide. Reading them after adoption is too late.
  • Treating approval as permanent. Providers change their terms quietly. An approved-tool list that is never re-assessed slowly fills with stale verdicts.
  • Letting tools bypass the gateway. Direct connections defeat the centralized control and logging the gateway exists to provide.
  • Keeping the approved list invisible. If people cannot find the list, they will just sign up for whatever is convenient and the whole intake collapses.

Signals it's working#

  • The team checks the approved-tool list before adopting anything, and the list answers the question.
  • Every approval on the list carries documented privacy and security settings.
  • Operations always knows what is currently under evaluation.
  • Periodic re-assessment catches a changed term of service before it becomes a problem.