AI Security Evaluation & Launch Readiness Project Manager Simulation — Wealthsimple

The scenario

Wealthsimple manages over $70 billion for roughly 3 million Canadians from its King Street office in Toronto. The company already runs an internal LLM Gateway — a governed proxy that gives staff access to frontier models with PII redaction, model routing, and full prompt logging. Every internal tool built on it has one thing in common: no client ever sees the output. Wealthsimple Guide breaks that pattern. Guide is a conversational assistant inside the client app that answers money questions in plain language — what an RRSP contribution room number means, why a transfer is still pending, how a TFSA over-contribution penalty is calculated. It has been in build for seven months and is functionally complete. What it does not have is permission to launch. Client-facing AI at a registered firm sits inside a boundary drawn by CIRO and the provincial securities regulators: a firm may give information, and it may not give advice unless it is registered to do so and has met a suitability obligation to that specific client. Guide does not know which side of that line it is standing on. Neither, yet, does anyone else. The AI Risk Council chartered this evaluation program to answer that question with evidence rather than opinion: a benchmark suite that scores Guide's answers at scale, and an adversarial red-team campaign run by Haldane Labs to find the inputs that break it. Compliance will not sign a launch without both. Marketing has already circulated a launch date to press under embargo. Your job is to produce the evidence that decides whether Guide ships — and to be the person holding the gate when the evidence and the date disagree.

What you'll do as the project manager

• →Ship a versioned evaluation benchmark suite of at least 1,200 scored prompts covering the seven Guide answer domains, with an agreed pass threshold per domain signed off by Compliance
• →Complete two adversarial red-team waves with Haldane Labs and triage every finding to a severity, an owner, and a disposition — fixed, accepted, or blocking
• →Drive the personalized-advice failure rate below the 0.5% ceiling the AI Risk Council set as the launch condition, measured on the frozen benchmark
• →Produce a launch readiness evidence pack the Chief Compliance Officer will sign, covering benchmark results, residual risk, guardrail coverage, and the monitoring commitment
• →Deliver a go/no-go recommendation to the AI Risk Council by Day 24 with the residual risk stated in writing

Project management skills you'll build

The challenges you'll navigate

• •The benchmark suite inherited from the build team was written by the people who built Guide, and scores its own product at 94% — a self-graded exam nobody outside engineering has audited
• •No one has written down where information ends and advice begins. Compliance, Legal, and Engineering each hold a different working definition, and the benchmark encodes Engineering's
• •The red-team engagement was scoped and signed before the benchmark existed, so Haldane Labs' two waves may test the wrong surface
• •Marketing has circulated a launch date to press under embargo, creating an external commitment the gate decision does not control
• •Remediation capacity is 2 FTE against a finding pipeline of unknown size, and every fix competes with the client-app feature backlog

Technology & stakeholders

You'll manage 7 stakeholders, including Tobias Wrenn (Chief Product Officer), Ingrid Bettencourt (Director, Technology Program Delivery), Yusuf Adeyemi (VP, Compliance & Chief Compliance Officer), and more.

What you'll walk away with

A verified, shareable record of a completed enterprise project — plus the PMO deliverables you produced along the way (charter, project plan, SteerCo deck, closure document). It's real, demonstrable project management experience you can put on your resume and speak to in interviews.

Frequently asked questions

Related simulations