Hybrid Project Manager Simulation — Bank of America

The scenario

Bank of America's Erica is the most-used virtual financial assistant in U.S. banking — more than 3 billion client interactions since its 2018 launch, roughly 50 million users, and over 58 million interactions a month across the mobile app, Merrill, Benefits OnLine, and CashPro. For most of that history the Erica team has measured the assistant the way contact centers measure themselves: containment rate (did the client avoid calling a human?) and a thumbs-up/thumbs-down satisfaction tap. Neither metric answers the question the enterprise AI Risk committee started asking in 2025 — was the answer Erica gave actually correct, appropriate, and compliant? The Conversational AI Quality Program is the answer to that gap. You are standing up Bank of America's first standing conversation-quality audit: every week, a representative sample of 2,000 anonymized Erica conversations is pulled, scored by trained human reviewers against a structured rubric, calibrated for consistency, and fed back to the NLU engineering team as a prioritized signal. The dimensions — factual accuracy, escalation appropriateness, regulatory-disclosure handling, and customer-experience quality — do not exist yet as a scorecard. Neither does the reviewer pool, the sampling frame, or the agreed definition of what 'good' means. This is a build from a blank sheet. The hard part is not the tooling. The interaction-analytics platform is licensed, the reviewer vendor is contracted, the data is accessible. The hard part is that four teams each believe they already know what quality is. The Erica product team measures engagement. The NLU engineering team measures intent-recognition accuracy and model confidence and has questioned, in writing, whether a human audit tells them anything their dashboards don't. CX research measures sentiment and effort. Compliance measures disclosure exposure. There is no prior quality function to inherit and no agreed metric to start from. Your job is to build the measurement before anyone can improve the thing being measured — and to get four teams who each own a piece of 'quality' to agree on one scorecard.

What you'll do as the project manager

• →Design and ratify a single Erica conversation-quality scorecard across four dimensions — accuracy, escalation appropriateness, compliance disclosure, and customer experience — agreed by product, NLU engineering, CX research, and compliance
• →Build a defensible weekly sampling frame that draws ~2,000 anonymized conversations representative of intent mix, channel, and client segment, with privacy and anonymization signed off by compliance
• →Train and calibrate a 12-person human-reviewer pool to an inter-rater-reliability target (Cohen's kappa ≥ 0.70) before any quality scores are reported as program output
• →Wire a closed feedback loop from scorecard findings to the NLU engineering backlog so the lowest-scoring intents become prioritized model and content work
• →Hand a documented, repeatable weekly audit operation — runbook, rubric, calibration protocol, and reporting pack — to Digital Banking operations within the $1.0M / 14-week baseline

Project management skills you'll build

The challenges you'll navigate

• •Four teams each hold a different definition of 'quality' (engagement, intent accuracy, sentiment, disclosure exposure); without a ratified scorecard the program produces numbers no one trusts
• •Devesh Behl's NLU engineering team has questioned in writing whether a human audit adds anything its automated metrics don't — the program's defining adoption risk is winning that team's buy-in, not building the tooling
• •Reviewer inter-rater reliability is unproven; if reviewers don't agree with each other, the weekly scores are noise and the first reported batch loses credibility
• •The sampling frame can quietly bias the program — over-sampling easy intents flatters Erica, over-sampling escalations indicts it; the frame must be defensible to both product and compliance
• •A live-conversation finding (Erica giving a technically-correct but harmful answer) can surface a disclosure obligation mid-program and pull compliance and legal into a program that was scoped as analytics

Technology & stakeholders

You'll manage 7 stakeholders, including Diane Calloway-Reed (SVP, Digital Banking & AI Experience), Roland Maddox (Director, Digital Channels PMO), Camila Restrepo (Senior Product Manager, Erica), and more.

What you'll walk away with

A verified, shareable record of a completed enterprise project — plus the PMO deliverables you produced along the way (charter, project plan, SteerCo deck, closure document). It's real, demonstrable project management experience you can put on your resume and speak to in interviews.

Frequently asked questions

Related simulations