STRATEGIC THINKING WEEKLY

Field Note

The Consulting Firm That Built the Wrong System

A mid-size consulting firm asked AI to help improve their proposal process. They got exactly what they asked for: a detailed 47-step workflow with templates, checklists, and automation suggestions.

Three weeks later, they'd implemented about 60% of it. Proposal creation time had actually increased. Team members were frustrated with extra documentation requirements. The system was technically complete but practically useless.

What went wrong?

The AI output passed the "sounds reasonable" test. It passed the "comprehensive coverage" test. It even passed the "expert-sounding language" test.

But it failed the only tests that matter: the three judgment gates that separate actionable frameworks from impressive-sounding advice.

The firm had asked "How can we improve our proposal process?" That's not a decision. It's an invitation for AI to generate infinite suggestions with no way to evaluate which ones actually matter.

After the failed implementation, we reframed the problem: "Should we prioritize proposal speed or win rate, given that we lose 70% of proposals we submit?"

That reframe changed everything. Suddenly the question wasn't "how do we document more thoroughly?" It was "why are we submitting proposals we can't win?"

The real solution had nothing to do with workflow optimization. They needed a qualification framework to stop wasting time on proposals they were going to lose anyway.

---

The Deep Dive

Why AI Outputs Fail the Framework Test, And How to Catch It Early

In Issue #1, we introduced the 3-tests framework. Here's the deeper logic behind why these tests work as judgment gates, and how each failure mode plays out in practice.

Test 1: Is the problem defined as a decision?

AI systems are trained to be helpful. When you give them an open-ended problem, they generate comprehensive responses that address every possible angle. This feels thorough but creates a specific failure pattern: the output can't be wrong because it never committed to anything.

"Improve communication" can mean anything. So AI gives you tips for emails, meetings, documentation, feedback loops, team structures, and communication tools. Technically helpful. Practically overwhelming. You end up with a buffet when you needed a prescription.

Decisions force commitment. "Should we move to async communication for project updates?" has a yes or no answer. That commitment creates evaluation criteria. You can measure whether async updates actually worked better than meetings. The framework can succeed or fail in observable ways. If your AI output doesn't answer a specific decision question, it will feel useful but won't be implementable. This is the first failure mode in any serious AI framework evaluation.

Test 2: Is a boundary named explicitly?

Boundaries create constraints that make frameworks testable. Without boundaries, AI generates ideal-state solutions that assume unlimited time, budget, expertise, and organizational cooperation.

"Build a world-class customer success program" sounds great until you realize you have two people and six weeks. The boundary "implementable by our existing team in Q1" eliminates 90% of AI suggestions immediately, which is exactly what you need. If your AI output doesn't acknowledge what can't be done, it's not a framework. It's a wish list.

Test 3: Is the primary metric's penalty clear?

This is the test most people skip, and it's the most important one. Without a clear penalty for failure, there's no way to prioritize tradeoffs. "Increase customer satisfaction" gives no guidance on how to balance satisfaction against cost, speed, or other priorities.

But "Reduce churn rate from 8% to 5% (penalty: each percentage point costs $240K annually)" creates real decision criteria. Now you can evaluate whether a satisfaction improvement is worth $50K in implementation cost. The math becomes possible. AI outputs almost never include penalty calculations because they're not operating under real constraints. Your job is to add the penalty before you evaluate whether the output is worth implementing. Test your pattern recognition with the AI Writing Detective, an interactive tool that trains you to spot generic AI patterns.

Case Slice Teardown

The 15-Minute Test That Saved 3 Weeks