Can AI Produce a Client-Ready Aggregate Analysis of Executive Assessment Data?
In the past few months, I conducted a hands-on test to explore a growing question in leadership consulting: Can AI executive assessment analysis deliver insights with the depth and accuracy needed for high-stakes decision-making? The rise of generative AI has sparked interest in its potential to process and synthesize executive feedback at scale—but I wanted to see for myself how it stacks up against the judgment of an experienced consultant.
The promise is compelling. AI can scan large volumes of text in seconds. It can categorize, summarize, and surface themes faster than any human. That kind of efficiency could transform how we deliver insight to leadership teams.
But theory doesn’t always hold up in practice—especially when subtle judgment calls are involved.
The Test: Comparing AI and Human Judgment
I used nine real executive assessments, each developed through stakeholder interviews and hands-on analysis. These were typical of what I deliver for clients—comprehensive and customized. Each report included:
- A one-page narrative executive summary
- A bullet-point list of strengths and development areas (each beginning with an action verb)
- Several pages of anonymized, polished stakeholder quotes categorized by theme
With those reports in hand, I ran two separate analyses:
- Manual (fully human) analysis: I reviewed all nine reports and synthesized aggregate themes across strengths and development areas. I also conducted a narrative-level thematic analysis—exactly what I’d do in preparation for a leadership team offsite.
- AI-generated analysis: I used generative AI to analyze the same reports, asking it to extract and summarize common strengths, development themes, and patterns across the executives.
Then, I compared the results head-to-head.
First Impressions: AI Looks the Part
The AI output was, at first glance, surprisingly good. The structure was clean. The language was professional. It offered clearly labeled categories, cohesive groupings, and summaries that looked like the kind of work we’d deliver to a client team.
If I had scanned the output without deep knowledge of the source material, I might have assumed it was a credible synthesis.
But once I compared it to my own analysis, the flaws began to surface.
Where the AI Fell Short
Despite the professional presentation, the AI executive assessment analysis contained significant gaps and inaccuracies:
- Shallow pattern recognition: It lumped together concepts that looked similar on the surface but had different meanings beneath. For example, it conflated “bias toward action” with “inflexible in ambiguity”—two very different patterns that require different coaching responses.
- False positives: Some themes showed up in only one or two individuals but were generalized across the entire group. This would have introduced misleading recommendations if used in a group setting.
- Poor contextual interpretation: The AI often failed to correctly interpret subtle differences in tone or intent. In a few cases, it reversed the meaning of a quote—misrepresenting praise as criticism.
- Lack of prioritization: While the AI could list many themes, it struggled to discern which ones were material and which were simply noise.
Ultimately, the analysis felt like something you’d expect from a capable but inexperienced junior consultant—technically sound in format, but off the mark in judgment.
What the AI Got Right
To be fair, the AI did a few things well:
- Speed and formatting: The analysis was fast. It produced a structured, presentable document in minutes—what would have taken a human hours.
- Basic grouping: It was competent at creating general categories (e.g., “collaboration,” “execution,” “communication”) and sorting quotes accordingly.
- Clean language: The output was polished and readable—something that could easily be misunderstood as being client-ready.
But none of those strengths offset the risks introduced by flawed insight.
Why This Matters: The Stakes Are High
Executive assessments aren’t casual documents. They’re used in decision-making that affects leadership pipelines, team performance, and organizational strategy.
Typically, clients use these reports to inform:
- Succession planning
- Team interventions
- Individual development plans
If a consultant misreads the data—or presents themes that aren’t actually present—the downstream effects can be serious: coaching that misses the mark, talent decisions that backfire, or team dynamics that worsen rather than improve.
This is why AI executive assessment analysis, despite its speed and surface-level polish, can’t yet be trusted without rigorous human review.
Implications for Consultants
Here are three clear takeaways for leadership consultants considering AI in their workflow:
- AI can’t replace human judgment. The heart of executive assessment is interpretation. Pattern recognition, prioritization, nuance—these are the consultant’s craft. AI doesn’t have it (yet).
- Use AI carefully—if at all. If you choose to use AI for quote sorting or structural drafting, build in thorough human oversight. Treat it like a raw material generator, not a trusted analyst.
- Client readiness is about substance, not style. Just because something looks good doesn’t mean it’s right. Our clients rely on us to bring insight, not just information.
What’s Next: AI as a Tool, Not a Strategist
AI is evolving quickly. Future versions will likely do a better job at understanding tone, drawing inferences, and integrating cross-report themes.
But today, it’s clear: AI isn’t ready to replace the consultant in executive assessment work. And it may never fully replace the level of trust, judgment, and discretion required in high-stakes leadership conversations.
Instead, the opportunity is to augment our work—using AI to handle the mechanical parts of data handling, quote collation, and first-draft synthesis—while keeping the human mind in charge of what matters most: judgment, clarity, and strategic relevance.
I’d love to discuss further with you.
No comment yet, add your voice below!