BlogStrategy

AI Productivity Has a Hidden Cost. It’s Called the Learning Penalty.

By 8 min read

AI can build the dashboard. It can’t tell your VP the dashboard is pulling from the wrong field. That skill is disappearing.

Your VP just walked into a strategy meeting with an AI-generated attribution dashboard. Clean numbers. Polished visualization. Confident recommendations about where to shift budget next quarter.

You’re sitting across the table, and something doesn’t feel right, but you can’t articulate it yet. Then it hits you: you don’t know which activity field the data is pulling from. And you’re fairly certain the person presenting it doesn’t either.

Welcome to the new normal in marketing operations. The dashboards have never looked better. The understanding behind them has never been thinner.

The Field That Looked Right and Wasn’t

At a previous company, our Salesforce org had two fields that sounded interchangeable: “Most Recent Activity” and “Last Activity.”

They weren’t the same thing.

“Most Recent Activity” was a custom field, updated by a specific workflow trigger that fired when certain criteria were met. “Last Activity” was the platform default, tracking something slightly different with different logic. Both were technically valid. Both would produce numbers that looked correct in a report. Neither was labeled in a way that told you which one mattered for the question you were actually trying to answer.

You had to know. And the only way you knew was because you’d been there when someone built the wrong dashboard on the wrong field, watched the numbers not reconcile during a QBR, and spent an afternoon tracing the discrepancy back to a field name that was off by one word.

That’s institutional knowledge. It doesn’t live in documentation — it lives in scar tissue.

Here’s the problem: AI doesn’t have scar tissue. And increasingly, neither do the people sitting around the table.

The Learning Penalty Nobody’s Measuring

Everyone is measuring AI’s productivity gains. Stanford’s 2026 AI Index Report catalogues them meticulously: software developers using GitHub Copilot completed 26% more pull requests. Marketing teams using AI for ad creation saw a 50% increase in output per worker. Customer support agents resolved 14–15% more issues per hour.

Those numbers are real. They’re also incomplete.

Buried in the same report is a finding that should concern anyone who manages a team: software engineers who relied heavily on AI for learning showed zero measurable speed improvement and faced what researchers call “learning penalties.” They used AI constantly. They produced output. They did not get better.

And in what might be the study’s most uncomfortable finding, the METR research group found that experienced developers actually became 19% slower when using AI assistance — while simultaneously believing the tools had made them faster. The gap between perceived and actual performance was significant enough that developers are now reluctant to work without AI at all, making the study difficult to replicate.

Let that sink in. The tool made them less effective. They were convinced it made them more effective. And now they can’t imagine working without it.

The Pipeline Is Thinning

If the learning penalty were happening in a vacuum, you could treat it as a training problem. It’s not happening in a vacuum.

Stanford’s labor data shows that employment for software developers ages 22–25 fell nearly 20% from its 2022 peak, even as headcount for older developers continued to grow. In the most AI-exposed occupations, early-career employment dropped 16% relative to the least-exposed roles, after controlling for firm-level effects.

The same pattern is emerging in marketing. The junior analyst role — the one where people used to learn how data actually works by stitching reports together, fixing tracking errors, and rebuilding taxonomies — is precisely the role most likely to be automated or eliminated.

As Angelina Eng wrote in MarTech recently, the issue shows up in moments like a Q1 review: senior leaders catch measurement problems that junior analysts didn’t know existed, because they’d never had to build the reports themselves. They’re “smart and fluent in the tools, but this conversation is different. It’s not about what the system surfaced. It’s about what the system missed.”

That’s the double cut. AI eliminates the junior work that built judgment, and it prevents the remaining juniors from developing judgment through the work that’s left.

The Squeeze from Both Directions

If you’re in marketing operations, you’re now getting compressed from two sides.

Below you, junior team members are learning to read dashboards but not to question them. They can navigate the tools. They can produce reports. But they’ve never had to reconcile a data discrepancy by hand, never traced a tracking failure back to its source, never sat through the meeting where the numbers didn’t match and had to figure out why. So they don’t know what “wrong” looks like when it’s wearing a clean visualization.

Above you, something arguably more alarming is happening. Leadership is building their own AI-generated dashboards and forming strategic conclusions without understanding the data underneath. I keep saying this in my roundtables and I’ll say it here: be careful — AI is so confidently wrong. A polished dashboard built on “Most Recent Activity” instead of “Last Activity” doesn’t look wrong. It looks decisive. And the decisions that flow from it are invisible errors until they compound into a budget misallocation or a missed quarter.

You — the marketing ops professional with the institutional knowledge, the scar tissue, the hard-won understanding of what the data actually means — are the last layer of organizational intelligence between AI-generated confidence and bad decisions.

And nobody is building more of you.

The Sequel to “Don’t Hire a Person First”

A few weeks ago, I argued that your first marketing hire shouldn’t be a person — that workflows should come before headcount. I stand by that.

But this is the cost you need to be tracking when you make that trade.

If you automate the execution without preserving the understanding of the work, you’re optimizing for this quarter’s output while eroding next year’s organizational intelligence. The workflow handles the report. Nobody learns how reporting works. The AI generates the analysis. Nobody develops the judgment to know when the analysis is subtly, confidently, dangerously wrong.

Workflows before headcount is still the right sequence. But “before” doesn’t mean “instead of.” At some point, you need people who understand the work beneath the automation — and the window to develop those people is narrower than you think.

What to Do About It (Without Abandoning the Gains)

This isn’t a “stop using AI” argument. That ship sailed, and it shouldn’t come back. The productivity gains are real and your team needs them. The question is how to capture those gains without quietly hollowing out the knowledge that makes them safe.

Assign junior team members to data remediation, not just reporting. When tracking breaks, when field mappings need to be rebuilt, when a taxonomy doesn’t hold — that’s not a cleanup task. That’s a development opportunity. The person who traces a data discrepancy back to its source carries that experience forward in a way that a dashboard reviewer never will. The messy work is the training.

Narrate your reasoning out loud. When you present analysis to your team, don’t just share conclusions. Walk through what felt off, what you dug into, what assumptions you chose to challenge. That running commentary is exactly what developing practitioners need to hear — and it’s the thing AI cannot model, because AI never had the experience that generated the instinct.

Build “beneath the dashboard” checkpoints into your workflow. Before AI-generated outputs go to leadership, require a structured review: where were estimates used? Where do gaps exist? What assumptions were made? This keeps critical thinking in the process rather than assuming the AI handled it upstream. It also gives junior team members a structured way to practice the skill of questioning output.

Evaluate your team on whether they can spot when the system is wrong. If your performance frameworks only measure how well someone works within the system, you’ll never know whether they can recognize when the system is broken — and you won’t build that capability in them, either. Ask candidates and current team members: “Tell me about a time the data looked right but wasn’t.” If they can’t answer, that’s your gap.

The Skill That’s Becoming Rare

Stanford’s data tells a clear story: AI is accelerating output at every level. It’s also thinning the population of people who understand what the output actually means.

The 26% productivity gains and the zero-percent skill improvement aren’t separate findings. They’re the same trade-off, viewed from two angles. You get speed now. You pay for it later in organizational judgment — unless you’re deliberate about preserving what AI can’t teach.

AI can build the dashboard. It can’t tell your VP that the dashboard is pulling from the wrong field. That skill — the ability to look at a clean, confident, AI-generated analysis and say “this doesn’t smell right” — is becoming rarer. And it only develops one way: by doing the messy work first.

The question isn’t whether your team uses AI. It’s whether you’re building the next generation of people who know when not to trust it.

Sources and Further Reading

Related Reading

Wondering whether your team is building AI skills or just AI dependence? That’s exactly the kind of question I help marketing teams answer. At Pallas Advisory, I work with marketing operations leaders to build systems that capture AI’s productivity gains without losing the institutional knowledge that makes those gains safe. If your team is producing more output but you’re not sure anyone understands what’s underneath it, let’s talk.

More in Strategy

View all →
Join the "AI Strategist"

Weekly tips on using AI without the hype.

No spam, unsubscribe anytime. Join hundreds of other leaders getting practical advice.

Subscribe for Free