THE EFFECTIVE PROBLEMSOLVER #112
The Paradox of Progress
I used to believe that more data always meant better decisions.
I don’t anymore.
I’ve helped healthcare employers, nonprofit training programs, foundations, and government agencies collect mountains of new data.
In theory, that should have led to smarter, fairer, more transparent choices.
Instead, in one case, it made things worse.
The Dashboard That Broke the System
A few years ago, I lobbied hard for a new performance dashboard—a tool to help legislators make more informed funding decisions.
At the time, those decisions were mostly driven by stories and marketing materials.
The few numbers that existed focused on how many people were “served,” not whether anything actually changed for participants.
I wanted to fix that.
So we built a dashboard that would finally show outcomes—pre-to-post data that revealed what changed for people after the intervention.
It included disaggregations, program comparisons, and every possible way to slice the data.
When it launched, it looked impressive—sleek pages, endless charts, a portal to perfect transparency.
But within weeks, it became clear: no one knew where to start.
The people who built it were proud. The people who needed it were paralyzed.
The data was clean. The insight was muddy.
Legislators were already drowning in information. They didn’t have time to navigate multiple pages and dozens of metrics.
And the burden of collecting and maintaining all that data fell on staff who quickly realized no one was using it.
Eventually, the data went stale, the dashboard fell into disrepair, and the whole system quietly died.
When the next round of funding decisions came up, legislators ignored outcomes altogether.
They turned back to anecdotes and gut instinct—just as before.
Better data hadn’t created better learning.
It had created a mirage.
What the Data Missed
That experience taught me something humbling:
better data only matters if the system knows how to learn from it.
It’s not enough to collect more data, or even better data.
The key question is whether the people using it have a learning loop in place—a clear process for turning information into insight, and insight into action.
A Simple Test: The Learning Loop Check
Ask these three questions before investing in any new data tool or dashboard:
- Does the data help you understand why something is happening—not just what’s happening?
If not, it’s descriptive noise, not learning. - Is there a mechanism for adjusting based on what’s learned?
Without feedback, data just piles up. - Are those adjustments visible and communicated to stakeholders?
Otherwise, people lose trust that data leads anywhere.
When those three conditions aren’t present, more data will only make things worse.
It looks like progress, but it’s actually confusion with a user interface.
Measure Less. Learn More.
We like to say, “what gets measured gets managed.”
But in complex systems, what gets measured often gets ignored.
Data doesn’t drive learning—reflection does.
Without a feedback loop, even the best dashboards become digital wallpaper: impressive, expensive, and irrelevant.
The lesson?
Build systems that learn, not just systems that measure.
Because in complex problems, the goal isn’t more data.
It’s better sensemaking.
See you in two weeks.
-Bryan
