Why Metrics Are Always Proxy

First, there is the real thing.

Then, you add something like tests to observe the real thing. You create a smoke test to check that nothing important was broken in the update. Then, you automate the tests to free your hands for something more important.

That’s one layer of abstraction on top of the real thing. Like optics that you chose to view the world.

Now the organization demands more visibility to testing and the quality of your product. So you add metrics to observe your tests.

That’s already the second layer of abstraction on top of the real thing.
Then, the organization needs to make decisions about future resourcing, deadlines, and go-lives.

Someone looks at your metrics and adds their judgment based on their experience and what they see.

A third and fourth layer of abstraction just got added on top of the real thing.

Finally, based on the tests, topped by metrics, topped by observer biases and judgments, someone makes a decision that is supposed to improve the quality of the project.

That decision is the fifth layer of abstraction stretched away from the real thing.

Decisions determine the direction of your project.

Decisions are the root cause upstream and outcomes flow into your field of vision much later.

Decisions are simple answers to questions like: Should we fix this specific bug? Should we look closer at our security? Should we plan a stabilization sprint before moving on? Should we hire another developer? Should we push the deadline one month further? Should we ask for more budget?

Now comes the most important distinction about the abstraction sediments that your tests and metrics create. Just think about it. Where do you make better decisions — Close to the real thing or far away from it? By observing the real thing directly or through several layers of abstraction?

Metrics are always proxy. They always are a stand-in for the real thing.

Teach this to your organization and figure out ways to minimize abstraction. Later, others will wonder how far ahead you and your team are in terms of quality!

P.S. If you liked this text and are committed to leading your organization to better quality, you need skills to explain these topics to clients, colleagues, and bosses. For this purpose, I wrote the Storytools of Testing. You can get it here: prove.guru/thebook — The ideas in will be worth your time!