In a Lean environment, we embrace the philosophy of everyone measuring everything. In such a situation where measurement is encouraged, how is it that we end up with so many evil metrics? That is metrics that drive undesirable behavior.
In this short article, I will outline what constitutes good and evil metrics, and how to use metrics to promote good behaviors.
EVIL METRICS
The evil metrics are those that many of us will recognize as the corporate standard. These metrics are not inherently evil, but the behavior they drive is responsible for all sorts of bad products, layoffs, and general heartburn. Let’s explore.
1 – Evil Metrics: Percent Complete
A traditional project management metric is the report of Percent Complete. The metric was aggregated across project areas and used to determine is the project was trending toward an on-time delivery, or not.
Inherently, not too evil.
However, two primary behaviors are driven by this metric:
A. “If I am behind, I get in trouble; therefore I will always trend positive.
There is not much risk in doing so, because I know the project will be late (they always are), and I will be able to make up the time later.”
Thoughts: Jack Welch taught us a lot of great things about management, but the idea of managing knowledge workers with carrots, sticks, and a healthy dose of fear, is not one of them. To delight the customer, we need to be able to communicate in an open and honest environment to set appropriate expectations. Quite simply, we can no longer afford traditional project management games.
B. “I’m not sure how done I am.”
Thoughts: When working in a technical, highly sophisticated environment, it is mostly impossible to know how “done” on is due to the complex nature of the work being tackled. For example, if someone is executing 100 test cases, and 75 are complete, how done is that person? 75%? What if that person saved the hardest 25 cases for last? What if the 99th test case results in a whole host of defects that require weeks to resolve? Was that person, in fact, 75% complete?
Maybe a better question: does it matter?
Quite frankly, how “done” something is, is irrelevant. If a solution is not done, it cannot be used. What we want to understand is “what is now working, that previously was not,” “what is not yet working,” and “what will be working next.”
2 – Evil Metrics: Colors – The Traffic Light
Typically tied to percent complete is a “red,” “yellow,” or “green” color code intended to inform dashboard owners how a deliverable is trending.
· Green – We’re looking good! We will hit that delivery date.
· Yellow – Risk could potentially cause a miss against our delivery date.
· Red – We will miss our delivery date.
Again, not inherently evil.
However, organizational politics typically drive different behaviors.
· Green – “We don’t want to be green, because if we’re green and something happens, we’ll get in trouble.”
· Yellow – “Yellow is safe. If something happens, we saw it coming. If nothing happens, we were cautious.”
· Red – “We don’t want to be red, because if we’re red, someone will come to yell at us.”
So, where does that leave us? A few greens (for things that we will deliver very soon), a few reds (for things that will miss very more quickly), and a bunch of yellows (because we don’t know.)
Thoughts: The inherent fuzziness around our ability to hit future dates is tied to the fact that we typically focus on delivering things that are way too big. The bigger the batch, the higher degree of variability we have in the solution, and the higher the amount of “unknown unknown variables” tied to the objective’s outcome. We cannot know everything in advance.
When we focus on processing smaller batches, we have a higher degree of control over the variability.
For example, if we were to set a goal to build the perfect house, we would likely fail. There are just far too many variables. However, if we set out to build the perfect bathroom, we would have a much greater chance to succeed, because a bathroom is a more clearly defined outcome, and the variables easier to account.
3 – Evil Metrics: Hours Worked
Quite frankly, I do not care how many hours you worked. I would rather you work as few hours as possible, deliver high-quality outcomes, and go home.
Many workers wear the number of hours worked as a badge of honor – “I worked 60 (70, 80, 90) hours last week!”
As referenced by both Dan Pink in “When,” and Cal Newport in “Deep Work,” the number hours a person can spend doing deep knowledge work deteriorates quite rapidly after somewhere between 6 and 7 in a given day.
In short, come in, focus, do quality work, and leave. Measuring hours worked can drive people to work more and deliver outcomes that can harm the organization.
4 – Evil Metrics: Productivity
Another nod to our good friend Jack Welch is our desire to measure productivity. When a line worker is assembling Tonka Trucks, it is useful to estimate how many Tonka Trucks are constructed by a given person in an assigned shift. This measure will help us with margin planning and sales forecasting. Additionally, I can assure the productivity measure by creating rewards for increasing productivity (carrots), and punishment (sticks) for missing quota.
When we attempt to apply similar models to disciplines like software engineering, a subject where less is usually more, incentives such as lines-of-code targets can create undesirable behaviors and unsatisfactory solutions.
For example, if a programmer were to have a 500 line-of-code daily quota, but discovers a solution that can more efficiently solve a problem in 50 lines, it becomes a negative to solve the problem in the more efficient way.
GOOD METRICS
Good metrics represent the measure of things that satisfy the customer and help the organization operate more efficiently. They are objective, data-driven, and help us make responsible decisions about how to manage the business.
1 – Good Metrics: On-Time-Delivery
Another way to consider on-time-delivery is to view it as a reliability measure. Or, “how confident am I that a team will deliver on what they say they will.”
A great example of this is in the iteration goals set by many delivery teams:
“Over the next two weeks, we will deliver X, Y, and Z.”
Then, the iteration goals are contrasted against the iteration outcomes:
“Over the last two weeks, we delivered X and Y. We were unable to deliver Z because of …”
If a team is generally able to meet their commitments, many of the “evil” metrics noted above become irrelevant because we have trust that the team will do what they say they will.
2 – Good Metrics: Value Delivered (Customer Satisfaction)
There is only one boss. The customer. And he can fire everybody in the company from the chairman on down, just by spending his money somewhere else.
– Sam Walton
The bottom line is that if the customer is not happy, nothing else matters. Focus first on engaging customers to understand their needs, and then measure their satisfaction as you (iteratively) deliver the solution.
3 – Good Metrics: Net Promoter
In addition to satisfying the customer, it is essential also to create customers who will promote your product or service. Net Promoter, or NPS, is a measure of the likelihood of a customer to actively promote, actively detract, or remain neutral on your company’s product or service.
4 – Good Metrics: Employee Engagement
We understand that satisfying the customer is the number one objective, but also need to understand that it is impossible to satisfy the customer if those who are building the solution are not engaged in the mission. We need to focus on what is making our employees happy, what is causing them distress, and then fix those problems.
An organization that does not put their employees first will have employees who do not put the customer first.
5 – Good Metrics: Cumulative Flow
A cumulative flow diagram is a tool used in queuing theory. It is an area graph that shows the amount of work in a given state, arrival times, lead time, and cycle time. The diagram is used to help organizations understand the rate of flow in their organization, and to uncover bottlenecks in the system.
— If you found this content useful, the greatest compliment you can give me is to share it with a friend or colleague.