Measuring Requirements May Reinforce Bad Behavior

Have you ever heard this phrase?

“What gets measured, gets done.”

It contains huge truths about human nature, or at least my nature. I pay more attention to something when someone else is paying attention to it. I didn’t have to study the Hawthorne Effect to know this. Looking at my own life, there are lots of examples where if the boss was looking, I did my stuff. Where they didn’t care to look, I soon learned it was safe to ignore this area. And I often did.

More relevant to being a Business Analyst, I was brought into one company to train their new BAs. We spent lots of time covering the basics and then we started working on projects. The documentation was very uneven between the BAs (including me), so I said “Everything has to be peer reviewed before it goes out of the team.” And within a couple months everyone was writing great requirements. Really great. A couple months later, after we had stopped our internal reviews, the clarity and readability of our documentation was in the toilet. We had fallen back to our old habits. Just knowing a peer was going to review our work led everyone to do better work. There is no doubt in my mind, measurement and feedback can and do improve performance.

Unfortunately, most requirement metrics don’t help a BA, a project, or a team. They hurt it.

Typical Requirement Metrics

Measuring requirements can be difficult. Here’s a list of easy-to-gather requirement metrics:

  • Quantity (of use cases, requirements, user stories, process flow diagrams, etc.)
  • Volatility due to scope changes
  • Volatility due to technical & design changes
  • Volatility due to missed requirements
  • Volatility due to changes to existing requirements
  • Volatility due to clarification required
  • Defects due to requirements
  • Schedule variance due to poor or missing requirements
  • Tool compliance
  • Process compliance
  • Comparisons of the above versus:
    • initial estimates or baseline
    • time
    • project phase
    • project size
    • scope

These metrics have a couple elements in common. First, they are relatively easy to measure and calculate, presuming you have the right tools in place. I think many requirement management systems have been sold because for the first time companies will have metrics about requirements! Second, they all help ensure a team is builds correctly and in accordance to the original vision of the product.

Value?

I think it is important to ask, “How do these measurements benefit a team or project?” When I look at the above metrics, the focus seems to be on comparing requirements from one phase to the next, from how well did we understand the needs at the beginning of the project to how what was built at the end, to ensuring the process was followed. These all help teams “build right.” Which is important, but I think it is significantly more important to ask if the team is building the “right product.” Take a look at Gojko Adzic‘s chart from Specification by Example:

This chart describes my very real concern about requirement metrics. They help us build right. They help us ensure everything is checked-off the list, but none of the metrics push us to break the mold and build something great. They help us build something we planned months ago. I want to reinforce BA behaviors that push us to ask hard questions, pushing us up the value stream. I want us willing to take a new direction because we can see a project could be better. We should love killing projects because the value has already been reached, or never will be.

Behavior Reinforcement

If you want to improve teamwork, you need to reinforce behaviors that support acting like a good team member. If you reward people based on individual behavior, you are undermining the goal of improving teamwork. Similarly, it’s not uncommon to find metrics being misused. Sometimes the problem is minor, metrics are unused and the only impact is the wasted effort to gather and analyze them. Other issues include inappropriately punishing a team because of metrics and unintentionally focusing more on metrics than on delivering business value. None of these are part of the initial goal when a company or team decides to track information, but they are pretty easy to find.

Better choices

When you have to measure requirements, the first thing I want measured is their quality. First, define what a quality requirement ( or use case, or user story, or . . .) looks like. Then audit everyone against the standard. This ensures the quality of what BAs produce stays high. If your team is currently using a different style, template, or process for every project, this step will also push for more process compliance.

In a similar vein, I encourage teams to ensure everything goes through a peer review. I’m not sure metrics around this are especially valuable, but it might help with enforcement.

Lastly, think about the dimensions in Gojko’s chart above. Think about what’s really makes a difference. Fellow ThoughtWorker and Agile Manifesto signatory, Jim Highsmith has a great talk on this and often discusses with executives interested in enterprise agility. You should read his post and learn more about the most important metric of all, delivered value.

 

  4Comments

  1. David Wright (@dwwright99)   •  

    Agreed, delivering a great product/system that is the wrong one is worse than delivering nothing at all.

    One way to avoid this is clear definition of business mission and objectives; subsequently defined requirements that don’t align with these are to be questioned.

    The main metric I try to use is nothing new, it is the number of changes to requirements made between requirements definition and product/system delivery. This does not tell you why you were getting the requirements wrong, but a lot of changes indicates there is a problem to be addressed.

  2. @JeffreyGoodReq   •     Author

    A LinkedIn conversation asked what good requirement measures are. I mention them, but don’t go into much detail. Here’s my online reply:

    If you are trying to measure the quality of an individual requirement, you probably need to audit it against something like this.

    For User Stories, I like INVEST.

    It really depends on what you’re trying to accomplish. I only just learned about the following: Goal – Question – Metric. I like what I’ve read about it this afternoon and will have to write a new post or addendum soon.

    But the real point of the article was, are you measuring what’s important? It’s much more important to understand the value of the project than it will ever be to measure the quality of the requirements.

    As Curtis Michelson tweeted, “An imprecise measure of the right thing is better than a precise measure of the wrong.”

  3. agile project manager   •  

    Hi there, I discovered your website by way of Google at the same time as searching for a comparable matter, your web site came up, it seems to be great. I have bookmarked to my favourites|added to bookmarks.

Comments are closed.