Scale

Measuring the Analysis Process

I’ve previously written about measuring requirements and business analysts. I am concluding the series with my current thoughts on measuring the analysis process.

What_gets_measured_gets_done

This quote is a truism is because it’s how we work. Measurements help us identify areas we should focus on or improve. Measurements let us know when we succeeded, or not. I love the idea of measuring BAs and yet, I have spent years balking at the idea of measuring requirements and BAs. It doesn’t work very well in practice.* I had cause to rethink the how to measure BAs when one team I worked on took down the following action item, “Decide (if,) how and what BA velocity to track.” Continue reading “Measuring the Analysis Process” »

512878595_dae3c75aab_o

Measuring Business Analysts; Don’t KPI Me

Good managers often ask, “How do I know my team is performing well? How can I spot which folks need help? Who should I reward for a job well done?” In today’s busy world, where managers have significant responsibilities in addition to nurturing their team, measurements and metrics can be a a help.

Unfortunately, it is really easy to measure analysts poorly. Continue reading “Measuring Business Analysts; Don’t KPI Me” »

Measuring Requirements May Reinforce Bad Behavior

Have you ever heard this phrase?

“What gets measured, gets done.”

It contains huge truths about human nature, or at least my nature. I pay more attention to something when someone else is paying attention to it. I didn’t have to study the Hawthorne Effect to know this. Looking at my own life, there are lots of examples where if the boss was looking, I did my stuff. Where they didn’t care to look, I soon learned it was safe to ignore this area. And I often did.

More relevant to being a Business Analyst, I was brought into one company to train their new BAs. We spent lots of time covering the basics and then we started working on projects. The documentation was very uneven between the BAs (including me), so I said “Everything has to be peer reviewed before it goes out of the team.” And within a couple months everyone was writing great requirements. Really great. A couple months later, after we had stopped our internal reviews, the clarity and readability of our documentation was in the toilet. We had fallen back to our old habits. Just knowing a peer was going to review our work led everyone to do better work. There is no doubt in my mind, measurement and feedback can and do improve performance.

Unfortunately, most requirement metrics don’t help a BA, a project, or a team. They hurt it.

Typical Requirement Metrics

Measuring requirements can be difficult. Here’s a list of easy-to-gather requirement metrics:

  • Quantity (of use cases, requirements, user stories, process flow diagrams, etc.)
  • Volatility due to scope changes
  • Volatility due to technical & design changes
  • Volatility due to missed requirements
  • Volatility due to changes to existing requirements
  • Volatility due to clarification required
  • Defects due to requirements
  • Schedule variance due to poor or missing requirements
  • Tool compliance
  • Process compliance
  • Comparisons of the above versus:
    • initial estimates or baseline
    • time
    • project phase
    • project size
    • scope

These metrics have a couple elements in common. First, they are relatively easy to measure and calculate, presuming you have the right tools in place. I think many requirement management systems have been sold because for the first time companies will have metrics about requirements! Second, they all help ensure a team is builds correctly and in accordance to the original vision of the product.

Value?

I think it is important to ask, “How do these measurements benefit a team or project?” When I look at the above metrics, the focus seems to be on comparing requirements from one phase to the next, from how well did we understand the needs at the beginning of the project to how what was built at the end, to ensuring the process was followed. These all help teams “build right.” Which is important, but I think it is significantly more important to ask if the team is building the “right product.” Take a look at Gojko Adzic‘s chart from Specification by Example:

This chart describes my very real concern about requirement metrics. They help us build right. They help us ensure everything is checked-off the list, but none of the metrics push us to break the mold and build something great. They help us build something we planned months ago. I want to reinforce BA behaviors that push us to ask hard questions, pushing us up the value stream. I want us willing to take a new direction because we can see a project could be better. We should love killing projects because the value has already been reached, or never will be.

Behavior Reinforcement

If you want to improve teamwork, you need to reinforce behaviors that support acting like a good team member. If you reward people based on individual behavior, you are undermining the goal of improving teamwork. Similarly, it’s not uncommon to find metrics being misused. Sometimes the problem is minor, metrics are unused and the only impact is the wasted effort to gather and analyze them. Other issues include inappropriately punishing a team because of metrics and unintentionally focusing more on metrics than on delivering business value. None of these are part of the initial goal when a company or team decides to track information, but they are pretty easy to find.

Better choices

When you have to measure requirements, the first thing I want measured is their quality. First, define what a quality requirement ( or use case, or user story, or . . .) looks like. Then audit everyone against the standard. This ensures the quality of what BAs produce stays high. If your team is currently using a different style, template, or process for every project, this step will also push for more process compliance.

In a similar vein, I encourage teams to ensure everything goes through a peer review. I’m not sure metrics around this are especially valuable, but it might help with enforcement.

Lastly, think about the dimensions in Gojko’s chart above. Think about what’s really makes a difference. Fellow ThoughtWorker and Agile Manifesto signatory, Jim Highsmith has a great talk on this and often discusses with executives interested in enterprise agility. You should read his post and learn more about the most important metric of all, delivered value.