Last month I attended the Gartner Security & Risk Management Summit in Washington, D.C. I attended a lot of very good sessions, but the one that left the biggest mark on me was a session called “Metrics That Matter,” delivered by Jeffrey Wheatman.
I went to this session because I’ve had a lot of conversations with information security executives this year, and a common question is “What should I really be measuring?,” or they make comments like “I report on a lot of things, but I am not sure what the top security indicators are that I should roll up to my executive team.”
Wheatman shared a really good list of “Five characteristics of effective metrics,” and I think it is a good litmus test for our metrics (security or otherwise). I’ll paraphrase some of my session notes so you can get a feel for this:
- Effective metrics must support the business’s goals, and the connection to those goals should be clear.
- Effective metrics must be controllable. (In other words, don’t report on the number of vulnerabilities in your environment, since you can’t control that. Instead, report on the % of “Critical” systems patched within 72 hours, which you can control)
- Effective metrics must be quantitative.
- Effective metrics must be easy to collect and analyze. (Wheatman says “If it takes 3 weeks to gather data that you report on monthly, you should find an easier metric to track.”)
- Effective metrics are subject to trending. (Tracking progress and setting targets is vital to get people to pay attention)
This set of guidelines really resonated with me, and I am going to run my metrics through this regimen to make my own metrics better. If you’re a Gartner client, there is a detailed research report from Wheatman on this topic, and I suggest you grab a copy.
The other thing I’ve noticed is that there seems to be a gap out here in the real world in terms of effective security metrics that are “universal” and also meet these criteria. So, I’m on a quest to find and/or establish some good ones that transcend company boundaries.
If you have either a) good metrics that are working; b) vexing metrics problems you’d like to collaborate on; I would love to hear from you. Drop me a line at “dm at tripwire.com” and let me know what’s on your mind.
On a related note, we here at Tripwire wanted to dive deeper into the conversations surrounding risk, so we commissioned the Ponemon Institute to conduct a global survey among security, risk and privacy professionals. You can download the report at http://www.tripwire.com/ponemon2012/?djinn=701U0000000EHE8&utm_medium=blog&utm_camp=twblog and follow the conversation in Twitter with #RiskyBiz2012