Many of the most common metrics used to measure the effectiveness of the security operations center (SOC) are at best inaccurate and at worst actively harm SecOps teams, the National Cyber Security Centre (NCSC) has warned.
The NCSC’s CTO for architecture, Dave Chismon, wrote in a blog post that organizations often gravitate to measurements that can be easily expressed numerically to individuals who aren’t security specialists.
However, if “number of tickets processed” or “time taken to close a ticket” are used as metrics, staff may perversely be incentivized to rapidly triage and close them as false positives rather than investigate.
Similarly, “number of detection rules” may incentivize analysts to write as many rules as possible, driving up the number of false positives and ineffective rules.
In the same way, focusing on volume of logs collected over the value of those logs is self-defeating if they don’t improve detection, Chismon said.
Read more on SecOps: NCSC Shares Alternatives to Using a SOC
According to the NCSC, the only SOC metric that matters is: “does it detect (and respond to) attacks in a timely manner?” In other words, time to detect/time to respond (TTD/TTR).
Chismon recommended using red/purple teaming to allow assessment of a SOC’s TTD/TTR.
“Whilst TTD/TTR are the only reportable metrics that demonstrate a SOC is working, a SOC manager is likely to want to track a number of other metrics to help them monitor the week-by-week health of their service,” he continued.
“These metrics could include things like numbers of tickets, but crucially, those metrics should not be reported outwards (or arguably inwards, to the SOC analysts) lest they drive the wrong activities.”
How to Boost Threat Detection
To reduce TTD/TTR in the SOC, analysts must understand both the threat landscape and what they’re protecting, be experts in the tools they’re using, have the right data to spot unusual behavior and have time to hunt for threats.
Chismon recommended several approaches to build on:
- Hypothesis-led hunting, where analysts hypothesize about likely attacks based on their understanding of threat actors and their techniques, and then search for evidence in logs
- Maximal true positives/minimal false positives, where SOCs “maintain hard thresholds for false positive rates” when they’re evaluating whether a detection rule is suitable or not
- Metrics based around analyst awareness of threats such as completeness of documentation about a threat actor, or training reports read and actioned
- Tracking analyst expertise in tooling through training and certifications
- Tracking SOC engagement with the wider organization to spot and flag suspicious activity
- Analyst job satisfaction, which should be high if they are “learning about attackers, understanding techniques, applying it to data, and working with people across an organization”
- Log coverage: tracking the percentage of relevant assets that are reporting the right logs can help to reduce blind spots
“With the wrong metrics, a SOC is ineffective and the job is miserable, with analysts describing themselves as ‘ticket monkeys’ measured on clicking ‘false positives’ as quickly as possible, whilst being shamed for missing real attacks,” Chismon concluded.
“If you’re worried your SOC might be falling into this trap, a red or purple team from a credible vendor will give you proof either way.”
