MITRE Engenuity has released its 2023 ATT&CK evaluations, examining how top cybersecurity vendors detect and prevent sophisticated cyberthreats. This year, the evaluations focused on the techniques of Turla, a Russia-based threat group.<\/p>\n\n\n\n
Turla uses a command-and-control network, as well as open source tools, which are more difficult to protect and easier to exploit because anyone can edit \u2014 and abuse \u2014 the code.<\/p>\n\n\n\n
This year’s MITRE analysis tested vendors’ ability to detect two scenarios called SNAKE and CARBON. MITRE used multiple offensive security tools, including Keylogger and Mimikatz, to launch attacks on vendors’ environments. The vendors were also tested on protection capabilities, undergoing thirteen tests \u2014 some with many steps \u2014 to see at which step they could halt an attack.<\/p>\n\n\n\n
MITRE’s detection and protection evaluations have usually attracted endpoint security vendors, with the detection evaluations best suited for endpoint detection and response (EDR)<\/a> products and the protection tests focusing on the abilities of endpoint protection platforms (EPP<\/a>). Vendors offering both EDR and EPP capabilities for Windows and Linux are able to participate in more steps of an evaluation than vendors with more limited offerings. Over time, security vendors whose primary strengths lie elsewhere have increasingly participated in the respected program.<\/p>\n\n\n\n We encourage security buyers to research these vendors, including their MITRE scores over time, before making a purchase. Our analysis provides one way to look at the MITRE evaluations from an angle that can be helpful. But as we noted in our analysis of last year’s results<\/a>, your organization will need to test security products in your own infrastructure before you know if it will work for you. And looking into the details of the MITRE tests may also give you significant information about how a product might perform in your environment.<\/p>\n\n\n\n The MITRE results<\/a> were separated into two categories: detection (SNAKE and CARBON scenarios) and protection (13 tests of a product’s ability to stop an attack).<\/p>\n\n\n\n The detection evaluation involved 143 total steps. For vendors who skipped the Linux tests, that number drops to 132. To calculate the detection scores, we divided the total number of successfully detected steps by 143 (or 132).<\/p>\n\n\n\n The detection tests include an analytics score, a telemetry score, and a visibility score. Analytics coverage not only detects the threat but also tags it with the MITRE standard identification. Telemetry coverage is the collection of raw data about a threat event, not necessarily including context. Visibility coverage is the overall number of detections that MITRE tested and the vendor successfully detected. We cover only the visibility score in our analysis of MITRE testing.<\/p>\n\n\n\n Cisco’s and Check Point’s detection and protection scores weren’t recorded due to technological issues, according to MITRE.<\/p>\n\n\n\n The protection component consists of 13 tests, evaluating which vendors can stop all thirteen attack sequences and how quickly they can do so. Most of the 13 tests had multiple attack steps. Protection tests were optional, and not all vendors participated, including Rapid7 and WithSecure.<\/p>\n\n\n\n On the protection side, there were a few tests in particular with which multiple vendors struggled. Many vendors missed test three, including Fortinet, Bitdefender, and Sophos. Malwarebytes couldn’t complete it either.<\/p>\n\n\n\n And many struggled with test seven as well. While Fortinet eventually completed seven, it took many steps. Same for Bitdefender. VMware Carbon Black didn’t complete test seven, and Tehtris missed every step. A few vendors ran into trouble with test 13 as well. This year’s evaluations revealed some common threads, with a few notable protection tests that appeared to be particularly hard.<\/p>\n\n\n\n Palo Alto Networks had a perfect score, detecting all 143 detection tests and stopping all 13 protection evaluations on the first step. Three other vendors \u2014 Microsoft, CrowdStrike and Cybereason \u2014 all successfully detected the 143 detection tests and stopped the 13 protection attacks, but missed a small number of the protection steps before stopping the threat.<\/p>\n\n\n\n A number of other vendors had strong showings, but the results for many vendors left room for improvement, especially on the protection end. Many high-profile security vendors failed to stop multiple protection tests. Product teams typically use the tests to improve their offerings, so participation is always a net positive.<\/p>\n\n\n\n Vendors have noted a number of caveats about the evaluations \u2014 namely, that the detection tests can potentially be gamed by vendors setting detection sensitivity levels high enough to produce false alerts in the real world, and some vendors have said they had to disable key security features to participate. Those caveats make the protection tests the more important of the two, and many welcomed them when they were introduced two years ago<\/a>.<\/p>\n\n\n\n Many of last year’s winners scored high for detection again this year, including Palo Alto, Microsoft, CrowdStrike, and Cybereason. Five vendors received perfect visibility scores in the detection evaluations, and Sophos was one vendor that scored a noteworthy comeback from a middling 2022 result, underscoring that vendors often use the results to better their products.<\/p>\n\n\n\n The following table gives the overall visibility score for each vendor in the detection tests, from highest number of successful detection tests to the lowest.<\/p>\n\n\n\n The protection test results this year displayed a wide performance range. Only seven vendors stopped all the tests they faced, the same number as last year. But many vendors missed multiple protection tests entirely. There were 13 tests total, and many contained multiple steps. This table compares three different items:<\/p>\n\n\n\n These metrics show different facets of the protection tests, rather than a single overall percentage.<\/p>\n\n\n\nHow to Analyze This Year\u2019s MITRE Results<\/h2>\n\n\n\n
Detection & Protection Results Were a Mixed Bag<\/h2>\n\n\n\n
Detection visibility score, in percent<\/strong><\/td><\/tr> Crowdstrike<\/td> 100%<\/td><\/tr> Cybereason<\/td> 100%<\/td><\/tr> Cynet<\/td> 100%<\/td><\/tr> Microsoft<\/td> 100%<\/td><\/tr> Palo Alto<\/td> 100%<\/td><\/tr> Sophos<\/td> 98.6%<\/td><\/tr> Fortinet<\/td> 97.9%<\/td><\/tr> Bitdefender<\/td> 91.61%<\/td><\/tr> Deep Instinct<\/td> 89.39%<\/td><\/tr> SentinelOne<\/td> 88.11%<\/td><\/tr> Trend Micro<\/td> 88.11%<\/td><\/tr> Uptycs<\/td> 88.11%<\/td><\/tr> Harfang Labs<\/td> 87.41%<\/td><\/tr> Malwarebytes<\/td> 82.52%<\/td><\/tr> Blackberry<\/td> 81.82%<\/td><\/tr> Trellix<\/td> 81.12%<\/td><\/tr> Elastic<\/td> 80.42%<\/td><\/tr> WatchGuard<\/td> 78.79%<\/td><\/tr> Qualys<\/td> 78.32%<\/td><\/tr> Taegis (secureworks)<\/td> 78.32%<\/td><\/tr> ESET<\/td> 77.62%<\/td><\/tr> Symantec<\/td> 75.52%<\/td><\/tr> Ahn Labs<\/td> 74.24%<\/td><\/tr> IBM Security<\/td> 72.03%<\/td><\/tr> VMware Carbon Black<\/td> 72.03%<\/td><\/tr> Rapid7<\/td> 70.63%<\/td><\/tr> Tehtris<\/td> 67.13%<\/td><\/tr> WithSecure<\/td> 67.13%<\/td><\/tr> Somma<\/td> 58.74%<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n \n