Evaluating Email Security: How to Select the Best Solution for Your Organization
In today’s saturated market for email security, it can be difficult to cut through the noise of AI hype and vendor claims. CISOs should be using a structured evaluation framework to support informed, objective comparisons of different vendors – to allow them to make the best decision for their organization.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Carlos Gray
Senior Product Marketing Manager, Email
Share
21
May 2025
When evaluating email security solutions, it’s crucial to move beyond marketing claims and focus on real-world performance. One of the most effective ways to achieve this is through an A/B comparison approach – a side-by-side evaluation of vendors based on consistent, predefined criteria.
This method cuts through biases, reveals true capability differences, and ensures that all solutions are assessed on a level playing field. It’s not just about finding an objectively good solution – it’s about finding the best solution for your organization’s specific needs.
An A/B comparison approach is particularly effective for three main reasons:
Eliminates bias: By comparing solutions under identical conditions, it’s easier to spot differences in performance without the fog of marketing jargon.
Highlights real capabilities: Direct side-by-side testing exposes genuine strengths and weaknesses, making it easier to judge which features are impactful versus merely decorative.
Encourages objective decision-making: This structured method reduces emotional or brand-driven decisions, focusing purely on metrics and performance.
Let’s look at the key factors to consider when setting up your evaluation to ensure a fair, accurate, and actionable comparison.
Deployment: Setting the stage for fair evaluation
To achieve a genuine comparison, deployment must be consistent across all evaluated solutions:
Establish the same scope: All solutions should be granted identical visibility across relevant tenants and domains to ensure parity.
Set a concrete timeline: Deploy and test each solution with the same dataset, at the same points in time. This allows you to observe differences in learning periods and adaptive capabilities.
Equal visibility and synchronized timelines prevent discrepancies that could skew your understanding of each vendor’s true capabilities. But remember – quicker results might not equal better learning or understanding!
Tuning and configurations: Optimizing for real-world conditions
Properly tuning and configuring each solution is critical for fair evaluation:
Compare on optimal performance: Consult with each vendor to understand what optimal deployment looks like for their solution, particularly if machine learning is involved.
Consider the long term: Configuration adjustments should be made with long-term usage in mind. Short-term fixes can mask long-term challenges.
Data visibility: Ensure each solution can retain and provide search capabilities on all data collected throughout the evaluation period.
These steps guarantee that you are comparing fully optimized versions of each platform, not underperforming or misconfigured ones.
Evaluation: Applying consistent metrics
Once deployment and configurations are aligned, the evaluation itself must be consistent, to prevent unfair scoring and help to identify true differences in threat detection and response capabilities.
Coordinate your decision criteria: Ensure all vendors are measured against the same set of criteria, established before testing begins.
Understand vendor threat classification: Each vendor may have different ways of classifying threats, so be sure to understand these nuances.
Maintain communication: If results seem inaccurate, engage with the vendors. Their response and remediation capabilities are part of the evaluation.
Making a decision: Look beyond the metrics
When it comes to reviewing the performance of each solution, it’s important to both consider and look beyond the raw data. This is about choosing the solution that best aligns with your specific business needs, which may include factors and features not captured in the results.
Evaluate based on results: Consider accuracy, threats detected, precision, and response effectiveness.
Evaluate beyond results: Assess the overall experience, including support, integrations, training, and long-term alignment with your security strategy.
Review and communicate: Internally review the findings and communicate them back to the vendors.
Choosing the right email security solution isn’t just about ticking boxes, it’s about strategic alignment with your organization’s goals and the evolving threat landscape. A structured, A/B comparison approach will help ensure that the solution you select is truly the best fit.
For a full checklist of the features and capabilities to compare, as well as how to perform a commercial and technical evaluation, check out the full Buyer’s Checklist for Evaluating Email Security.
[related-resource]
A Buyer’s Checklist for Evaluating Email Security
Get the most effective system for comparing email security solutions – based on real-world performance and measurable capabilities.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Why attack-centric approaches to email security can’t cope with modern threats
Despite evolving email threats, many organizations still rely on SEGs or outdated, attack-focused methods. These approaches can’t counter advanced, AI-driven attacks. The solution? Business-centric email security that understands users and inbox behavior, not just threats.
Why Data Classification Isn’t Enough to Prevent Data Loss
In a world of growing data volume and diversity, protecting and keeping track of your organization’s sensitive information is increasingly complex – particularly when 63% of breaches stem from malicious insiders or human error. This blog explores how security teams can achieve visibility beyond the limits of data classification, without adding to the burden of data management.
Fighting the Real Enemy: The Importance of Responsible Vulnerability Disclosure Between Email Security Vendors
This blog explores an exploitation capability observed by Darktrace in another email security vendor’s link rewriting and the steps Darktrace took to inform and resolve the issue.
Tracking CVE-2025-31324: Darktrace’s detection of SAP Netweaver exploitation before and after disclosure
Introduction: Exploiting SAP platforms
Global enterprises depend extensively on SAP platforms, such as SAP NetWeaver and Visual Composer, to run critical business processes worldwide. These systems; however, are increasingly appealing targets for well-resourced adversaries:
In March 2025, CISA issued an alert confirming active exploitation of a 2017 SAP NetWeaver vulnerability (CVE‑2017‑12637), enabling attackers to perform directory traversal and exfiltrate sensitive files, including credentials, from internet-facing systems
CVE-2025-31324 affects SAP’s NetWeaver Visual Composer, a web-based software modeling tool. SAP NetWeaver is an application server and development platform that runs and connects SAP and non-SAP applications across different technologies [2]. It is commonly used by process specialists to develop application components without coding in government agencies, large enterprises, and by critical infrastructure operators [4].
CVE-2025-31324 affects SAP’s Netweaver Visual Composer Framework 7.1x (all SPS) and above [4]. The vulnerability in a Java Servlet (/irj/servlet_jsp) would enable an unauthorized actor to upload arbitrary files to the /developmentserver/metadatauploader endpoint, potentially resulting in remote code execution (RCE) and full system compromise [3]. The issue stems from an improper authentication and authorization check in the SAP NetWeaver Application Server Java systems [4].
What is the severity rating of CVE-2025-31324?
The vulnerability, first disclosed on April 24, 2025, carries the highest severity rating (CVSS v3 score: 10.0) and could allow remote attackers to upload malicious files without requiring authentication [1][5]. Although SAP released a workaround on April 8, many organizations are hesitant to take their business-critical SAP NetWeaver systems offline, leaving them exposed to potential exploitation [2].
How is CVE-2025-31324 exploited?
The vulnerability is exploitable by sending specifically crafted GET, POST, or HEAD HTTP requests to the /developmentserver/metadatauploader URL using either HTTP or HTTPS. Attackers have been seen uploading malicious files (.jsp, .java, or .class files to paths containing “\irj\servlet_jsp\irj\”), most of them being web shells, to publicly accessible SAP NetWeaver systems.
External researchers observed reconnaissance activity targeting this vulnerability in late January 2025, followed by a surge in exploitation attempts in February. The first confirmed compromise was reported in March [4].
Multiple threat actors have reportedly targeted the vulnerability, including Chinese Advanced Persistent Threats (APTs) groups Chaya_004 [7], UNC5221, UNC5174, and CL-STA-0048 [8], as well as ransomware groups like RansomEXX, also known as Storm-2460, BianLian [4] or Qilin [6] (the latter two share the same indicators of compromise (IoCs)).
Following the initial workaround published on April 8, SAP released a security update addressing CVE-2025-31324 and subsequently issued a patch on May 13 (Security Note 3604119) to resolve the root cause of the vulnerability [4].
Darktrace’s coverage of CVE-2025-31324 exploitation
Darktrace has observed activity indicative of threat actors exploiting CVE-2025-31324, including one instance detected before the vulnerability was publicly disclosed.
In April 2025, the Darktrace Threat Research team investigated activity related to the CVE-2025-31324 on SAP devices and identified two cases suggesting active exploitation of the vulnerability. One case was detected prior to the public disclosure of the vulnerability, and the other just two days after it was published.
Early detection of CVE 2025-31324 by Darktrace
Figure 1: Timeline of events for an internet-facing system, believed to be a SAP device, exhibiting activity indicative of CVE-2025-31324 exploitation.
On April 18, six days prior to the public disclosure of CVE-2025-31324, Darktrace began to detect unusual activity on a device belonging to a logistics organization in the Europe, the Middle East and Africa (EMEA) region. Multiple IoCs observed during this incident have since been linked via OSINT to the exploitation of CVE-2025-31324. Notably, however, this reporting was not available at the time of detection, highlighting Darktrace’s ability to detect threats agnostically, without relying on threat intelligence.
The device was observed making domain name resolution request for the Out-of-Band Application Security Testing (OAST) domain cvvr9gl9namk9u955tsgaxy3upyezhnm6.oast[.]online. OAST is often used by security teams to test if exploitable vulnerabilities exist in a web application but can similarly be used by threat actors for the same purpose [9].
Four days later, on April 22, Darktrace observed the same device, an internet-facing system believed to be a SAP device, downloading multiple executable (.exe) files from several Amazon Simple Storage Service (S3). Darktrace’s Threat Research team later found these files to be associated with the KrustyLoader malware [23][24][25].
KrustyLoader is known to be associated with the Chinese threat actor UNC5221, also known as UTA0178, which has been reported to aggressively target devices exposed to the internet [10] [14] [15]. It is an initial-stage malware which downloads and launches a second-stage payload – Sliver C2. Sliver is a similar tool to Cobalt Strike (an open-source post-exploitation toolkit). It is used for command-and-control (C2) connections [11][12]13]. After its successful download, KrustyLoader deletes itself to evade detection. It has been reported that multiple Chinese APT groups have deployed KrustyLoader on SAP Netweaver systems post-compromise [8].
The actors behind KrustyLoader have also been associated with the exploitation of zero-day vulnerabilities in other enterprise systems, including Ivanti devices [12]. Notably, in this case, one of the Amazon S3 domains observed (abode-dashboard-media.s3.ap-south-1.amazonaws[.]com ) had previously been investigated by Darktrace’s Threat Research team as part of their investigation into Ivanti Connect Secure (CS) and Policy Secure (PS) appliances.
In addition to the download of known malicious files, Darktrace also detected new IoCs, including several executable files that could not be attributed to any known malware families or previous attacks, and for which no corresponding OSINT reporting was available.
Post-CVE publication detection
Exploit Validation
Between April 27 and 29, Darktrace observed unusual activity from an SAP device on the network of a manufacturing customer in EMEA.
Figure 2: Darktrace / NETWORK’s detection of an SAP device performing a large volume of suspicious activity between April 27 and April 29.
The device was observed making DNS requests for OAST domains (e.g. aaaaaaaa.d06qqn7pu5a6u25tv9q08p5xhbjzw33ge.oast[.]online and aaaaaaaaaaa.d07j2htekalm3139uk2gowmxuhapkijtp.oast[.]pro), suggesting that a threat actor was testing for exploit validation [9].
Figure 3: Darktrace / NETWORK’s detection of a SAP device making suspicious domain name resolution requests for multiple OAST domains.
Privilege escalation tool download attempt
One day later, Darktrace observed the same device attempting to download an executable file from hxxp://23.95.123[.]5:666/xmrigCCall/s.exe (SHA-1 file hash: e007edd4688c5f94a714fee036590a11684d6a3a).
Darktrace / NETWORK identified the user agents Microsoft-CryptoAPI/10.0 and CertUtil URL Agent during the connections to 23.95.123[.]5. The connections were made over port 666, which is not typically used for HTTP connections.
Multiple open-source intelligence (OSINT) vendors have identified the executable file as either JuicyPotato or SweetPotato, both Windows privilege escalation tools[16][17][18][19]. The file hash and the unusual external endpoint have been associated with the Chinese APT group Gelsemium in the past, however, many threat actors are known to leverage this tool in their attacks [20] [21].
Figure 4: Darktrace’s Cyber AI Analyst’s detection of a SAP device downloading a suspicious executable file from hxxp://23.95.123[.]5:666/xmrigCCall/s.exe on April 28, 2025.
Darktrace deemed this activity highly suspicious and triggered an Enhanced Monitoring model alert, a high-priority security model designed to detect activity likely indicative of compromise. As the customer was subscribed to the Managed Threat Detection service, Darktrace’s Security Operations Centre (SOC) promptly investigated the alert and notified the customer for swift remediation. Additionally, Darktrace’s Autonomous Response capability automatically blocked connections to the suspicious IP, 23.95.123[.]5, effectively containing the compromise in its early stages.
Figure 5: Actions taken by Darktrace’s Autonomous Response to block connections to the suspicious external endpoint 23.95.123[.]5. This event log shows that the connections to 23.95.123[.]5 were made over a rare destination port for the HTTP protocol and that new user agents were used during the connections.
Conclusion
The exploitation of CVE-2025-31324 to compromise SAP NetWeaver systems highlights the persistent threat posed by vulnerabilities in public-facing assets. In this case, threat actors leveraged the flaw to gain an initial foothold, followed by attempts to deploy malware linked to groups affiliated with China [8][20].
Crucially, Darktrace demonstrated its ability to detect and respond to emerging threats even before they are publicly disclosed. Six days prior to the public disclosure of CVE-2025-31324, Darktrace detected unusual activity on a device believed to be a SAP system, which ultimately represented an early detection of the CVE. This detection was made possible through Darktrace’s behavioral analysis and anomaly detection, allowing it to recognize unexpected deviations in device behavior without relying on signatures, rules or known IoCs. Combined with its Autonomous Response capability, this allowed for immediate containment of suspicious activity, giving security teams valuable time to investigate and mitigate the threat.
Credit to Signe Zaharka (Principal Cyber Analyst), Emily Megan Lim, (Senior Cyber Analyst) and Ryan Traill (Analyst Content Lead)
Appendices
List of IoCs
23.95.123[.]5:666/xmrigCCall/s.exe - URL- JuicyPotato/SweetPotato - high confidence
29274ca90e6dcf5ae4762739fcbadf01- MD5 file hash - JuicyPotato/SweetPotato - high confidence
Proactive OT Security: Lessons on Supply Chain Risk Management from a Rogue Raspberry Pi
Understanding supply chain risk in manufacturing
For industries running Industrial Control Systems (ICS) such as manufacturing and fast-moving consumer goods (FMCG), complex supply chains mean that disruption to one weak node can have serious impacts to the entire ecosystem. However, supply chain risk does not always originate from outside an organization’s ICS network.
The implicit trust placed on software or shared services for maintenance within an ICS can be considered a type of insider threat [1], where defenders also need to look ‘from within’ to protect against supply chain risk. Attackers have frequently mobilised this form of insider threat:
Many ICS and SCADA systems were compromised during the 2014 Havex Watering Hole attack, where via operators’ implicit trust in the trojanized versions of legitimate applications, on legitimate but compromised websites [2].
In 2018, the world’s largest manufacturer of semiconductors and processers shut down production for three days after a supplier installed tainted software that spread to over 10,000 machines in the manufacturer’s network [3].
During the 2020 SolarWinds supply chain attack, attackers compromised a version of Orion software that was deployed from SolarWinds’ own servers during a software update to thousands of customers, including tech manufacturing companies such as Intel and Nvidia [4].
Traditional approaches to ICS security have focused on defending against everything from outside the castle walls, or outside of the ICS network. As ICS attacks become more sophisticated, defenders must not solely rely on static perimeter defenses and prevention.
A critical part of active defense is understanding the ICS environment and how it operates, including all possible attack paths to the ICS including network connections, remote access points, the movement of data across zones and conduits and access from mobile devices. For instance, original equipment manufacturers (OEMs) and vendors often install remote access software or third-party equipment in ICS networks to facilitate legitimate maintenance and support activities, which can unintentionally expand the ICS’ attack surface.
This blog describes an example of the convergence between supply chain risk and insider risk, when a vendor left a Raspberry Pi device in a manufacturing customer’s ICS network without the customer’s knowledge.
Case study: Using unsupervised machine learning to detect pre-existing security issues
Raspberry Pi devices are commonly used in SCADA environments as low-cost, remotely accessible data collectors [5][6][7]. They are often paired with Industrial Internet of Things (IIoT) for monitoring and tracking [8]. However, these devices also represent a security risk because their small physical size and time-consuming nature of physical inspection makes them easy to overlook. This poses a security risk, as these devices have previously been used to carry out USB-based attacks or to emulate Ethernet-over-USB connections to exfiltrate sensitive data [8][9].
In this incident,a Darktrace customer was unaware that their supplier had installed a Raspberry Pi device on their ICS network. Crucially, the installation occurred prior to Darktrace’s deployment on the customer’s network.
For other anomaly detection tools, this order of events meant that this third-party device would likely have been treated as part of the customer’s existing infrastructure. However, after Darktrace was deployed, it analyzed the metadata from the encrypted HTTPS and DNS connections that the Raspberry Pi made to ‘call home’ to the supplier and determined that these connections were unusual compared to the rest of the devices in the network, even in the absence of any malicious indicators of compromise (IoCs).
Darktrace triggered the following alerts for this unusual activity that consequently notified the customer to the pre-existing threat of an unmanaged device already present in their network:
Compromise / Sustained SSL or HTTP Increase
Compromise / Agent Beacon (Short Period)
Compromise / Agent Beacon (Medium Period)
Compromise / Agent Beacon (Long Period)
Tags / New Raspberry Pi Device
Device / DNS Requests to Unusual Server
Device / Anomaly Indicators / Spike in Connections to Rare Endpoint Indicator
Figure 1: Darktrace’s External Sites Summary showing the rarity of the external endpoint that the Raspberry Pi device ‘called home’ to and the model alerts triggered.
Darktrace’s Cyber AI Analyst launched an autonomous investigation into the activity, correlating related events into a broader incident and generating a report outlining the potential threat along with supporting technical details.
Darktrace’s anomaly-based detection meant that the Raspberry Pi device did not need to be observed performing clearly malicious behavior to alert the customer to the security risk, and neither can defenders afford to wait for such escalation.
Why is this significant?
In 2021 a similar attack took place. Aiming to poison a Florida water treatment facility, attackers leveraged a TeamViewer instance that had been dormant on the system for six months, effectively allowing the attacker to ‘live off the land’ [10].
The Raspberry Pi device in this incident also remained outside the purview of the customer’s security team at first. It could have been leveraged by a persistent attacker to pivot within the internal network and communicate externally.
A proactive approach to active defense that seeks to minimize and continuously monitor the attack surface and network is crucial.
The growing interest in manufacturing from attackers and policymakers
Significant motivations for targeting the manufacturing sector and increasing regulatory demands make the convergence of supply chain risk, insider risk, and the prevalence of stealthy living-off-the-land techniques particularly relevant to this sector.
Manufacturing is consistently targeted by cybercriminals [11], and the sector’s ‘just-in-time’ model grants attackers the opportunity for high levels of disruption. Furthermore, under NIS 2, manufacturing and some food and beverage processing entities are now designated as ‘important’ entities. This means stricter incident reporting requirements within 24 hours of detection, and enhanced security requirements such as the implementation of zero trust and network segmentation policies, as well as measures to improve supply chain resilience [12][13][14].
How can Darktrace help?
Ultimately, Darktrace successfully assisted a manufacturing organization in detecting a potentially disruptive 'near-miss' within their OT environment, even in the absence of traditional IoCs. Through passive asset identification techniques and continuous network monitoring, the customer improved their understanding of their network and supply chain risk.
While the swift detection of the rogue device allowed the threat to be identified before it could escalate, the customer could have reduced their time to respond by using Darktrace’s built-in response capabilities, had Darktrace’s Autonomous Response capability been enabled. Darktrace’s Autonomous Response can be configured to target specific connections on a rogue device either automatically upon detection or following manual approval from the security team, to stop it communicating with other devices in the network while allowing other approved devices to continue operating. Furthermore, the exportable report generated by Cyber AI Analyst helps security teams to meet NIS 2’s enhanced reporting requirements.
Sophisticated ICS attacks often leverage insider access to perform in-depth reconnaissance for the development of tailored malware capabilities. This case study and high-profile ICS attacks highlight the importance of mitigating supply chain risk in a similar way to insider risk. As ICS networks adapt to the introduction of IIoT, remote working and the increased convergence between IT and OT, it is important to ensure the approach to secure against these threats is compatible with the dynamic nature of the network.
Credit to Nicole Wong (Principal Cyber Analyst), Matthew Redrup (Senior Analyst and ANZ Team Lead)
[related-resource]
Appendices
MITRE ATT&CK Mapping
Infrastructure / New Raspberry Pi Device - INITIAL ACCESS - T1200 Hardware Additions
Device / DNS Requests to Unusual Server - CREDENTIAL ACCESS, COLLECTION - T1557 Man-in-the-Middle
Compromise / Agent Beacon - COMMAND AND CONTROL - T1071.001 Web Protocols
.jpg)
.jpg)
![Actions taken by Darktrace’s Autonomous Response to block connections to the suspicious external endpoint 23.95.123[.]5. This event log shows that the connections to 23.95.123[.]5 were made over a rare destination port for the HTTP protocol and that new user agents were used during the connections.](https://cdn.prod.website-files.com/626ff4d25aca2edf4325ff97/685195a1592b0029efa4fd53_Screenshot%202025-06-17%20at%209.19.36%E2%80%AFAM.png)
