Hackers’ Guide to Rogue VM Deployment: Lessons from the MITRE hack

In the rapidly evolving landscape of cybersecurity, sophisticated attacks continue to challenge even the most prepared organizations. A recent cyber intrusion targeting MITRE’s Networked Experimentation, Research, and Testing environment has brought to light the vulnerabilities within virtualized infrastructures, particularly those utilizing VMware. This article provides an in-depth analysis of the attack, the methodologies employed by the attackers, and the broader implications for cybersecurity practices.

The Attack: A Detailed Examination

The cyber intrusion into MITRE’s environment was a meticulously planned and executed operation, highlighting the attackers’ advanced technical capabilities and understanding of virtualized environments. The attackers exploited specific vulnerabilities in Ivanti Connect Secure (ICS), identified as CVE-2023-46805 and CVE-2024-21887. These vulnerabilities allowed unauthorized access to the VMware infrastructure, providing the attackers with a foothold within the network.

Initial Penetration and Exploitation: The attackers began by identifying and exploiting weaknesses in the Ivanti Connect Secure (ICS) infrastructure. The vulnerabilities in question were zero-day exploits, meaning they were unknown to the vendor and had no existing patches or mitigations at the time of the attack. By exploiting these vulnerabilities, the attackers could bypass authentication mechanisms and gain administrative access to the virtualized environment.

Deployment of Rogue Virtual Machines (VMs): Once inside the network, the attackers created and deployed rogue VMs. These VMs were crafted to mimic legitimate virtual machines, allowing them to blend into the existing infrastructure and evade detection. The deployment of rogue VMs served multiple purposes:

• Persistence: Rogue VMs provided a stable and resilient presence within the network, ensuring that the attackers could maintain access over an extended period.
• Evasion: By operating within the virtualized environment, the rogue VMs could bypass traditional security measures that focus on physical or network-based threats.
• Expansion: The rogue VMs acted as a base for further malicious activities, including data exfiltration, lateral movement within the network, and the deployment of additional malware.

Command and Control (C2) Operations: The attackers established robust C2 channels to maintain control over the rogue VMs. These channels allowed the attackers to issue commands, receive data, and monitor the status of their malicious operations. The C2 infrastructure was designed to be resilient, utilizing techniques such as encryption and redundancy to avoid detection and disruption.

Technical Deep Dive: Understanding the Attack

To fully appreciate the sophistication of the attack, it is essential to delve into the technical aspects of the methodologies employed by the attackers.

1. Vulnerability Exploitation:
   • The vulnerabilities exploited, CVE-2023-46805 and CVE-2024-21887, were critical flaws within the Ivanti Connect Secure (ICS) software. These flaws allowed the attackers to execute arbitrary code and gain administrative privileges within the virtualized environment.
   • The attackers used a combination of social engineering, phishing, and advanced scanning techniques to identify vulnerable systems. Once identified, they deployed custom exploit scripts to gain access.
2. Rogue VM Deployment:
   • The deployment process involved creating VMs that were virtually identical to legitimate ones, making detection difficult. The attackers leveraged existing VM templates and modified them to include their malicious payloads.
   • These rogue VMs were configured to operate with minimal resource usage, further reducing the likelihood of detection through performance monitoring.
   • Rogue VMs are created and managed through service accounts directly on the hypervisor, rather than through the vCenter administrative console. As a result, these VMs do not appear in the inventory.
   • The adversary created their own rogue VMs within the VMware environment, leveraging compromised vCenter Server access. They wrote and deployed a JSP web shell (BEEFLUSH) under the vCenter Server’s Tomcat server to execute a Python-based tunneling tool, facilitating SSH connections between adversary-created VMs and the ESXi hypervisor infrastructure.
   • By deploying rogue VMs, adversaries can evade detection by hiding their activities from centralized management interfaces like vCenter. This allows them to maintain control over compromised systems while minimizing the risk of discovery.
3. Persistence Mechanisms:
   • To ensure persistence, the attackers implemented several techniques within the rogue VMs. These included installing rootkits and other low-level malware that could survive reboots and updates.
   • The attackers also manipulated the VM management tools to hide the presence of the rogue VMs from administrators.
4. Evasion Tactics:
   • The attackers employed various evasion tactics to avoid detection by security tools. These included using encrypted communication channels, obfuscating malicious code, and leveraging legitimate administrative tools to carry out their activities.
   • They also frequently rotated their command and control servers to avoid being blacklisted or shut down.

Implications for Cybersecurity

The MITRE cyber intrusion serves as a stark reminder of the evolving tactics used by cybercriminals and the vulnerabilities inherent in virtualized environments. This incident highlights several critical areas for improvement in cybersecurity practices:

Enhanced Vulnerability Management: Organizations must adopt rigorous vulnerability management practices to identify and remediate vulnerabilities promptly. This includes regular patching, conducting vulnerability assessments, and staying informed about emerging threats.

Advanced Detection Mechanisms: Traditional security measures are often inadequate in virtualized environments. Organizations need to implement advanced detection mechanisms that can identify anomalous activities within virtualized infrastructures. This includes behavior-based monitoring, anomaly detection, and machine learning algorithms to identify suspicious activities.

Comprehensive Security Training: Human factors remain a significant vulnerability in cybersecurity. Comprehensive training programs for employees can help reduce the risk of social engineering and phishing attacks, which are often the initial vectors for intrusions.

Robust Incident Response Plans: Having a well-defined incident response plan is crucial for mitigating the impact of cyber intrusions. This plan should include procedures for identifying, containing, and eradicating threats, as well as recovery strategies to restore normal operations.

Detecting Adversary Activity in VMware Ecosystem

In VMware’s environment, spotting adversary activity demands meticulous scrutiny. For instance, adversaries might enable SSH on hypervisors and log in by routing traffic through the vCenter Server. This technique underscores the importance of monitoring SSH activity for signs of unauthorized access.

What to Look For:

1. Anomalous SSH Enablement: Keep a close watch for unexpected occurrences of “SSH login enabled” messages. Any activation of SSH outside the normal administrative cycle could indicate malicious activity.
2. Unusual SSH Sessions: Monitor for deviations from the expected pattern of SSH sessions being opened. Look out for instances where “SSH session was opened for” messages occur unexpectedly or at unusual times.

Notable ATT&CK Techniques: Deploying Rogue VMs

Moving forward to January 7, 2024, the adversary accessed VMs and deployed malicious payloads, the BRICKSTORM backdoor and the BEEFLUSH web shell. The adversary also used a default VMware account, VPXUSER, to make seven API calls that enumerated a list of mounted and unmounted drives.

The adversary bypassed detection mechanisms by deploying rogue VMs, as VPXUSER, directly onto hypervisors using SFTP to write files then executed them with /bin/vmx. By doing this, these rogue VMs were not discoverable via vCenter, the ESXi web interface, and even some on-hypervisor command-line utilities that query the API.

These rogue VMs contained the BRICKSTORM backdoor and persistence mechanisms, configured with dual network interfaces for communication with both the Internet/C2 and core administrative subnets within the prototyping network.

Leveraging the VPXUSER Account

Adversaries often can leverage the VPXUSER account to perform various administrative tasks, such as enumerating VMs, accessing configuration settings, and interacting with the underlying hypervisor infrastructure. Additionally, adversaries may deploy rogue VMs directly onto hypervisors to evade detection mechanisms and maintain persistence within the environment. Rogue VMs, which are created and operated without proper authorization and management by the hypervisor, provide adversaries with a stealthy foothold for conducting malicious activities. These VMs can bypass visibility controls within VMware management interfaces, making them difficult to detect and mitigate.

Detecting Rogue VMs

Safeguarding against rogue VMs and any ensuing persistence demands a vigilant approach. Simply using the hypervisor management interface to manage VMs is often insufficient and can be pointless when it comes to dealing with rogue VMs. This is because rogue VMs operate outside the standard management processes and do not adhere to established security policies, making them difficult to detect and manage through the GUI alone. Instead, one needs special tools or techniques to identify and mitigate the risks associated with rogue VMs effectively.

What to Look For:

1. Command-Line Usage: Utilize the following commands on an ESXi hypervisor to identify unregistered VMs:
   • vim-cmd vmsvc/getallvms
   • esxcli vm process list | grep Display
2. Comparison of VM Lists: Compare the output of vim-cmd (API-based VM check) with the list of running VMs obtained from esxcli. Differences in the list of VMs between the output of a vim-cmd (that will check for VMs via the API) and the list of running VMs that esxcli sees (which directly queries the host hypervisor) indicate a potential problem. A VM running on a hypervisor that is not seen via the registered VM data via API warrants further investigation as a possible unregistered/rogue VM.

Detecting VMware Persistence

To address the persistence of these rogue VMs, it is crucial to scrutinize the hypervisor’s startup scripts.

What to Look For:

1. Persistence Mechanism: Monitor for modification of the legitimate /etc/rc.local.d/local.sh file to include the following line:
   • /bin/vmx -x /vmfs/volumes/<REDACTED_VOLUME>/<REDACTED_VM_NAME>/<REDACTED_VM_NAME>.vmx 2>/dev/null 0>/dev/null &
2. Persistence Identification: Search for invocations of the /bin/vmx binary within /etc/rc.local.d/ or more specifically by manually reviewing the local.sh startup script with the following commands:
   • grep -r \/bin\/vmx /etc/rc.local.d/
   • cat /etc/rc.local.d/local.sh

The infiltration of MITRE’s network through VMware vulnerabilities underscores the need for heightened vigilance and advanced security measures in virtualized environments. As attackers continue to refine their techniques, organizations must evolve their defenses to protect against these sophisticated threats. By adopting comprehensive security practices, staying informed about emerging vulnerabilities, and fostering a culture of cybersecurity awareness, organizations can better defend against future intrusions.