Autopsy of the SolarWinds Hack

A Timeline as of 01/24/2021

Our Newest Employee’s FIRST Technical Article
Another interim post-mortem review . . . .

A Note About Updates:

We are leaving this article as is, but for any updates to the timeline, check the Autopsy of the SolarWinds Hack Timeline Update article!

     – Vigilize

Introduction:

As the managing partner of infotex, I am proud to introduce Tanvee Dhir’s “debut article.”  We wanted to start providing more “technical content.”  I told Tanvee, “write a technical article that I can understand.”  I understood most of this article!  Let us know what you think of Tanvee’s first article! 

     – Dan Hadaway, CRISC CISA CISM

High Level Summary

As we step into the second month since the discovery of one of the most sophisticated and persistent intrusion attacks aimed at U.S. government agencies, critical infrastructure entities, and recognized private sector organizations in North America, Europe, Asia, and the Middle East; several ongoing investigations from security researchers all over the cyber community have helped establish a detailed timeline of the SolarWinds Orion Security breach.

This highly intricate and professional attack, which appears to be conducted by suspected nation-state operators, looks like a well-planned long game, with indications of threat actors (TAs) assessing the SolarWinds internal network starting as far back as September 2019. FireEye, a managed security service provider using SolarWinds, uncovered this widespread campaign during its own breach investigation.

The actors behind this attack gained access to numerous private and public networks via distributing a legitimately signed trojanized update to the SolarWinds Orion network management and monitoring software. The Cybersecurity Infrastructure and Security Agency (CISA), in their updated alert, stated that there is evidence indicating other initial access vectors other than the SolarWinds Orion platform, which could have been used by the TAs to accomplish their goals. This attack campaign is depicted to be the work of highly knowledgeable and sophisticated advisories and was conducted with high operational security.

The timeline continues to be “updated” as more and more information is reported, so this is the first* of a series of articles designed to keep our Clients up to date on the SolarWinds timeline.

Who was affected?

As the extended scope of the attack and final count of compromised devices and networks are still under analysis, the list of victims (either Phase 1 or 2) keeps increasing gradually. Phase 1 of the attack accounts for all the victims that were infected with the malicious update of the Orion software, while Phase 2 victims are the more focused upon targets by the TAs as it involved more of their real hands-on activity.

Close to 200 organizations were affected, including: SolarWinds, the U.S. Commerce and Treasury departments, the Department of Homeland Security, the National Institutes of Health, and the State Department.  Microsoft and more than 40 of their IT customers were also affected.  The scope of the attack also included technology firms like CrowdStrike, Malwarebytes, Cisco Systems, Deloitte, Nvidia, VMware, Belkin, Intel, Mimecast, Palo Alto, and Fidelis.  And, of course, there are many other undisclosed entities affected by this attack, highlighting the rise of “Supply Chain Risk.”

The Supply Chain Compromise

SolarWinds offers several different applications but, so far, the application in question is SolarWinds Orion. Being a network management suite, Orion requires necessary visibility into the network and the pervasive privileges make it a sweet target for adversaries. According to the ongoing investigation by SolarWinds, indicators of compromise go back to September 2019; code modification by the TAs can be seen as early as October 2019.  In that version of the Orion Platform release, there appears to be modifications designed to test the perpetrators’ ability to insert code into their builds. Still, the first weaponized software update was not released until March 2020.

Please see Fig. 1, which illustrates the working of how the backdoor triggers and communicates with the adversaries. Attackers compromised the software build process for the Orion software and added a malicious version of the binary SolarWinds.orion.core.business.dll component which was further digitally signed by SolarWinds before releasing the update to their 18,000 customers. This malicious component contains a backdoor termed “SUNBURST,” which is activated when a trusted task ‘SolarWinds.businesslayerhost.exe’ is executed. After an initial dormant period, the malware retrieves its functionality to communicate via HTTP to third party servers and uses multiple techniques to mask its network traffic as legitimate traffic from the Orion software all while staying undetected by the eyes of Security analysts.

The backdoor connects to a Command-and-Control (C2) server, whose domain is computed using a domain generation algorithm (DGA), and is related to the internal domain name of the organization (compromised victim) it is acting from. The response from the server directed the backdoor as to what task to perform along with hands-on keyboard activity capabilities to the hacker.

Figure 1: Malware Infection chain observed in compromised systems (Courtesy: Microsoft)

Technical Timeline

New discoveries related to this highly operational attack are being made every day, because of the unceasing analysis being put in by professionals all over the industry. In order to further illustrate the sophistication and events of the attack, we created a timeline (See Fig. 2) interpreting the major post-mortem discoveries from various ongoing investigations.

As depicted by the SolarWinds investigation, evidence of TAs accessing their network dates back to September 2019, and depicts them performing a trial run through in November 2019 on the subsequent October 2019 version of the Orion Platform release, which appears to have contained modifications designed to test the perpetrators’ ability to insert code into their builds.

After their supposedly successful trial run, TAs started building their infrastructure by obtaining DGA domain ‘[DGA].avsvmcloud.com’ which was found to be the server (found through common subdomain ‘.avsvmcloud.com’) with which most of the victim machines made their initial communication. While these communications were held without any significant activity, there were instances where the initial communication was followed by network traffic on port 443 with other visibly legitimate C2 domains.

An updated version of the malicious code injection source that inserted the SUNBURST malicious code into the Orion Platform was compiled in February, and the software update was released to the SolarWinds customers in March 2020.

The backdoor implements sophisticated functionality to communicate with the TA infrastructure and applies logic to determine what actions should be taken.  What is also frightening, as well as revealing, is the level of sophistication of this attack:  if a certain set of services and processes were found active in the system, the backdoor terminates making sure it wasn’t boobytrapped into a security researcher’s system.

Microsoft, in their investigations, believes that there is a possibility of real hands-on-keyboard activity, performed around May 2020, where TAs spent time selecting specific victims and designing unique Cobalt Strike implants for their target machines. Cobalt Strike is a software generally used for red team operations, to test their networks by replicating the techniques and tactics of an advanced adversary. It comes with a toolkit for developing shellcode loaders which also makes it popular with malicious actors, as they can customize their implants (also known as beacon). Once TAs had reached a sufficient number of targets and an acceptable foothold on their Cobalt Strike implants, they went on to remove the backdoor-generation function from the SolarWinds VM in June 2020.

The finesse of the attack can be realized by looking at how it progresses to Phase 2, which Microsoft explains in detail in their report. The attacker’s infrastructure had control over multiple C2 domains, which are used to control and communicate to the backdoor and the Cobalt strike implants. These Phase 2 implants have been identified as TEARDROP, Raindrop, and other custom Cobalt strike loaders which present a different structure, but are generated from the same kit and ultimately load a Beacon Reflective loader which loads a DLL into a process memory without using a Windows loader. Following that, the attackers continued with their hands-on keyboard activities shifting their focus onto Phase 2 targets.

Jumping to early December 2020, FireEye discovered and unveiled in their SEC filing that their network was broken into and the company’s Red Team penetration testing tools were stolen. They shared concerns about the potential use of stolen penetration testing tools to attack additional companies. Although there has been no evidence to date indicating the tools have been used by the attacker, FireEye still continues to monitor closely for clues. Upon further internal investigation, FireEye informed SolarWinds about the breach of the Orion software which also triggered an emergency NSC (National Security Council) meeting at the White House to discuss the potential breach of multiple government agencies and businesses.

In addition to the investigation and emergency directives released by CISA, a collaborated effort by the researchers of Fireye, GoDaddy, and Microsoft are working to stop this from further spreading; a Killswitch is deployed that would affect new and previous SUNBURST infections by disabling SUNBURST deployments that are still beaconing to avsvmcloud[.]com (identified malicious domain name).

Around the end of December, Microsoft also released a statement revealing that their source code was viewed by the attackers, but they were unable to access or modify anything which also led them to release a deep-dive investigation report on the attack.

Assessment/Evaluation

There have been several reports of the attack involving the use of additional techniques to compromise other services and software to obtain information and achieve the final goals. TAs have been suspected to exploit vulnerabilities and bypass MFAs in VMware, Duo, Microsoft 365 Exchange web services, and Mimecast, among others.

It is highly likely that a large amount of confidential information was accessed/viewed by adversaries for a certain period of time. Truesec, in their blog, explains the detailed functioning of how they were able to identify a partial list (still updating) of breached organizations and differentiating the ones accessed for Phase 2 from the ones terminated during Phase 1.

SolarWinds, Microsoft, FireEye, Palo Alto, Volexity, and researchers from other enterprises, have released a detailed analysis of the attack and the various vectors involved.

Mitigation/ Measures

In response to this incident, CISA issued an emergency directive to disconnect the compromised devices from the network and await their updates. Organizations need to focus on solutions that not only monitor their network externally, but is also equipped to detect ongoing attacks inside their network.

In the words of Dan Hadaway, our managing partner, “FireEye, a competitor of ours, really stepped up to the ‘Good Response Plate’ on this one. Imagine the damage had they not been so transparent.” FireEye even released the red team kit (lots of intellectual property) which the SolarWinds breach originally stole.

A cybersecurity firm, SentinelOne, released a free SUNBURST identification tool to help enterprises determine attack readiness. This open-source assessment tool allows users to identify if the SUNBURST malware variant at the heart of the SolarWinds attack campaign would have infected their devices.

If you have not already reached out to your SOC or MSSP, you should be doing that. Your security monitoring process should be reviewing responses like this one from infotex.

As an enterprise to be protected from these attacks, these directives can be followed:

• Implement a SIEM with an additional layer of detection focusing on IOBs (Indicator of Behaviors) of certain processes and files along with the known IOCs (Indicator of Compromise).
• Use up-to-date signatures released by FireEye to identify related activity.
• Check for traffic related to the identified malicious domains in your network.
• Ensure all the secret keys associated with MFA and passwords are reset following a breach.
• Integrate appropriate security measures at each stage of the Software Development Cycle.

This incident also establishes the fact that protecting an enterprise starts with each individual and those who are responsible for writing and deploying software need to take special measures in terms of code integrity.

This was not the first Supply-chain compromise and the pattern of such attacks will continue and security teams need to be more geared up than ever to tackle them. The challenges faced in detecting and analyzing these kinds of incidents prepare the cybersecurity industry for the next generation of cyberattacks we might face in the future.

References

• https://www.channele2e.com/technology/security/SolarWinds-orion-breach-hacking-incident-timeline-and-updated-details/
• https://orangematter.SolarWinds.com/2021/01/11/new-findings-from-our-investigation-of-sunburst/
• https://www.microsoft.com/security/blog/2020/12/18/analyzing-solorigate-the-compromised-dll-file-that-started-a-sophisticated-cyberattack-and-how-microsoft-defender-helps-protect/
• https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-SolarWinds-supply-chain-compromises-with-sunburst-backdoor.html
• https://www.volexity.com/blog/2020/12/14/dark-halo-leverages-SolarWinds-compromise-to-breach-organizations/
• https://unit42.paloaltonetworks.com/solarstorm-supply-chain-attack-timeline/
• https://www.zdnet.com/article/four-security-vendors-disclose-SolarWinds-related-incidents/
• https://www.zdnet.com/article/partial-lists-of-organizations-infected-with-sunburst-malware-released-online/
• https://blog.truesec.com/2020/12/17/the-SolarWinds-orion-sunburst-supply-chain-attack/
• https://krebsonsecurity.com/2020/12/malicious-domain-in-SolarWinds-hack-turned-into-killswitch/

Original article by Tanvee Dhir,CEH. Data Security Analyst, infotex