A Timeline Update as of 02/22/21

An update to our Newest Employee’s FIRST Technical Article
Another interim post-mortem review . . . .

A Note About Updates:

We have decided to leave the original article as it was originally posted and to update this post with any changes that have been made. You can see the original post here. The update below will contain the original article with any additional information that has been added on 02/22/21 noted by text like this.

There is an additional article on the SolarWinds hack and the Exchange Server Hack. Click here to go to the link at the end of this article.

     – Vigilize

Introduction:

As the managing partner of infotex, I am proud to introduce Tanvee Dhir’s “debut article.”  We wanted to start providing more “technical content.”  I told Tanvee, “write a technical article that I can understand.”  I understood most of this article!  Let us know what you think of Tanvee’s first article! 

     – Dan Hadaway, CRISC CISA CISM

High Level Summary

As we step into the second month since the discovery of one of the most sophisticated and persistent intrusion attacks aimed at U.S. government agencies, critical infrastructure entities, and recognized private sector organizations in North America, Europe, Asia, and the Middle East; several ongoing investigations from security researchers all over the cyber community have helped establish a detailed timeline of the SolarWinds Orion Security breach.

This highly intricate and professional attack, which appears to be conducted by suspected nation-state operators, looks like a well-planned long game, with indications of threat actors (TAs) assessing the SolarWinds internal network starting as far back as September 2019. FireEye, a managed security service provider using SolarWinds, uncovered this widespread campaign during its own breach investigation.

The actors behind this attack gained access to numerous private and public networks via distributing a legitimately signed trojanized update to the SolarWinds Orion network management and monitoring software. The Cybersecurity and Infrastructure Agency (CISA), in their updated alert, stated that there is evidence indicating other initial access vectors other than the SolarWinds Orion platform, which could have been used by the TAs to accomplish their goals. This attack campaign is depicted to be the work of highly knowledgeable and sophisticated advisories and was conducted with high operational security.

The timeline continues to be “updated” as more and more information is reported, so this is the first* of a series of articles designed to keep our Clients up to date on the SolarWinds timeline.

Who was affected?

As the extended scope of the attack and final count of compromised devices and networks are still under analysis, the list of victims (either Phase 1 or 2) keeps increasing gradually. Phase 1 of the attack accounts for all the victims that were infected with the malicious update of the Orion software, while Phase 2 victims are the more focused upon targets by the TAs as it involved more of their real hands-on activity. Truesec, in their blog, explains the detailed functioning of how they were able to identify a partial list (still updating) of breached organizations and differentiating the ones accessed for Phase 2 from the ones terminated during Phase 1. Kaspersky released a report analyzing the industrial organizations that used the backdoored SolarWinds version (Phase 1 victims) and distinguished it based on different sectors.

Close to 200 organizations were affected, including: SolarWinds, the U.S. Commerce and Treasury departments, the Department of Homeland Security, the National Institutes of Health, and the State Department.  Microsoft and more than 40 of their IT customers were also affected.  The scope of the attack also included technology firms like CrowdStrike, Malwarebytes, Cisco Systems, Deloitte, Nvidia, VMware, Belkin, Intel, Mimecast, Palo Alto, and Fidelis.  And, of course, there are many other undisclosed entities affected by this attack, highlighting the rise of “Supply Chain Risk.”

As of Feb 17, 2021, in a statement by Anne Neurberger, who is leading the U.S. response to the SolarWinds attack, nine federal agencies and about 100 private-sector companies were compromised.

The Supply Chain Compromise

SolarWinds offers several different applications but, so far, the application in question is SolarWinds Orion. Being a network management suite, Orion requires necessary visibility into the network and the pervasive privileges make it a sweet target for adversaries. According to the ongoing investigation by SolarWinds, indicators of compromise go back to September 2019; code modification by the TAs can be seen as early as October 2019. In that version of the Orion Platform release, there appears to be modifications designed to test the perpetrators’ ability to insert code into their builds. Still, the first weaponized software update was not released until March 2020.

Attackers used a malicious tool SUNSPOT to hijack the build process of the Orion software in order to insert a malicious version of the binary SolarWinds.orion.core.business.dll into the update packages. A Dynamic Link Library (DLL) is a shared library of resources and executable code which is loaded when an application is executed.  The update containing the component was digitally signed and released to their 18,000 customers by SolarWinds.

This malicious DLL component contains a backdoor termed ‘SUNBURST’, which is activated when a trusted task SolarWinds.businesslayerhost.exe is executed. After an initial dormant period, the malware retrieves its functionality to communicate via HTTP to third party servers and uses multiple techniques to mask its network traffic as legitimate traffic from the Orion software all while staying undetected by the eyes of Security analysts. Please see Fig. 1, which illustrates the working of how the backdoor triggers and communicates with the adversaries.

The backdoor connects to a Command-and-Control (C2) server, whose domain is computed using a domain generation algorithm (DGA) and is related to the internal domain name of the organization (compromised victim) it is acting from. The response from the server directed the backdoor as to what task to perform along with possible hands-on keyboard activity capabilities to the hacker. There are various functions defined within the backdoor which tell the backdoor to collect and transfer system information, create/delete files and processes, reboot system, set process privileges, and most importantly to terminate the malicious DLL component without interrupting the parent SolarWinds process if any of the hard coded services were found on the system. CISA released a detailed Malware Analysis Report (MAR) for the malicious artifacts involved in the attack which describes the functions in detail.

Figure 1: Malware Infection chain observed in compromised systems (Courtesy: Microsoft)

Technical Timeline

New discoveries related to this highly operational attack are being made every day, because of the unceasing analysis being put in by professionals all over the industry. In order to further illustrate the sophistication and events of the attack, we created a timeline (See Fig. 2) interpreting the major post-mortem discoveries from various ongoing investigations.

As depicted by the SolarWinds investigation, evidence of TAs accessing their network dates back to September 2019, and depicts them performing a trial run through in November 2019 on the subsequent October 2019 version of the Orion Platform release, which appears to have contained modifications designed to test the perpetrators’ ability to insert code into their builds.

After their supposedly successful trial run, TAs started building their infrastructure by obtaining DGA domain ‘[DGA].avsvmcloud.com’ which was found to be the server (found through common subdomain ‘.avsvmcloud.com’) with which most of the victim machines made their initial communication. While these communications were held without any significant activity, there were instances where the initial communication was followed by network traffic on port 443 with other visibly legitimate C2 domains.

An updated version of the malicious code injection source (SUNSPOT) that inserted the SUNBURST malicious code into the Orion Platform was compiled in February, and the software update was released to the SolarWinds customers in March 2020.

The backdoor implements sophisticated functionality to communicate with the TA infrastructure and applies logic to determine what actions should be taken.  What is also frightening, as well as revealing, is the level of sophistication of this attack:  if a certain set of services and processes were found active in the system, the backdoor terminates making sure it wasn’t boobytrapped into a security researcher’s system.

Microsoft, in their investigations, believes that there is a possibility of real hands-on-keyboard activity, performed around May 2020, where TAs spent time selecting specific victims and designing unique Cobalt Strike implants for their target machines. Cobalt Strike is a software generally used for red team operations, to test their networks by replicating the techniques and tactics of an advanced adversary. It comes with a toolkit for developing shellcode loaders which also makes it popular with malicious actors, as they can customize their implants (also known as beacon). Once TAs had reached a sufficient number of targets and an acceptable foothold on their Cobalt Strike implants, they went on to remove the backdoor-generation function from the SolarWinds VM in June 2020.

The finesse of the attack can be realized by looking at how it progresses to Phase 2, which Microsoft explains in detail in their report. The attacker’s infrastructure had control over multiple C2 domains, which are used to control and communicate to the backdoor and the Cobalt strike implants. These Phase 2 implants have been identified as TEARDROP, Raindrop, and other custom Cobalt strike loaders which present a different structure but are generated from the same kit; ultimately loading a Beacon Reflective loader which loads a DLL into a process memory without using a Windows loader. Following that, the attackers continued with their hands-on keyboard activities shifting their focus onto Phase 2 targets.

Jumping to early December 2020, FireEye discovered and unveiled in their SEC filing that their network was broken into and the company’s Red Team penetration testing tools were stolen. They shared concerns about the potential use of stolen penetration testing tools to attack additional companies. They were able to detect this intrusion by detecting a simple anomaly in one of their employee’s additionally registered 2FA device.  Although there has been no evidence to date indicating the tools have been used by the attacker, FireEye continues to monitor closely for clues.

Upon further internal investigation, FireEye informed SolarWinds about the breach of the Orion software which also triggered an emergency NSC (National Security Council) meeting at the White House to discuss the potential breach of multiple government agencies and businesses.

In addition to the investigation and emergency directives released by CISA, a collaborated effort by the researchers of Fireye, GoDaddy, and Microsoft are working to stop this from further spreading; a Killswitch is deployed that would affect new and previous SUNBURST infections by disabling SUNBURST deployments that are still beaconing to avsvmcloud[.]com (identified malicious domain name).

Around the end of December, Microsoft also released a statement revealing that their source code was viewed by the attackers, but they were unable to access or modify anything which also led them to release a deep-dive investigation report on the attack.

Entering 2021, where we are left with remediating the attack and evaluating its wide scope, the White House instituted a probe and appointed Anne Neuberger to oversee the U.S response to the attack and a deep inspection of the attack is performed by various organizations.

In the midst of the investigation, in early February, a separate intrusion is discovered in the SolarWinds network which is suspected to be the work of a supposedly Chinese hacker group other than the already pointed out nation-state group.

In their official response to the incident, Microsoft disputes the allegations on being pointed out as one of the initial vector for the attack, and said that even though the data hosted in Microsoft services was a target in some instances, there is no evidence of Microsoft services being the initial entry point to the compromised networks. Although they did reveal in their final update to the investigation posted late February, that there were some cases encountered where the TAs were able to download source code for a small subset of Azure, Intune and Exchange components.

The Senate Intelligence Committee has scheduled a hearing on February 23rd where the officials from SolarWinds, FireEye, Microsoft, CrowdStrike will be giving their testimonies in accordance with the hack. This will help the federal government understand the breach more closely and devise an appropriate response plan.

Assessment/Evaluation

There have been several reports of the attack involving the use of additional techniques to compromise other services and software to obtain information and achieve the final goals. TAs have been suspected to exploit vulnerabilities and bypass MFAs in VMware, Duo, Microsoft 365 Exchange web services, and Mimecast, among others.

It is highly likely that a large amount of confidential information was accessed/viewed by adversaries for a certain period of time.

As an additional evaluation into other SolarWinds products, Trustwave in their SpiderLabs blog recently published discoveries and proof of concepts (POCs) for three new critical vulnerabilities which they found in other SolarWinds products. These vulnerabilities were not exploited, and patched in a timely manner by SolarWinds. Additionally, Solarwinds faces a class action lawsuit and insured loss claims totaling nearly $90 million.

CISA, SolarWinds, CrowdStrike, Microsoft, FireEye, Palo Alto, Volexity, and researchers from other enterprises are working to progressively release detailed analysis of the attack and the various vectors involved. Microsoft, in fact, created a team of 500 researchers to reverse engineer the details of the attack and in a recent interview revealed that the attack seems to have a fingerprints of possibly more than 1000 software developers.

Mitigation/ Measures

In response to this incident, CISA issued an emergency directive to disconnect the compromised devices from the network and await their updates. Organizations need to focus on solutions that not only monitor their network externally but are also equipped to detect ongoing attacks inside their network.

In the words of Dan Hadaway, our managing partner, “FireEye, a competitor of ours, really stepped up to the ‘Good Response Plate’ on this one. Imagine the damage had they not been so transparent.” FireEye even released their red team tools(lots of intellectual property) which the SolarWinds breach originally stole.

A cybersecurity firm, SentinelOne, released a free SUNBURST identification tool to help enterprises determine attack readiness. This open-source assessment tool allows users to identify if the SUNBURST malware variant at the heart of the SolarWinds attack campaign would have infected their devices.

If you have not already reached out to your SOC or MSSP, you should be doing that. Your security monitoring process should be reviewing responses like this one from infotex.

As an enterprise to be protected from these attacks, these directives can be followed:

• Implement a SIEM with an additional layer of detection focusing on IOBs (Indicator of Behaviors) of certain processes and files along with the known IOCs (Indicator of Compromise).
• Use up-to-date signatures released by FireEye to identify related activity.
• Check for traffic related to the identified malicious domains in your network.
• Ensure all the secret keys associated with MFA and passwords are reset following a breach.
• Integrate appropriate security measures at each stage of the Software Development Cycle.

This incident also establishes the fact that protecting an enterprise starts with each individual and those who are responsible for writing and deploying software need to take special measures in terms of code integrity.

This was not the first Supply-chain compromise and the pattern of such attacks will continue and security teams need to be more geared up than ever to tackle them. The challenges faced in detecting and analyzing these kinds of incidents prepare the cybersecurity industry for the next generation of cyberattacks we might face in the future.

A follow-up article to this can be found here: SolarWinds Update and Exchange Server Hack

References

• https://www.channele2e.com/technology/security/SolarWinds-orion-breach-hacking-incident-timeline-and-updated-details/
• https://orangematter.SolarWinds.com/2021/01/11/new-findings-from-our-investigation-of-sunburst/
• https://www.microsoft.com/security/blog/2020/12/18/analyzing-solorigate-the-compromised-dll-file-that-started-a-sophisticated-cyberattack-and-how-microsoft-defender-helps-protect/
• https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-SolarWinds-supply-chain-compromises-with-sunburst-backdoor.html
• https://www.volexity.com/blog/2020/12/14/dark-halo-leverages-SolarWinds-compromise-to-breach-organizations/
• https://unit42.paloaltonetworks.com/solarstorm-supply-chain-attack-timeline/
• https://www.zdnet.com/article/four-security-vendors-disclose-SolarWinds-related-incidents/
• https://www.zdnet.com/article/partial-lists-of-organizations-infected-with-sunburst-malware-released-online/
• https://blog.truesec.com/2020/12/17/the-SolarWinds-orion-sunburst-supply-chain-attack/
• https://krebsonsecurity.com/2020/12/malicious-domain-in-SolarWinds-hack-turned-into-killswitch/

Original article by Tanvee Dhir, CEH. Data Security Analyst, infotex