Rethinking Logging for Critical Assets

Posted by Matt Suiche | Published on February 20, 2018

Categories: comae malware

Tags: stardust
Reading time: 5 minutes.

Table Of Contents:

Going beyond log files, accepting memory as its own format.

Logging is a common practice for IT and Security purposes. Mature organizations tend to have extensive and in-depth logging capabilities using either commercial or free solutions.

Although, logging is a powerful way to troubleshoot and investigate events it’s often limited by the initial input format of the logs during the collection process.

As the complexity of attacks increase, it’s almost natural for defensive capabilities to also evolve — particularly in the logging capabilities area. Seasoned attackers focus their operational efforts on advanced infiltration techniques but also on anti-forensics techniques to erase evidences of their visits as we have seen in the some of the files leaked by TheShadowBrokers last year.

The evolution of backdoors, such as in-memory kernel-mode backdoors like DOUBLEPULSAR, makes it even harder for traditional log capture and analysis softwares to understand when an attacker was in your network, and for how long.

“Your perimeter is not the boundary of your network it’s the boundary of your telemetry” — the grugq

Workflow

As you can see above, we designed a four-step compressive process for organizations willing to increase the health-check process of their critical assets by integrating memory logging/forensics to their capabilities and processes.

Identify

Lately, more and more incidents against financial institutions have been publicly reported in the media. As part of defining your perimeter of defense, you’ll identify what “critical assets” such as servers you want to increase your visibility on. Those critical assets can be production servers or even honeypots, that you’d tightly control.

Capture

Memory acquisition of a running system is a very simple & straightforward process which can be achieved quickly through our standalone utility, Comae DumpIt, downloadable for free on our platform at https://my.comae.com.

Comae DumpIt generates full memory Microsoft crash dumps, rather that legacy raw memory dumps, on the fly for interoperability and efficiency purposes. Furthermore, Comae DumpIt is standalone allowing the user more flexibility (PsExec, PsRemote etc.) in how to deploy the utility without having to deal with the friction of installing an agent or endpoint.

There are three main scenarios that we identified where memory acquisition is valuable:

Manually, after an incident. This scenario is the most common for memory forensics.
As part of orchestration. Many of our users leverage memory forensics to have a more comprehensive understanding of the alerts returned by their endpoints. This could be automated through playbooks to automatically perform a memory acquisition after a suspicious activity has been detected by a third-party endpoint.

Configuring a scheduled task for DumpIt.

Periodically through scheduled jobs. This scenario is the most interesting for increasing logging capabilities, as acquisition is triggered by periodical event (once a month, once a week, etc…) before archiving the memory copies locally (e.g. a file share) as you’d normally do with logs. Configurable directly through PowerShell or Windows’ Task Scheduler, this allows the user to keep periodical copies of their machines states to later enable possibilities such as retro-threat-hunting.$Action = New-ScheduledTaskAction -Execute C:\Comae-Toolkit-3.0.20180207.1\x64\DumpIt.exe -Argument ‘/Q /R’ -WorkingDirectory ‘C:\Logs’ $Trigger = New-ScheduledTaskTrigger -Weekly -At ‘9am’ -DaysOfWeek ‘Monday’ $Settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -StartWhenAvailable -Hidden Register-ScheduledTask -Action $Action -Trigger $Trigger -Settings $Settings -TaskName ‘DumpIt’ -User System -RunLevel Highest

Performance and scalability are vital metrics when it comes to memory acquisition of large servers.

Acquisition benchmark.

One of the main pain point reported from our users, before they adopting Comae DumpIt, was that several vendors have issues during memory acquisition for servers with more than 64GBs of RAM which prevented them to investigate large servers.

Attached are the results of our acquisition benchmark on traditional desktops and servers with up to half a terabyte of memory. As you can see, the acquisition time is significantly faster (less than 10 minutes) if our built-in compression flag is enabled on such servers.

Analyze

Machine view from Comae Stardust Platform.

Memory images management can easily be done locally using our PowerShell command line interface available on GitHub. The below command is a sample command that allows the users to pre-process several acquired images from a given folder into their Comae metadata snapshot format before uploading them to our analysis platform.

Get-ChildItem -Path C:\Logs -File | ForEach-Object { Convert-DumpFileToSnapshot -FilePath $_.FullName -Directory "C:\Snapshots" }

Get-ChildItem -Path C:\Snapshots -File | ForEach-Object { Send-ComaeSnapshot -Key "<APIKEY>" -Path $_.FullName -ItemType "File" }

Machines View

Infected machine