Now Is Not the Time for Changes – Don’t “Take the Opportunity” During an Ongoing Incident

Answer from Hasain

Based on your thousands of hours in incident response work, what would be a Red Flag for you? 

Altering or removing information, knowingly or unknowingly, from systems that might result in difficulties to analyze and understand what happened and what needs to be performed to revert what the threat actor did and finally restore the systems and operations. 

Performing an incident response engagement depends on the availability of information that can be analyzed to get as many details as possible about the breach. It is critical to have logs from systems, applications and other supporting systems such as network components, storage, IT service management, third-party systems and cloud services. 

Carrying out actions in the environment while troubleshooting will alter systems and, in many cases, have a negative impact on the availability of local logs especially when log is not shipped to logging systems and/or policies are not set to define retention times with incident response in mind. 

Performing seemingly classic troubleshooting steps like system restart, updates, installations of new tools or removal of components, without proper documentation, can also interfere with activities performed by the threat actor making it more difficult to understand what was performed by who. The same approach applies to modifications of accounts, especially changes in privileges and permissions on different systems as well as the introduction of new administrative accounts on compromised systems. 

Why is so important to work with an experienced CSIRT team during an incident? 

In the event of and incident a different set of skills is often required. The teams performing the different activities to investigate, respond and restore need to work close with each other, and even more important rely on each other to perform critical actions like containment and eradication once the complete attack timeline has been established.  

It is of huge importance to conduct a good forensic investigation to identify all or as much as possible of the threat actor activities in the environment. It is very important to understand where the attack started and how it progressed until the target goals were reached. It is also critical to understand how systems were modified and how did the threat actor maintain access and control to the environment, especially usage of different persistence techniques. 

Not all system can be restored from backup and many systems will need to undergo a cleaning process to remove the artifacts left by the threat actor. When systems been fully or partially encrypted it is important to understand what can be kept and not. Ultimately it is necessary to repair some systems that cannot be resorted, or it is extremely impractical to restore due to lost of valuable data. 

It is equally important to make sure that the internal IT organization is well prepared to start validation of systems once the incident response team has started to handover recovered systems. It is critical to have well tested plans on how to perform the different steps, so the internal IT organization is not getting overwhelmed and exhausted. 

Having an experienced CSIRT team helps reduce and resolve issues related to stress, availability of key personnel, effective planning of resources, as well as relying on proven experience rather than guessing and trial and errors approach. Training and simulation are a good approach but having a seasoned and experienced partner that could help, support, and do the heavy lifting often result in well invested resources and a huge relief for the organization with faster investigation, response, and recovery at the bottom line! 

If you compare a perfect incident to a failed incident, what would be the biggest difference? 

An IT environment with good incident response preparation means that most or all of the important artifacts are preserved and available and that this investigation can get started immediately to understand the chain of activities and what needs to be done to promptly respond to the threat and get back to operational mode. Putting proper monitoring on top of that should result in early disarm of threats meaning that the risk or business interruption is effectively minimized. 

Part of the suggested preparation is to also have all necessary agreements, none-disclosure, security vetting, and other necessary work preestablished to shorten the time needed to engage an externa incident response team whenever that is needed. 

To give an example, if forensic artifacts are not preserved, collections of artifacts would rely more on restoring from backup from multiple points in time to get enough resolutions and cover the whole timeline of an inciden. That means the forensic activities will need to wait for reocurring restore operations as well as dela with the uncertinity of the possible no availability of artifacts beacause of damaged or no backups of systems.  

A well done preparation also covers necessary facts about how critical systems are, dependencies between systems, and in what order these systems needs to be restored to effectivly resotre operations.