Incident response

Security Incident Response Testing To Meet Audit Requirements

November 22, 2010 by Russ McRee

Description: Practical guidance and tools to ensure maximum readiness for incident response teams including drill tactics. PCI-DSS audits often require IR testing validation; drill quarterly and be ready next audit cycle.

Introduction

Incident response teams in enterprise environments beholden to regulatory requirements can conduct drills that will help satisfy auditors and keep their incident handlers sharp.

A quick search through the latest PCI DSS, version 2.0, for the term “incident response” will reveal a number of requirements and testing procedures; following is a summary of those requirements.

In particular, 12.9 states “implement an incident response plan. Be prepared to respond immediately to a system breach.” Subsections of this requirement dive deeper, including:

  • 12.9.1 discusses an incident response plan inclusive of specific procedures.
  • 12.9.2 requires testing the plan annually (I suggest quarterly, more on this below)
  • 12.9.3 requires 24/7 personnel coverage to respond
  • 12.9.4 indicates the need for appropriate training
  • 12.9.5 specifies the need for monitoring alerts from IDS/IPS/FIMS
  • 12.9.6 stresses modification and evolution of your response plan base on lessons learned

I propose a practice by which you can satisfy the testing procedures for the vast majority of PCI DSS 12.9, sustainable under audit, and useful for additional compliance requirements such as ISO 27001 and SAS70.

We’ll make some assumptions as minimum requirements to achieve testing success:

  • You have a documented incident response plan, ideally a fully realized standard operating procedure (SOP) that you keep current. To really make auditors happy, keep it versioned, update it at least biannually, and map it to NIST SP800-61 as much as possible; following standards strengthens your plan and procedures under scrutiny.
  • You have defined incident response personnel and they are available to respond 24/7.
  • You have monitoring mechanisms in place; IDS, IPS, FIMs or all of the above. You already receive alerts from these systems as part regular business.
  • A moderate level of technical incident response capability on your IR team. If they’re in house personnel this process is easier, but you can (must) require of it your vendor/consultant if you outsource your incident response function

First, while the PCI DSS states that an annual test is sufficient, I say step up your game. The more your drill, the better your team’s incident response chops will be. Quarterly testing gives you an opportunity to improve between audit cycles should you have identified deficiencies.

Second, as part of your drill, be prepared to fully document and report the process; ensure the report document is concisely written and includes methods, timelines, participants, and findings. Telling an auditor you conduct incident response drills without supporting documentation won’t get you very far.

I suggest three mandatory roles, and an optional observer, for incident response drills:

  • The moderator is usually the incident response team lead.
  • The coordinator is usually the target system owner.
  • The testers are your incident handlers.
  • The observer typically rides shotgun with the tester.

Include these roles and their individual findings or summaries in your report. See Reporting below for more guidelines.

Incident Response Testing

Now, with pre-testing guidelines established, how can we conduct a drill that helps validate your capabilities as defined by PCI DSS 12.9?

Establish your “target” system(s) in advance. Depending on the size of your team, one or two hosts should suffice. For this article we’ll operate with one target system.

The target system should be “in scope” for PCI, but not mission critical or in active rotation. It should definitely NOT be available to external traffic; I recommend the likes of a file server, or standby host such as a passive cluster participant or a host out of production rotation for patching/maintenance.

Clearly you’ll need to work with the system owner, who should be identified in your after action report as the coordinator. You can decide together if you want to conduct a blind test or use a scheduled test window; you’ll likely make this decision based on coverage and capability.

Once you (the moderator) and the coordinator have selected targets and the window of operation, you have to “compromise” the target host in preparation for escalation of the test incident to your handlers. Undertake this process just prior to the test event window if scheduled in advance, or in close cooperation with the coordinator for a blind test.

NOTE: You assume all the risks associated with introducing what is identified by most anti-virus providers as a Trojan backdoor to a host for incident response testing. Remember to disable and remove it upon conclusion of your test! InfoSec Resources and the author assume no responsibility.

I’ve used a many different tactics during the “compromise” phase; it’s important to mix it up when you test quarterly in order to keep your handlers on their toes. You can combine any number of these ideas or build your own variations as long as they help validate your capabilities specific to the PCI or other requirement. Remember that you’re playing with binaries that will likely cause antivirus to fire. You can choose to add temporary exclusions for the files and directories you are manipulating, or you can let the alerts fire, then allow the activity via the AV client, particularly if the alerts are monitored by your incident responders (more evidence for your after-action report).

In addition to planting a binary on a system, you can opt to do so in a manner that should trigger an alert from your FIMS solution, if in use. Additionally, you can configure your “backdoor” to set off IDS/IPS alerts, ultimately testing PCI DSS 12.9.5 all in one fell swoop.

Netcat, Cryptcat, and Tini, the 3kb backdoor from NTSecurity, can all be utilized in a variety of ways but I favor Tini. Tini is a true backdoor (no other evil included) that listens on TCP port 7777 by default; you can edit the port value in a hex editor to a port of your choosing. To do so, open Tini in your favorite hex editor, search for 1E61, the hex value for decimal 7777, replace it with a value of your choosing (see Figure 1), and save it under a new temporary file name.


Image: Edit Tini port hex value

Here’s where to unify efforts.

First, if you know that a test network is monitored by IDS/IPS, choose a port for Tini that will likely be flagged; as an example, hex 56CE for decimal 22222 can result in an alert generated by one of Snort’s backdoor.rules when you connect to your Tini instance (see Figure 3 below).

Second, consider saving your updated version of Tini named for a known good system file, for two reasons.

  1. A system that is being monitored to ensure integrity by a FIMS installation will likely flag the activity if, as an example, you’ve renamed your new version of Tini to csrss.exe and tried to copy it to C:WindowsSystem32, assuming FIMS is enabled to monitor csrss.exe or the System32 directory. Said alert might look something like Figure 2. Depending on the OS this activity may also be prevented by the likes of TrustedInstaller as well.

    image: FIMS alert

  2. Mimicking the behavior of many malware samples, hide your Tini instance (now csrss.exe) on the target system somewhere that may not be prevented but that may challenge your handlers when its location is not immediately obvious. Tucking it in the likes of C:WindowsSysWOW64WindowsPowerShellv1.0 will require your handlers to conduct process, network stack, MAC times, and other analysis to find the planted backdoor.

You can take further optional steps to make the test more interesting and challenging by turning your renamed version of Tini into a service. Setting it to start automatically ensures that it runs even if the target system is rebooted, and a service named for or similarly to a known good system process/service may throw your handlers off the trail.

Following the naming convention we’ve established above, create said service as follows:

sc.exe create RuntimeSubsystem binPath= “C:WindowsSysWOW64WindowsPowerShellv1.0csrss.exe -k runservice” DisplayName= “Runtime Subsystem”

Just remember, clean up after yourself when the exercise is complete:

sc.exe delete RuntimeSubsystem should result in [SC] DeleteService SUCCESS

Then confirm:

sc.exe query RuntimeSubsystem should result in [SC] EnumQueryServicesStatus:OpenService FAILED 1060:

The specified service does not exist as an installed service.

Just prior to the beginning of the test period, connect to the backdoor. You can leave a hanging connection if you wish to leave a network stack bread crumb trail for your handlers to follow.

Said connection should lead to the above mentioned IDS/IPS alert, depending on your configuration, as seen in Figure 3.

Image: Connection to “backdoor” leads to alert.

If you’re using IDS/IPS to monitor/block egress traffic from in-scope network, making an additional outbound connection attempt from your target server to an external host over an interesting service port such as 22 should trigger an alert as well.

At this point, you should have the coordinator escalate the “compromise” per your incident response plan. In an enterprise with mature monitoring and incident response, an escalation may not even be necessary; your “malicious” activity should set the incident response wheels in motion. If you’re not going for a blind test, consider declaring the fact that it is a test in the escalation email or call, particularly in larger organizations with tiered support and response models.

Allow your handlers no more than a two hour window to get to the “root cause” of the test incident.

If you’re interested in some of the security incident tooling and methodology I’ve utilized and prescribe for others, feel free to reach out to me via email.

Once your handlers have succeeded in identifying your planted backdoor it is vital that they remove it as part of the response process and document having done so. You, as the moderator, are obligated to confirm that the system has been cleaned up and returned to a pre-test state.

If your handlers fail to identify the backdoor for some reason, you are further obligated to ensure a complete cleanup, and document your process for planting the backdoor, as well as lessons learned derived from how your handlers failed to identify it.

Reporting

Once your test exercise is complete, you’ll definitely want to button it up with a clean, legible, audit-worthy report. Figure 4 gives you the basic gist of what you may wish to include. Keep in mind that, per PCI DSS 12.9.6, auditors love to hear about the post-mortem meeting you had to improve the process and make use of lessons learned.

Again, feel free to reach out to me via email for additional reporting guidelines.

Summary

There are so many good reasons to test your security incident response capability, compliance and audit requirement should be but one. Like most other activities, practice makes perfect. While perfection in incident response is unlikely, the experience acquired and lessons learned from regular testing speak for themselves. Go forth and test your incident response plan!

Posted: November 22, 2010
Articles Author
Russ McRee
View Profile

Russ McRee is a senior security analyst, researcher, and founder of holisticinfosec.org, where he advocates a holistic approach to the practice of information assurance. He is also a security researcher for InfoSec Institute. His predominant focuses are incident response and web application security; he does both as team leader of Microsoft Online Service’s Security Incident Management team. Russ speaks and writes frequently on information security topics; including toolsmith, a monthly column for the ISSA Journal. IBM's ISS X-Force cited him as the 6th ranked Top Vulnerability Discoverers of 2009.

18 responses to “Security Incident Response Testing To Meet Audit Requirements”

  1. Uma Mahesh says:

    Hi,
    Can you please provide the Audit Test Plan document or template for the screenshot above. Your article is good and it gave insight but would like to have a clear view over the Audit Procedure. Please send me over mail to umamahesh@ymail.com or my corporate mail provided

    Thank you very much in advance

  2. Raul Ramos says:

    Dr. Mr. McRee,

    Your methodology is very good and easy to understand for a non-IT like me. I was assigned to oversee the PCI compliance of University of British Columbia (Vancouver, Canada). One of the internal audit action items is to test the IRP but I do now know where to start. I will try to follow your guidelines.

    Is it possible too to email me the Audit Test Plan document or template?

    Sincerely,

    Raul Ramos
    rramos@finance.ubc.ca

  3. Semeh says:

    Hi,

    Excellent article 🙂
    Can you please provide me with the the Audit Test Plan document template ?
    email : semeharbi@gmail.com

    Best regards,
    Semeh

  4. Nhan says:

    Very interesting article. I am implementing the PCI DSS compliance at Monster Cable and this gave me great insight into the Incident Response testing approach. It would be great if you can help share the Test Plan Procedure with me.
    nphan@monstercable.com

    Thanks a million!
    Nhan

  5. Lino says:

    Hi,

    Very good for learning.

    Can you please also send me the the Audit Test Plan document template ?

    Best regards,
    Lino

  6. Russ says:

    Lino, reply to russ at holisticinfosec dot org with your email address so I can send the template to you.

  7. Lori says:

    Hi Russ,
    Excellent article. Could I also trouble you for a copy of the Audit Test Plan document template?

    Thanks in advance,

    Lori

  8. sunil says:

    Really grate article ,Could you pls share the audit template? as we are in the process of PCI DSS re-certification audit.
    Thanks
    Sunil

  9. Isaac Nolan says:

    Fantastic article, Could you please send me the Incident Response Test Report template that you have shown above, it would be most appreciated 🙂
    inolan@hotmail.com.au

    Regards

    Isaac

  10. Nguyen Hoang Phuong says:

    Your guidance in this article is great and helpful. We are also in process of PCI DSS certification audit. Can you pls share with me the audit template as you shown in the screenshot. Email: nguyenhoangphuong.vn@gmail.com
    Thanks
    Phuong

  11. Nguyen Hoang Phuong says:

    I posted comments but it disappeared, so I posted it again. Your article is great and very helpful. We are also in process of PCI DSS certification audit. So could you pls share with me your audit template that shown in the screenshot.
    Thanks a lot
    Phuong Nguyen Hoang

  12. Jo says:

    Hi Russ

    Could you please email me the Incident Response Audit Test Plan document template ?

    Regards
    Jo

  13. Dennis Nwaigbo says:

    I really like your in-depth knowledge of IR testing. Please, I would like to get your template to help me design my test plan.

  14. Dennis Nwaigbo says:

    Hi Russ –
    Can you kindly share your IR template to help me design my test plan? I would also appreciate any suggestions for various test scenarios that will be acceptable in order to satisfy the requirements for PCI DSS testing for example, compromise of CHD, etc.). My email is : dnwaigbo@yahoo.com

  15. Andrew Ackerman says:

    Hi Russ –
    Thanks for the insight on PCI readiness. I’ve recently been tasked with facilitating a test of our IR plan and was wondering if you provide a copy of your Incident Response Audit Test Plan template? Looks like it’d be a great start in the direction I’m going.
    Email: ackerman06@gmail.com

    Thanks again Russ!

  16. Andrew Ackerman says:

    Hi Russ,
    Would you be so kind as to share your IR Audit Test Plan document? You have some great insight and I think this template will be a great start in developing a test scenario with a proper audit trail. Email: ackerman06@gmail.com

    Thanks again for the insight!

  17. Russ McRee says:

    Ping me directly at holisticinfosec at gmail dot com for a copy of the IR test plan.

  18. Mac says:

    Russ,

    Could I also get a copy of the test plan please? I would greatly appreciate it. Thanks, Mac (AMcCray@woh.rr.com)

Leave a Reply

Your email address will not be published. Required fields are marked *