The Art of Writing Penetration Test Reports
You close the lid of your laptop; it’s been a productive couple of days. There are a few things that could be tightened up, but overall the place isn’t doing a bad job. Exchange pleasantries with the people who have begrudgingly given up time to escort you, hand in your visitors badge and head for the door. Just as you feel the chill of outside against your skin, you hear a muffed voice in the background.
“Hey, sorry, I forgot to ask, when can we expect the report?”
Ugh, the report. Penetration testing’s least favorite cousin, but ultimately, one of the most important.
There are thousands of books written about information security and pen testing. There are hundreds of hours of training courses that cover the penetration testing process. However, I would happily wager that less than ten percent of all the material out there is dedicated to reporting. This, when you consider that you probably spend 40-50% of the total duration of a pen test engagement actually writing the report, is quite alarming.
It’s not surprising though, teaching someone how to write a report just isn’t as sexy as describing how to craft the perfect buffer overflow, or pivot round a network using Metasploit. I totally get that, even learning how the TCP packet structure works for the nineteenth time sounds like a more interesting topic.
A common occurrence amongst many pen testers. Not allowing enough time to produce a decent report.
No matter how technically able we are as security testers, it is often a challenge to explain a deeply technical issue to someone who may not have the same level of technical skill. We are often guilty of making assumptions that everyone who works in IT has read the same books, or has the same interests as us. Learning to explain pen test findings in a clear and concise way is an art form, and one that every security professional should take the time to master. The benefits of doing so are great. You’ll develop a better relationship with your clients, who will want to make use of your services over and over again. You’ll also save time and money, trust me. I once drove a 350 mile round trip to go and explain the contents of a penetration test report to a client. I turned up, read some pages of the report aloud with added explanations and then left fifteen minutes later. Had I taken a tiny bit more time clarifying certain issues in my report, I would have saved an entire day of my time and a whole tank of gas.
Diluted: “SSH version one should be disabled as it contains high severity vulnerabilities that may allow an attacker already on the network to intercept and decrypt communications, although the risk of an attacker gaining access to the network is very low, so this reduces the severity.”
Clarified: “It is advisable to disable SSH version one on these devices, failure to do so could allow an attacker with local network access to decrypt and intercept communications.”
Why is a penetration test report so important?
Never forget, penetration testing is a scientific process, and like all scientific processes it should be repeatable by an independent party. If a client disagrees with the findings of a test, they have every right to ask for a second opinion from another tester. If your report doesn’t detail how you arrived at a conclusion, the second tester will have no idea how to repeat the steps you took to get there. This could lead to them offering a different conclusion, making you look a bit silly and worse still, leaving a potential vulnerability exposed to the world.
Bad: “Using a port scanner I detected an open TCP port”.
Better: “Using Nmap 5.50, a port scanner, I detected an open TCP port using the SYN scanning technique on a selected range of ports. The command line was: nmap –sS –p 7000-8000.”
The report is the tangible output of the testing process, and the only real evidence that a test actually took place. Chances are, senior management (who likely approved funding for the test) weren’t around when the testers came into the office, and even if they were, they probably didn’t pay a great deal of attention. So to them, the report is the only thing they have to go on when justifying the expense of the test. Having a penetration test performed isn’t like any other type of contract work. Once the contract is done there is no new system implemented, or no new pieces of code added to an application. Without the report, it’s very hard to explain to someone what exactly they’ve just paid for.
Who is the report for?
While the exact audience of the report will vary depending on the organization, it’s safe to assume that it will be viewed by at least three types of people.
Senior management, IT management and IT technical staff will all likely see the report, or at least part of it. All of these groups will want to get different snippets of information. Senior management simply doesn’t care, or doesn’t understand what it means if a payment server encrypts connections using SSL version two. All they want to know is the answer to one simple question “are we secure – yay or nay?”
IT management will be interested in the overall security of the organization, but will also want to make sure that their particular departments are not the cause of any major issues discovered during testing. I recall giving one particularly damming report to three IT managers. Upon reading it two of them turned very pale, while the third smiled and said “great, no database security issues then”.
IT staff will be the people responsible for fixing any issues found during testing. They will want to know three things. The name of the system affected, how serious the vulnerability is and how to fix it. They will also want this information presented to them in a way that is clear and organized. I find the best way is to group this information by asset and severity. So for example, “Server A” is vulnerable to “Vulnerability X, Y and Z. Vulnerability Y is the most critical”. This gives IT staff half a chance of working through the list of issues in a reasonable timeframe. There is nothing worse than having to work your way backwards and forwards through pages of report output to try and keep track of vulnerabilities and whether or not they’ve been looked at.
Of course, you could always ask your client how they would like vulnerabilities grouped. After all, the test is really for their benefit and they are the people paying! Some clients prefer to have a page detailing each vulnerability, with affected assets listed under the vulnerability title. This is useful in situations where separate teams may all have responsibilities for different areas of a single asset. For example, the systems team runs the webserver, but the development team writes the code for the application hosted on it.
Although I’ve mentioned the three most common audiences for pen test reports, this isn’t an exhaustive list. Once the report is handed over to the client, it’s up to them what they do with it. It may end up being presented to auditors, as evidence that certain controls are working. It could be presented to potential customers by the sales team. “Anyone can say their product is secure, but can they prove it? We can, look here is a pen test report”.
Reports might even end up getting shared with the whole organization. It sounds crazy, but it happens. I once performed a social engineering test, the results of which were less than ideal for the client. The enraged CEO shared the report with the whole organization, as a way of raising awareness of social engineering attacks. This was made more interesting, when I visited that same company a few weeks later to deliver some security awareness training. During my introduction, I explained that my company did security testing and was responsible for the social engineering test a few weeks back. This was greeted with angry stares and snide comments about how I’d gotten them all into trouble. My response was, as always, “better to give me your passwords than a genuine bad guy”.
What should the report contain?
Sometimes you’ll get lucky and the client will spell out exactly what they want to see in the report during the initial planning phase. This includes both content and layout. I’ve seen this happen to extreme levels of detail, such as what font size and line spacing settings should be used. However, more often than not, the client won’t know what they want and it’ll be your job to tell them.
So without further ado, here are some highly recommended sections to include in pen test reports.
A Cover Sheet. This may seem obvious, but the details that should be included on the cover sheet can be less obvious. The name and logo of the testing company, as well as the name of the client should feature prominently. Any title given to the test such as “internal network scan” or “DMZ test” should also be up there, to avoid confusion when performing several tests for the same client. The date the test was performed should appear. If you perform the same tests on a quarterly basis this is very important, so that the client or the client’s auditor can tell whether or not their security posture is improving or getting worse over time. The cover sheet should also contain the document’s classification. Agree this with the client prior to testing; ask them how they want the document protectively marked. A penetration test report is a commercially sensitive document and both you and the client will want to handle it as such.
The Executive Summary. I’ve seen some that have gone on for three or four pages and read more like a Jane Austen novel than an abbreviated version of the report’s juicy bits. This needs to be less than a page. Don’t mention any specific tools, technologies or techniques used, they simply don’t care. All they need to know is what you did, “we performed a penetration test of servers belonging to X application”, and what happened, “we found some security problems in one of the payment servers”. What needs to happen next and why “you should tell someone to fix these problems and get us in to re-test the payment server, if you don’t you won’t be PCI compliant and you may get a fine”. The last line of the executive summary should always be a conclusion that explicitly spells out whether or not the systems tested are secure or insecure, “overall we have found this system to be insecure”. It could even be just a single word.
A bad way to end an executive summary: “In conclusion, we have found some areas where security policy is working well, but other areas where it isn’t being followed at all. This leads to some risk, but not a critical amount of risk.”
A better way: “In conclusion, we have identified areas where security policy is not being adhered to, this introduces a risk to the organization and therefore we must declare the system as insecure.”
Summary of Vulnerabilities. Group the vulnerabilities on a single page so that at a glance an IT manager can tell how much work needs to be done. You could use fancy graphics like tables or charts to make it clearer – but don’t overdo it. Vulnerabilities can be grouped by category (e.g. software issue, network device configuration, password policy), severity or CVSS score –the possibilities are endless. Just find something that works well and is easy to understand.
Test Team Details. It is important to record the name of every tester involved in the testing process. This is not just so you and your colleagues can be hunted down should you break something. It’s a common courtesy to let a client know who has been on their network and provide a point of contact to discuss the report with. Some clients and testing companies also like to rotate the testers assigned to a particular set of tests. It’s always nice to cast a different set of eyes over a system. If you are performing a test for a UK government department under the CHECK scheme, including the name of the team leader and any team members is a mandatory requirement.
List of the Tools Used. Include versions and a brief description of the function. This goes back to repeatability. If anyone is going to accurately reproduce your test, they will need to know exactly which tools you used.
A copy of the original scope of work. This will have been agreed in advance, but reprinting here for reference purposes is useful.
The main body of the report. This is what it’s all about. The main body of the report should include details of all detected vulnerabilities, how you detected the vulnerability, clear technical expiations of how the vulnerability could be exploited, and the likelihood of exploitation. Whatever you do, make sure you write your own explanations, I’ve lost count of the number of reports that I’ve seen that are simply copy and paste jobs from vulnerability scanner output. It makes my skin crawl; it’s unprofessional, often unclear and irrelevant. Detailed remediation advice should also be included. Nothing is more annoying to the person charged with fixing a problem than receiving flakey remediation advice. For example, “Disable SSL version 2 support” does not constitute remediation advice. Explain the exact steps required to disable SSL version 2 support on the platform in question. As interesting as reading how to disable SSL version 2 on Apache is, it’s not very useful if all your servers are running Microsoft IIS. Back up findings with links to references such as vendor security bulletins and CVE’s.
Getting the level of detail in a report right is a tricky business. I once wrote a report that was described as “overwhelming” because it was simply too detailed, so on my next test I wrote a less detailed report. This was subsequently rejected because it “lacked detail”. Talk about moving the goalposts. The best thing to do is spend time with the client, learn exactly who the audience will be and what they want to get out of the report.
When a pilot lands an airliner, their job isn’t over. They still have to navigate the myriad of taxiways and park at the gate safely. The same is true of you and your pen test reports, just because its finished doesn’t mean you can switch off entirely. You still have to get the report out to the client, and you have to do so securely. Electronic distribution using public key cryptography is probably the best option, but not always possible. If symmetric encryption is to be used, a strong key should be used and must be transmitted out of band. Under no circumstances should a report be transmitted unencrypted. It all sounds like common sense, but all too often people fall down at the final hurdle.