Phishing Simulation – How Do You Calculate Effectiveness and ROI? (Part 2 of 2)

June 3, 2016 by

Jonathan Lampe

In our first article about calculating the effectiveness and return on investment (ROI) of a phishing simulation, we discussed theoretical or model situations. This article shows how you can demonstrate the value (or lack of value) of your existing phishing simulator.

Review: How to Calculate ROI

Simply put, calculating the return on your project’s investment involves comparing the project’s benefit (or gain) the project will produce with the cost involved in realizing that benefit. The basic formula works like this:

Phishing simulations & training

Build the knowledge and skills to stay cyber secure at work and home with 2,000+ security awareness resources. Unlock the right subscription plan for you.

VIEW PRICING

ROI = (Gain_of_Project – Cost_of_Project) / Cost_of_Project

To calculate the benefit of a phishing simulation project, you generally need to calculate how much risk the project will help the organization avoid. This involves estimating your expected losses due to phishing, and how much better you expect your employees to perform after phishing simulation. The two related formulas work like this:

Expected_Loss = (Documented_Loss / Documented_Employees) x Actual_Employees

Gain_of_Project = Expected_Loss x Expected_Improvement

For example, if you read that a thousand-person organization can expect to lose $1.6M a year on phishing attacks and related clean-up, and you work for a 300-person organization, you can plug those figures into the “Expected Loss” formula.

Expected_Loss = (Documented_Loss / Documented_Employees) x Actual_Employees

$480K loss = ($1.6M loss / 1,000 employees) x 300 employees

Then, you can take that value and how much better your employees are after phishing simulation (let’s say 50%), and plug those values into the “Gain of Project” formula.

Gain_of_Project = Expected_Loss x Expected_Improvement

$240K benefit = $480K loss x 50% improvement

And finally, you can plug the project gain and the actual cost of your solution into the ROI formula.

ROI = (Gain_of_Project – Cost_of_Project) / Cost_of_Project

500% ROI = ( $240K benefit – $40K cost ) / $40K cost

How to Calculate Your Actual ROI

You probably noticed that the calculations detailed so far depended on two estimates, or “expected” values:

• Expected Improvement (e.g., 50%)
• Expected Loss (e.g., $480K for a 300-employee organization)

You would be forgiven if your thought those “estimates” were more accurately called “wild guesses.” Fortunately, you can get more accurate and reliable figures from your own organization.

Actual Improvement

Your existing phishing simulator has reporting that allows you to tell how much “smarter” your employees are than they were in the past. Ideally, you would have sent your original phishing campaign to all employees without any notice, and many of them (let’s say 50%) would clicked on one or more of the initial phishing emails. Then, ideally, you would have sent the same (or similar) campaign to everyone again and seen if they fell for the same test (usually not - let’s say 40%).

For example, here’s what a campaign configured to repeat five times in the SecurityIQ phishing simulator looks like just before it gets launched.

If you run the same phishing campaign to the same employees about a year apart, you can compare the results of your beginning-of-year and end-of-year runs. The difference between your initial assessment and your end-of-year assessment allows you to estimate the actual improvement your employees showed after participating in your phishing simulation.

• Actual Improvement replaces Expected Improvement
• Actual_Improvement = (Initial_Test - EndOfYear_Test) / Initial_Test

For example:

• 58% actual improvement = ( 48% phish_start - 20% phish_end ) / 48% phish_start

Often, you can graph the difference between the initial and end-of-year assessments, or even drill into different improvements such as clicking drive-by attacks vs. just opening them. For example, the SecurityIQ phishing simulator graphs different reactions to phishing tests in different colors.

Actual Loss

One way to calculate actual losses in your organization is to actually suffer a hack caused by a phishing attack and then interview your executives about actual costs. (If they haven’t already asked you to make sure it never happens again.) However, there is another way to calculate actual loss for organizations fortunate enough to avoided a major hack caused by phishing so far.

If you have a phishing simulator that includes an Outlook plug-in or other technology that allows employees to report phishing attacks that have penetrated all your other defenses, you can get an idea of how many real-world phishing attacks are hitting people’s desks. The math behind that calculation is typically performed by the phishing simulator and works like this:

• Attacks_Per_Employee = Real_World_Attacks / (Employees x Participation_Rate)

For example, if an organization with 1,500 employees and a “report phishing” participation rate of 50% fields 3,000 reports of real-world phishing attacks in a year, then the formula suggests that four (4) real world attacks are landing on each employee’s desk each year.

• 4 attacks/employee = 3,000 reported attacks / (1,500 employees x 50% participation)

From there, you can use a per email phishing cost to estimate the cost of each attack that lands in a user’s inbox. Such a figure can be derived from the $1,525 cost per successful email and 10% success rate taken from a Cyveliance 2015 phishing report or other sources.

• Cost_Of_Attack = Cost_Of_Successful_Phishing_Email x Success_Rate
• $152.50/attack = $1,525/successful_attack x 10% success rate

That can be combined with your organization’s size and attacks per employee to yield your actual cost of phishing risk.

• Actual_Loss = Employees x Attacks_Per_Employee x Cost_Of_Attack
• $915K loss = 1,500 employees x 4 attacks/employee x $152.50/attack

Calculating Actual ROI

Now that you have more realistic loss and improvement figures, you can calculate a much more realistic ROI for your existing project. Remember the ROI formula:

ROI = (Gain_of_Project – Cost_of_Project) / Cost_of_Project

Now we update the “Expected” components of the project benefit formulas with “Actual” components:

Gain_of_Project = Actual_Loss x Actual_Improvement

For example, using the figures we calculated above, we can recalculate the benefit of the project with realistic figures.

$530K project benefit = $915K loss x 58% improvement

Finally, you can plug your realistic project gain (derived from actual reporting) with the actual cost of your solution to get a true ROI calculation.

ROI = (Gain_of_Project – Cost_of_Project) / Cost_of_Project

1225% ROI = ($530K benefit – $40K cost) / $40K cost

As compared to the “mere” 500% ROI from the estimated calculations, a 1225% is quite something! However, the difference in values also demonstrates the need to use real-world, assessment-based in your calculations to realistically understand the true value of your solution.

Phishing simulations & training

Build the knowledge and skills to stay cyber secure at work and home with 2,000+ security awareness resources. Unlock the right subscription plan for you.

VIEW PRICING

If you're ready to start your own phishing campaigns and calculate real-world ROI figures for your company, download a free trial of SecurityIQ now. SecurityIQ combines a phishing simulator and computer-based security awareness training in one easy-to-use cloud-based service.