Phishing

Phishing Simulation - How Do You Calculate Effectiveness and ROI? (Part 1 of 2)

Jonathan Lampe
May 6, 2016 by
Jonathan Lampe

We now know that the best way to instill people with a healthy degree of "inbox suspicion" is to flood them with harmless but realistic-looking phishing messages. People who click the fake phishing messages are tracked and automatically educated with video-based training and exercises, thereby reducing the chances they will get fooled by real phishing messages.

Today, you can purchase "phishing simulation" services like this from a variety of vendors (including SecurityIQ from the InfoSec Institute), but how do you know if you are getting a good deal, if your investment is sound, or how your phishing simulation is performing against industry norms?

Phishing simulations & training

Phishing simulations & training

Build the knowledge and skills to stay cyber secure at work and home with 2,000+ security awareness resources. Unlock the right subscription plan for you.

Fortunately, you can pull a variety of hard numbers from your phishing simulator to calculate and compare traditional business KPIs like return on investment (ROI) and emerging KPIs like "open rates." Today, we will learn how to calculate the ROI of a phishing simulation and a follow-up article will look at other effectiveness KPIs.

Calculating ROI of Phishing Simulation

Introduction to ROI

"Return on Investment" or ROI is the one calculation all managers depend on. ROI figures calculated on multiple projects let managers rapidly determine which projects provide the "best bang for the buck", and detailed analysis of each project's ROI calculations can help managers understand the risks involved in achieving the goals of each project.

As you probably already know, ROI is calculated by dividing a project's "return" by its "investment".

ROI = Return / Investment

"Investment" is, of course, the project cost. Project "return" is itself calculated by subtracting the value of a project's gains from its initial cost. This subtraction is performed so a project with an ROI of 0% indicates a project with no return after costs are accounted for - in other words a bad project!

ROI = (GainProject - CostProject) / CostProject

Calculating the Value of Phishing Simulation (Gain)

Figuring out the cost of a phishing simulation project is fairly easy. In fact, many phishing simulators use an "all in" software-as-a-service cost to make the calculation of this figure particularly easy. But how do you calculate the value of better-trained users?

The answer lies in expected loss-due-to-phishing and improvement figures. Since phishing simulation will be reducing your organization's expected losses due to phishing attacks, you can show that the value of your phishing simulation is the amount that it will reduce your organization's expected losses.

To calculate your organization's expected losses to phishing, you can adopt one of the industry's well-known figures, such as:

  • $3.7M per year for an average 10K+ employee organization [1]
  • $1.6M per year for an average 1K+ employee organization in a regulated environment [2]
  • Approximately $2M for successful phishing attack, regardless of company size [3]

Keep in mind that these figures typically reflect organizations with limited security awareness training programs, and rarely reflect organizations with active anti-phishing programs (since cost-per-attack research is often conducted by phishing simulation firms after attacks on wide-open organizations). Now apply one or more figures to your organization, depending on your circumstances. For example:

LossExpectedDueToPhishing = $1.6M * 300 employees / 1000 employees = $480,000 / year

Once you have a phishing-loss estimate, you can move on to expected improvement. The best number to concentrate on here is the "click rate", which tracks what percentage of end users are expected to click on a phishing message during the year. Industry click rate figures range from 12% [4] to 40% [5] so a beginning number of 20-25% is not unreasonable.

More importantly, you must also estimate the expected improvement of the phishing simulation. This can range from a pessimistic, academic-backed estimate of 30% [6] through another academic-backed estimate of 60% [7] and on to wild-eyed vendor estimates of 80-90% or more.

ImprovementDueToPhishingSimulation = 50%

Once you have all these figures, you can use them to calculate a gain value based on reduced expected phishing losses.

GainProject = LossExpectedDueToPhishing X ImprovementDueToPhishingSimulation

For example:

GainProject = $480,000 / year X 50% = $240,000 / year

Calculating the ROI of Phishing Simulation

Once you have calculated the value of the gain you expect to realize by avoiding phishing attacks in your organization, you simply have to plug in the cost of the solution that will provide the phishing simulation and training to calculate its ROI.

ROI = (GainProject - CostProject) / CostProject

For example (with a $40,000/year solution yielding $240,000/year of protection):

ROI = ($240K/yr - $40K/yr)/$40K/yr = 500%

While it's unlikely that an anti-phishing solution for 300 users would really cost $40,000 a year, it's easy to see how and why so many phishing simulator projects get approved due to their high ROI's compared to other (less valuable) security projects!

In the second part of this article, we will show how you can demonstrate the value (or lack of value) of your existing phishing simulator.

Get six free posters

Get six free posters

Reinforce cybersecurity best practices with six eye-catching posters found in our free poster kit from our award-winning series, Work Bytes.


References:

  1. http://www.csoonline.com/article/2975807/cyber-attacks-espionage/phishing-is-a-37-million-annual-cost-for-average-large-company.html
  2. http://www.hipaajournal.com/cost-of-spear-phishing-8268/
  3. Multiple Sources: http://www.coindesk.com/bitpay-sues-insurer-after-losing-1-8-million-in-phishing-attack/ , http://www.hotforsecurity.com/blog/top-5-corporate-losses-due-to-hacking-1820.html , etc.
  4. Verizon 2016 Data Breach Report
  5. http://lorrie.cranor.org/pubs/pap1162-sheng.pdf
  6. http://lorrie.cranor.org/pubs/pap1162-sheng.pdf
  7. https://www.cs.cmu.edu/~ponguru/eCrime_APWG_08.pdf
Jonathan Lampe
Jonathan Lampe

Jonathan Lampe, CISSP has led the development of award-winning security software and supporting services for Standard Networks, Ipswitch, and  SolarWinds.  He holds computer science and business degrees from Northern Illinois University and the University of Wisconsin, and currently holds SANS GSNA and CCSK certifications in addition to his (ISC)2 credentials.  When not coding, hacking, or writing, Lampe likes to spend time with his family in the beautiful Wisconsin outdoors.