There’s no way to measure your security awareness program’s success unless you identify the behaviors you want to address and develop a clear, actionable strategy. However, many cyber security leaders struggle to develop a framework to quantify the success of their security awareness training. As a result, their organizations rely on intuition rather than clearly defined objectives and supporting data.
Lack of accurate measurements has meant that a substantial proportion of security awareness programs are ineffective. shows that 43% of employees are unaware that opening email attachments or links is likely to lead to malware infection and that 1 in 3 employees believe that not securing their laptop or mobile devices presents little to no security risk.
Even security and risk management leaders who go the extra mile to evaluate their security awareness programs often end up reliant on gathering feedback from users, a method that’s time-consuming and prone to bias.
The only way to accurately measure security awareness training’s success is by developing a measurement methodology that incorporates tests, verifications, interviews, simulated events, and employee feedback. This article will highlight what to measure and how.
What to Measure: Key Metrics and Data Points
Things you can measure to ascertain your security awareness program’s success range from objective metrics like course completion and employee knowledge to non-objective data points like employee feedback. Some of the key data points to measure are listed below:
- Training statistics – Used to measure how engaged employees are with training materials and gauge their knowledge level. Metrics include the percentage of participants who have or haven’t completed training, the time spent training, the pass/fail rate, and course completion rates within different departments.
- Participant satisfaction – Measure to evaluate how satisfied employees are with your training strategy. You can monitor the percentage of satisfied employees, the ease of accessibility of training materials, content relevance, and the overall appeal of the content.
- Training effectiveness – Use effectiveness metrics to assess your current training efforts’ quality and cost-effectiveness. Relevant metrics include most popular awareness activities sorted by cost, number of attendees per event, the average cost per attendee of an event, and the most popular newsletter article.
- Return on investment (ROI) – Measuring ROI metrics highlights the real-world benefits of investing in behavioral change. Essential ROI indicators can include decreases in password reset tickets, computer reinstallations due to infections, downtime, and device theft. An increase in reporting suspicious events such as phishing is also an important metric.
- Subjective indicators – Use non-objective indicators to assess your security awareness program’s general reception. Indicators include employee comments, perception of security, informal discussions, interviews, surveys, and more.
How to Measure Your Security Awareness Program’s Success
Measuring a security awareness program’s success is about building an evaluation framework that incorporates various techniques like surveys, simulated events, event logging and monitoring, and assessments to test training effectiveness from multiple perspectives.
Having a versatile testing methodology is essential because each method provides you with a different data point you can use to evaluate the success or failure of various segments of your training program. Your methodology should contain a mix of objective and subject information to accurately ascertain your program’s success.
For example, measuring observed behavior objectively through simulated events like phishing simulations will provide you with objective employee knowledge insights. But, while those are important, they won’t tell you if employees are satisfied with the training experience. To provide a complete picture of a program’s performance, you’ll also need to gather subjective user feedback via interviews and surveys.
The metrics you choose to use and track during your will depend on your overall campaign objectives. The measuring process will involve the following three phases:
- Phase 1: Gathering data
- Phase 2: Tracking progress
- Phase 3: Reporting
Below is an example of how to track progress using the three phases and based on the training objective of having all employees receive training on the phishing attack method.
Phase 1: Gathering Data
The first phase of monitoring your security awareness program is all about gathering data such as training statistics, participant satisfaction, training effectiveness, ROI, and subjective indicator metrics. As mentioned above, the metrics you pick will depend on the type of goal you set.
For example, if your campaign goal is to create a security awareness program with testing materials that employees adopt, you can use user completion rates to measure that metric.
When considering what metrics you’re going to measure, it’s essential to ensure that the metrics are relevant to your campaign goal, are readily available, and that you know what collection methods you’re going to use.
This data is typically available in the Learning Management System (LMS) you use to deploy your learning activities. Ensure the system you use can provide the required data and reports.
Phase 2: Tracking Progress
The key to tracking processes is to track metrics and KPIs long-term as your security program matures. Tracking metrics over time will allow you to verify that your organization’s overall security awareness is improving or if there are any gaps in your strategy you can address.
For instance, if you’re monitoring employee engagement and discover that 25% aren’t participating in phishing awareness training or are not learning, then you can then create a strategy to incentivize those employees to take part in assessments and enhance their cyber security knowledge.
This data is typically available in your Phishing Simulation Platform. Ensure the platform you use can provide instant feedback to clickers and track participation.
Phase 3: Report Insights to Your Team
Once you’ve collected a metric and tracked its progress long-term, it’s time to create reports to share with other members of your team and management to provide them with the insights you’ve collected.
To effectively create reports, you need a streamlined digital reporting solution that supports automated email reports. All reports should provide a graphical display that highlights data trends so that the recipient can seamlessly interpret data points.
So if you created a report detailing the number of employees who have completed training, you could use a reporting solution that offers pie charts so the recipient can easily visualize the proportion of employees who have or haven’t completed training.
You can then act and improve your awareness program to meet your objectives and keep security top of mind across your organization over the long term.
How to Collect User Feedback (And Avoid Survey Fatigue!)
There are two main techniques you have to collect user feedback:
- Surveys – Written questions with multiple-choice answers you can use to test employee knowledge, collect feedback on topics of interest or gauge the appreciation of the current program activities.
- Interviews – Discussions where you can ask employees to answer specific structured questions or take part in unstructured verbal feedback (typically as part of a focus group).
As a method for gathering user feedback, surveys have the advantage of scalability as they’re easy to collect. Still, they require employee participation, and the level of employee engagement can affect the reliability of the data gathered.
Before selecting your target audience size for your survey, determine the respondents’ confidence level requirements (e.g., 90%) and assume a 25-30% participation rate.
In comparison, interviews provide more in-depth information on employee experiences but aren’t scalable because they require a considerable time investment to interview a small group of respondents.
It’s important to note that no matter what method you choose to gather employee feedback, it’s vital to avoid survey fatigue. Survey fatigue is where an organization overwhelms employees with interviews and surveys to the point where they start to disengage from a training program and provide incomplete answers to get the assessment out of the way.
Survey fatigue leads to inaccurate responses to assessments or surveys and causes employee dissatisfaction. To ensure employees don’t get demotivated, make surveys and interviews low-investment by minimizing the time they take to complete.
The metrics you collect on your security training provide you with essential insights you can use to make informed improvements to your program in the future. Testing employee knowledge of cyber security regulations and external threats will let you know where your defenses are most vulnerable. In addition, it will also dictate where you should focus on your future training efforts to boost your organization’s overall awareness.
Download Measure the Success Document (Measure the Success of Your Security Awareness Program Without Asking)
Measure the Success of Your Security Awareness Program Without Asking
As cyber attacks become more and more common, security awareness programs have become a top priority for businesses of all sizes. However, as Gartner’s latest research pointed out, most Security Risk Management (SRM) professionals are still struggling to accurately measure the performance of their programs.
Efficiency made simple: download the report for a closer look.