There’s no way to measure your security awareness program’s success unless you identify the behaviors you want to address and develop a clear, actionable strategy. However, many cyber security leaders struggle to create a framework to quantify the success of their security awareness training. As a result, their organizations rely on intuition rather than clearly defined objectives and supporting data.
According to Mastercard-+, 95% of data breaches are due to human error. However, many of these errors could be fixed simply by having a cyber security awareness mindset.
There isn’t a lack of belief in the efficiency of cyber security awareness programs either. 66% of organizations expect their cyber security budget to grow in the next year, one of the most significant categories being awareness programs. What there might be a lack of is accurate data to justify the additional spending.
The only way to accurately measure security awareness training’s success is by developing a measurement methodology that incorporates tests, verifications, interviews, simulated events, and employee feedback. This article will highlight what to measure and how.
Security Awareness Program: Key Metrics and Data Points
Things you can measure to ascertain your security awareness program's success range from objective metrics like course completion and employee knowledge to non-objective data points like employee feedback. Some of the critical data points to measure are listed below:
- Training statistics - Used to measure employees' engagement with training materials and gauge their knowledge level. Metrics include the percentage of participants who have or haven't completed training, the time spent training, the pass/fail rate, and course completion rates within different departments.
- Participant satisfaction - Measure how satisfied employees are with your training strategy. You can monitor the percentage of happy employees, the ease of accessibility of training materials, content relevance, and the overall appeal of the content.
- Training effectiveness - Use effectiveness metrics to assess your current training efforts' quality and cost-effectiveness. Relevant metrics include the most popular awareness activities sorted by cost, the number of attendees per event, the average cost per attendee, and the most popular newsletter article.
- Return on investment (ROI) - Measuring ROI metrics highlights the real-world benefits of investing in behavioral change. Essential ROI indicators can include decreases in password reset tickets, computer reinstallations due to infections, downtime, and device theft. An increase in reporting suspicious events such as phishing is also an important metric.
- Subjective indicators - Use non-objective indicators to assess your security awareness program's general reception. Indicators include employee comments, perception of security, informal discussions, interviews, surveys, and more.
How to Measure Your Security Awareness Program's Success
Measuring a security awareness program’s success is about building an evaluation framework that incorporates various techniques like surveys, simulated events, event logging, monitoring, and assessments to test training effectiveness from multiple perspectives.
A versatile testing methodology is essential because each method provides you with a different data point you can use to evaluate the success or failure of various segments of your training program. Your methodology should contain objective and subject information to accurately ascertain your program’s success.
For example, measuring observed behavior objectively through simulated events like phishing simulations will provide you with objective employee knowledge insights. But, while those are important, they won’t tell you if employees are satisfied with the training experience. You’ll also need to gather subjective user feedback via interviews and surveys to provide a complete picture of a program’s performance.
The metrics you choose to use and track during your will depend on your overall campaign objectives. The measuring process will involve the following three phases:
- Phase 1: Gathering data
- Phase 2: Tracking progress
- Phase 3: Reporting
Below is an example of how to track progress using the three phases and based on the training objective of having all employees receive training on the phishing attack method.
Phase 1: Gathering Data
The first phase of monitoring your security awareness program is gathering training statistics, participant satisfaction, training effectiveness, ROI, and subjective indicator metrics. As mentioned above, the metrics you pick will depend on the type of goal you set.
For example, if your campaign goal is to create a security awareness program with testing materials that employees adopt, you can use user completion rates to measure that metric.
When considering what metrics you’ll measure, it’s essential to ensure that the metrics are relevant to your campaign goal, are readily available, and know what collection methods you’ll use.
This data is typically available in the Learning Management System (LMS) you use to deploy your learning activities. Ensure the system you use can provide the required data and reports.
Phase 2: Tracking Progress
The key to tracking processes is to track metrics and KPIs long-term as your security program matures. Tracking metrics over time will allow you to verify that your organization’s overall security awareness is improving or if there are any gaps in your strategy you can address.
For instance, if you’re monitoring employee engagement and discover that 25% aren’t participating in phishing awareness training or are not learning, then you can create a strategy to incentivize those employees to take part in assessments and enhance their cyber security knowledge.
This data is typically available in your Phishing Simulation Platform. Ensure the platform you use can provide instant feedback to clickers and track participation.
Phase 3: Report Insights to Your Team
Once you’ve collected a metric and tracked its long-term progress, it’s time to create reports to share with other team members and management to provide them with the insights you’ve collected.
To create reports effectively, you need a streamlined digital reporting solution that supports automated email reports. All reports should provide a graphical display highlighting data trends so that the recipient can seamlessly interpret data points.
So if you created a report detailing the number of employees who have completed training, you could use a reporting solution that offers pie charts so the recipient can easily visualize the proportion of employees who have or haven’t completed training.
You can then act and improve your awareness program to meet your objectives and keep security top of mind across your organization over the long term.
How to Collect User Feedback (And Avoid Survey Fatigue!)
There are two main techniques you have to collect user feedback:
- Written questions with multiple-choice answers you can use to test employee knowledge, gather input on topics of interest or gauge the appreciation of the current program activities.
- Discussions where you can ask employees to answer specific structured questions or participate in unstructured verbal feedback (typically as part of a focus group).
As a method for gathering user feedback, surveys have the advantage of scalability as they’re easy to collect. Still, they require employee participation, and the level of employee engagement can affect the reliability of the data gathered.
Before selecting your target audience size for your survey, determine the respondents’ confidence level requirements (e.g., 90%) and assume a 25-30% participation rate.
In comparison, interviews provide more in-depth information on employee experiences but aren’t scalable because they require a considerable time investment to interview a small group of respondents.
It’s important to note that no matter what method you choose to gather employee feedback, it’s vital to avoid survey fatigue. Survey fatigue is when an organization overwhelms employees with interviews and surveys to the point where they start to disengage from a training program and provide incomplete answers to get the assessment out of the way.
Survey fatigue leads to inaccurate responses to assessments or surveys and causes employee dissatisfaction. To ensure employees don’t get demotivated, make surveys and interviews low-investment by minimizing their time to complete.
Combine Metrics with the Human Element to Boost Cyber Culture
The metrics you collect on your security training provide essential insights you can use to make future-informed improvements to your program. Testing employee knowledge of cyber security regulations and external threats will let you know where your defenses are most vulnerable.
In addition, it will also dictate where you should focus on your future training efforts to boost your organization’s overall awareness. It’s also the best way to identify potential cyber security awareness champions in each department that will allow you to uphold the standards required.
While metrics are crucial in this process, don’t forget the human element at the center of it all. Your data should always direct you towards improvement in human behaviors.
From Data Protection to Cyber Culture
Understand your user better by learning how they feel about cyber security awareness in our recent study with Ipsos.
