By Diana-Lynn Contesti (Chief Architect, CISSP-ISSAP, ISSMP, CSSLP, SSCP), and Richard Nealon  (Senior Security Consultant, CISSP-ISSMP, SSCP, SABSA SCF)

Ever find yourself in a struggle to defend your security budget or to introduce a change? This guide is a baseline to help you present the risk your organization faces.

We (CISOs) believe in notifying management regularly on the risk health of an organization and know the best time to approach management for funding is directly after a security breach. However, none of us want that to happen, so we find ourselves struggling to defend the current security budget when trying to implement a change. It is worthwhile looking at the other side of the coin here – not only do we focus on risk, but we should also be conscious of enabling opportunities. Metrics can also help us measure our ability to enable/pivot (e.g., some organisations’ ability to already have secure remote working in place ahead of the pandemic; other organisations’ ability to easily divest parts of the business due to having good network segregation in place; etc.)

As a CISO dealing with risk, we typically use metrics to supply information detailing how the company’s revenue and value are protected. This information is important to ensure that management is aware they are covered but also aware of dire impacts they could be facing as an organization. *Author’s note: The latter must be done in a way that does not appear to be scare tactics or they will not be taken seriously.

Using metrics to show the risk that an organization faces provides the stakeholders the ability to make educated decisions on the existing risk and then apply capital (funds, time, and people) appropriately.

Before working on metrics, you must decide what problem(s), you want to solve. Does management understand that all risks are not equal or that all systems may not have the same value? Can you quantify the security risks to aid in the prioritization? Are stakeholder/system owners held accountable? Is the organisation passively accepting a level of risk that far exceeds its risk appetite/risk tolerance?

The title of Andrew Jaquith’s book “Security Metrics: Replacing Fear, Uncertainty, & Doubt” details the viewpoint management has on Information Security. If members of the management team do not understand the value they are getting for their spend, they will start making things up and this can lead to loss of budget, staff cuts, etc. Therefore, you must ensure metrics are meaningful, convey a clear picture and support a business goal(s). Quite simply put, do not use fear when dealing with metrics, explain the unknown and remove any doubt that the security program is effective.

This guide provides the reader with a brief tutorial on how to develop meaningful metrics.

We recommend that the following steps take place before trying to generate metrics:

Step 1: Use an industry accepted guidance.

Align security metrics to risk exposure, and risk appetite/tolerance. If organisations do not understand their business objectives for security, then you are fighting an uphill battle. Metrics (at least at Board/C-Suite level) should be focusing on the major risks (avoid reporting on stuff that does not really matter).

We suggest that you use enterprise frameworks for comparison. Frameworks include (but not limited to):

• ITIL
• ANSI’s ISO 27002
• NIST Cybersecurity Framework

You will need to decide which model suits your organization best.

Step 2: Know the audience.

It is critical to understand the audience. The audience that you will present to can include:

• C-Suite
• Management
• Stakeholders (system owners, business system owners, data owners)
• Board of Directors

Once you know your audience, create a matrix to show what metric will be presented to each group.

C-Suite	Mid Management	System Owners
State of Security (number of Red, Amber, Yellow, Green risks recorded on the risk register)	Incident response times	Number of incidents investigated
Compliance reporting	Patch management reporting	Patch compliance
Open vulnerabilities (and their criticality)	Open vulnerabilities (and their criticality)	Information on threats

(for example only)

In developing this table, understand the audience and the message they need to receive. Decide what message each stakeholder needs to receive, and what will they do with the information.

You can also break the C-Suite metrics scorecard down into a matrix of “Technology, Process, and People & Environment” (each with a few subheadings), mapped onto “First Line (operational), Second Line (internal oversight) and Third Line (external oversight, e.g. audit, supervision, due diligence).

The CEO of the corporation or the Board of Directors does not need the details of systems not patched or the number of virus incidents that are being responded to; however, the owners of the various systems do need to understand when systems are not patched.

Step 3: Understand the data.

Before collecting data and trying to develop metrics, ask yourself questions.

1. What data do you collect?
2. How do you user the data?
3. To whom are you presenting the data?
4. Do you understand what is important to the business and how IT supports it?
5. What story do you want the data to tell?
6. What are the sources of the data?
7. What are the actions or expectations that you want to occur because of the metrics?

Systems and tools (anti-virus scanners, vulnerability scanners, etc.) can generate terabytes of data but not all of it is meaningful. Holding on to and generating excess data can be costly.

Looking at the data, decide the characteristics it has. Does it deliver a meaningful message? Decide which source of data is most useful for the information you are providing.

Keep in mind that there are many sources of data (external penetration tests, internal vulnerability scans, system logs) but not all these sources are consistent or relevant. When collecting data, ensure that it is easy to collect, replicable, shows a trend and define the action(s) to remedy the risk(s).

Typically for service owners, the data is very granular and may have percentages; and typically displayed as a graph. However, we recommend using charts with colors for more senior management (with drill down information being available as needed).

Step 4: Understanding what makes a good metric is important.

For metrics to be useful they should be consistent, economical to collect, actionable, predictive and aligned with organizational goals

Once you have decided on a method, develop a Plan of Action and Milestones (PoAMs) that will guide you and the audience in understanding the next steps.

Developing the Metrics:

1. Define the purpose of the metric. What does the metric predict or evaluate and what understanding does it supply the reader? Ensure that the reader understands the corrective steps needed to improve or manage the process around the metric.
2. Examine the costs or effectiveness of the metric from the stakeholder’s point of view.
3. Determine the environment considering all the pertinent factors (people, problems, tools, constraints).

Presenting Metrics:

When presenting data/metrics set up the preferred method with your team(s). Meet with management to decide acceptable levels of risk and thresholds. Do not assume that having desktops patched at the 90% level is acceptable. Understand the granularity needed to convey the message. This information will all become part of your PoAMs. Discuss with the potential audience member how they would like to see the data presented, understand their risk tolerance, and talk about follow up actions.

As an example, when looking at desktop patching, should you report on the types of devices (all devices or desktops and laptops) or lump them all together? One mistake typically made is to generate a percentage patched monthly. However, unless the total number of computers in the environment is also presented, the metric could be misleading on a month to month or year over year basis. This change can be the result of factors, such as computers taken out of service for maintenance and laptops that have not been attached to the network in a while.

It is recommended that predetermination of presentation method be defined.

C-Suite	Management	Stakeholders
-use colours instead of numbers (red, yellow, and green)	-use colours instead of numbers (red, yellow, and green)
	– graphics	– graphics

 (for example only)

Using colours is one way, however you need to be cognizant and design your presentation to be accessible to the visually impaired. Ensure that when using colours there is a full understanding of what each colour represents.

Once you have chosen the method of presentation, begin development.

Using graphs may provide simple visualization, a graphical visualization, or a more complex visualization. A simple visualization might include a grid showing the results (a typical example could include: the number of privileged accounts, the number of accounts that have remote access, or the number of “vendor” accounts). A graphical visualization may be a time-series with values plotted for each period and may also display differing incident classifications. Complex visualizations may be used to display cross-functional results.

A word of Caution:

In general

• Understand the difference between measurement and metrics, and only report metrics.
  • Measurements are values without any context (e.g., patching on all servers is at 80%. This does not indicate whether 80% is good or bad).
  • Metrics put those measurement into context (e.g., patching on Business-Critical servers is at 80% as of 08:00 this morning. Our patching level threshold for BC servers is 87%, so this is marked as RED (Potential Severe Business Impact).
    • Supplementary information: The reason that we score Red on this metric is that 20 new BC servers have been deployed without being hardened (see Security Incident #123). The infrastructure team expect that this work will be carried out in the next two days and be fully remediated. This has been logged as Operational Risk item #567 on the OR Register). An update will be provided when this work has been completed.
  • Ensure accuracy. Validate the figures if you’re not producing them yourself (people make mistakes).
  • Make sure they are meaningful (i.e., prompt/assist decision making or inform key stakeholders) and align with the business objectives of the organisation. There is no point in reporting that a control is working (e.g., 10 DDoS attacks prevented by DDoS protection control).
  • Be mindful of the internal politics. Some parts of the business may potentially try to hinder you (because plausible deniability seems more palatable than accountability; because they are not meeting their security obligations; etc.). take time to explain that you may be able to help them e.g., if they’re unable to do the work because they’re being squeezed for resources – that’s something that you can measure.
  • When agreeing value targets or ranges for metrics, be cautious that some stakeholders may be keen to set unachievable/unrealistic values. Values should be informed by the risk appetite/tolerance of the organisation.

When tailoring the message to the audience:

• Never “cry wolf” – this tactic may get instant results but overall can be detrimental to one’s credibility.
• Ensure your message allows for two-way conversation Ensure you deliver on your commitments – do not say that you will patch to 99.99% unless you are certain that you can deliver or that there will be zero (0) incursions from virus.

In closing:

Not currently presenting metrics? Start small and ensure that reporting these metrics becomes a regular process.

As stakeholders understand the metrics and begin using them, discuss what other metrics they could use. Gradually begin adding new metrics that are meaningful.

The key to metrics is to alert management to issues (risks) facing them as well as justifying your security budget.

Don’t let the numbers drive the strategy. Sometimes stakeholders get fixated with the metrics (the first time that they’ve been able to see their security posture) to the detriment of all else. Hence, it sometimes transpires that it’s only what’s on the scorecard that gets attention.

Remember, the metrics are just a by-product – an assurance that the controls being reported on are operating effectively. They shouldn’t be used to dictate the security program.

How do you present metrics to your stakeholders to defend your security budget? What works best? Share your best practices.