How Much Risk Assessment Is Enough?

Nearly all compliance regulation and standards addressing aspects of information security (aka cyber security ;-)) contain a mandate for IT risk assessment (or analysis) and management.

Most often, these requirements are fairly generic. They do not prescribe a specific methodology for risk assessments, and they do not mandate a specific depth (level of detail) for performing them, either. This is both (generally) appropriate and (sometimes) problematic: 

• Appropriate, because organizations should be able to use whatever assessment methodology works well for them, and perform assessments to an extent that is commensurate with their size, value of assets, etc.
• Problematic, because it does not help newcomers to understand how much effort is appropriate in order for the assessments to be an effective tool in managing organizational risks, and which methodology might work well for them.

As a result, it is possible for inexperienced organizations to implement a superficial risk assessment process that meets the letter of the compliance requirements but otherwise doesn’t do the organization any good. This is why information security professionals like to point out on a weekly basis that compliance with, say, the PCI Data Security Standard doesn’t mean that a company (or their data) is secure, but that a good security program usually implies compliance without much additional effort.

This blog entry will help you recognize that risk assessments can (and should) be performed at different levels of abstractions. And I will provides some guidance on how to determine how much of it might be enough. (The question of which methodology to use is a different one and not discussed here. There are a number of decent ones out there.) Needless to say that the following discussion is based on the assumption that you actually want to do the right thing, rather than trying to get by with a minimum “check-box” attitude.

---

Requirements

First, let’s look at the requirement for risk assessments in a few common standards and regulations:

HIPAA, in 45 C.F.R. § 164.308 (a)(1)(ii)(A), tells entities involved in healthcare:

Risk analysis (Required).

Conduct an accurate and thorough assessment of the potential risks and vulnerabilities to the confidentiality, integrity, and availability of electronic protected health information held by the covered entity or business associate.

PCI DSS v3.0, requirement 12.2, tells the payment card industry:

Implement a risk-assessment process that:

* Is performed at least annually and upon significant changes to the environment (for example, acquisition, merger, relocation, etc.),

* Identifies critical assets, threats, and vulnerabilities, and

* Results in a formal risk assessment.

Similar requirements can be found in best practice frameworks. 

ISO/IEC 27001:2013, in laying out requirements for information security management systems, states in section 6.1.2:

The organization shall define and apply an information security risk assessment process that:

a) establishes and maintains information security risk criteria that include:
    1) the risk acceptance criteria; and
    2) criteria for performing information security risk assessments;

b) ensures that repeated information security risk assessments produce consistent, valid and comparable results;

c) identifies the information security risks […]

d) analyses the information security risks […]

e) evaluates the information security risks […]

And NIST’s Preliminary Cybersecurity Framework advises the critical infrastructure industry in category ID.RA to consider that:  

The organization understands the cybersecurity risk to organizational operations (including mission, functions, image, or reputation), organizational assets, and individuals.

Subcategories further spell out the need to identify and document vulnerabilities and threats for identified assets, analyze potential impacts, and identify risk responses.

---

Abstraction Levels of Risk Assessments

OK, so we have to perform risk assessments. But how much assessment is enough?

Risk assessments can obviously be performed at varying degrees of abstraction, each addressing (if so desired) the oft-required aspects of assets, threats, and vulnerabilities. (Implying that, eventually, you will also want to weight the value of an asset and the perceived or measured likelihood or frequency of a threat to actually occur in order to be able to qualify or quantify your risks in a meaningful way.)

For example, let’s consider this high-level analysis, fit to be understood by a VP or at a board level (adjust this view of corporate hierarchies accordingly to your organization size ;-)):

• There is a risk that we loose $$ because we cannot process orders (the primary information asset) because our e-commerce system goes offline for two days as a result of an attacker exploiting (threat) a security flaw in our system (vulnerability).

On an operational and technical level, this risk can be broken down into many facets that could cause this situation to occur, including:

• The risk that a denial-of-service attack by a socially motivated, sophisticated attacker (threat) exceeds the processing capacities of our load balancers (vulnerability) and prevents customers from placing orders.
  
• The risk that one of the systems contributing to the e-commerce infrastructure has been accidentally misconfigured (vulnerability) by an administrator, allowing a “script kiddie” to remotely access it and shut it down (threat).
  
• The risk of unexpectedly having to take the order processing system offline because a software flaw has been widely publicized that unintentionally exposes customers’ credit card data to unauthorized third parties on a publically accessible part of the web portal, and a fix isn’t available yet (vulnerability). (The threat of exploitation of that flaw here hasn’t materialized yet, but is so obviously eminent that you don’t have a choice. Or is the threat that you are forced to take the site offline? Either way. ;-))
  
• Etc.

Thus, when we are supposed to perform a risk assessment to achieve compliance with a respective requirement, we often seem to have two options:

1. Achieve compliance by documenting a risk assessment that estimates the likelihood and impact of a handful of management-level risks based on the gut feeling of what might be going on in our environment.
   
2. Improve IT security by defining a risk management process that assesses individual threats at a low-enough level to provide an informed management view of IT risk at higher levels, and allows to identify areas that might need treatment. Compliance is a side-effect. If we do things efficiently, we might even enable additional value for our business.

Some people like to get into terminology quibbles when discussing this topic. For example, they might tell you that risk assessments are worthless, and that threat modeling is the way to go. (What they may mean to say is that considering high-level risks is useless, and that a low-level risk assessment for specific technology or functionality should be done instead, because only the techies understand what’s really going on anyway. Or some such. ;-)) In my opinion, they are missing the point that you need to consider all levels of abstraction of risks to the extent that makes sense for your situation, and that risk assessments can very well be performed at a “threat modeling” level of detail and still be called risk assessments.

Without a high-level understanding of major risks, your board or VP won’t be able to provide the necessary acknowledgement, direction, and support to manage them in alignment with corporate objectives and to have your back; without breaking things down in sufficient detail for your IT department to understand the many different vectors through which a high-level risk might materialize, you might under- or overestimate the likelihood for things to happen or they might miss the point of which countermeasures will actually help to address things on an implementation level.

---

Contributing Factors

Regardless of whether we are instituting risk management based on compliance needs or simply as the foundation for proper information security management in an organization, the question remains “How much is enough?”. Do I need to engage every individual sysadmin in reviewing what they are responsible for and how it might affect our risk posture? Architects? Department heads? Or do I just sit in my office and try to make up things myself?

As so often in life (and information security), the answer is: It depends.

The Office of Civil Rights’ Guidance on Risk Analysis for HIPAA puts this fairly aptly by pointing out that “methods will vary dependent on the size, complexity, and capabilities of the organization”.

I can think of two main factors when trying to gauge how much effort you should put into performing and documenting your regular risk assessments: Organization size and the value of the organization’s assets. As always, common sense is an important aspect, too.

Organization Size

Size here might be influenced by the number of employees, complexity of operations, etc…

If the systems administrator is at the same time the Director of IT, he or she obviously has multiple levels of abstraction to consider (and a lot more diverse tasks on their plate, anyway). Does a smaller organization size mean that a flaw in your Apache web server isn’t as relevant as for a large company? That obviously depends on the type and value of information assets that could be affected if the flaw gets exploited by an attacker (for example, a static website vs. an underlying SQL database with 100,000 credit card numbers), and by the potential motivation of more or less resourceful attackers to exploit it.

A smaller organization does not typically mean that you can just overlook the lower-level threats, but likely the systems you have to deal with are less complex and less in number, and maybe the (aggregated) value of the assets you are trying to protect is lower, making it less likely that highly sophisticated attackers will waste their hard-earned zero-day exploits on you. Likely the communication levels between the techies and whoever is at the helm of accepting the overall (IT and other organizational) risks are shorter, requiring less formal documentation and reporting structures. (I do NOT mean to imply that you can completely omit documenting your risks, just that having less stakeholders means that a simpler reporting structure might very well be sufficient.)

On the contrary, a large enterprise with many different departments contributing to operations, productions, IT, etc. that all depend on IT performing as expected and information being available and uncompromised probably needs to look at risks at multiple, hierarchical levels of abstractions, with a well thought-out aggregation of overall risks that allows the top of the food chain to prioritize which ones are acceptable and which ones need further treatment.

Asset Value

A system that protects assets worth $ 10 billion, or the IP whose confidentiality the existence of your company hinges on, might require more dedication in terms of assessing the risks associated with its individual technical components than the one that hosts the Wiki documenting how to operate the microwave in the cafeteria. Prioritizing efforts based on asset values works both from a governance perspective when looking at overall company assets and the departments that contribute to protecting them, and from organizational and technical perspectives when trying to determine, for example, how the availability of individual systems would affect the overall capability of the organization to run its business in a more or less orderly fashion.

---

Defining An Approach

In general, a top-to-bottom approach might not be the worst idea when starting on a blank page. Understand what your major assets are, in terms of information and IT services that the organization depends on, and what they are worth to the organization. Then come up with types of events that might cause them harm. Since we are primarily talking about information and IT risks here, think about how IT is involved in protecting and enabling your assets and processes. This is where you might start polling more technical subject matter experts in your organization that can actually help you to understand how likely certain things are to happen, and what else could go wrong with your IT. Circle back, maybe through multiple iterations, until you are confident that you have an understanding of your risks that is clear and complete enough to start managing them.

An exercise that might help to get things started is to perform a business impact analysis that focuses on events that could cause potentially critical disruptions to the business. Those can then be taken and looked at from a more formal risk assessment perspective.

Whatever you do, settle on a methodology / documentation framework soon enough that meets your needs. Otherwise things will end in chaos!

COBIT, in particular COBIT 5 for Risk, also has some advice on “how much is enough”: It advertises the technique of using “risk scenarios” for performing assessments, and advises that the number of scenarios “should be representative and reflect business reality and complexity”, further explaining that:

Risk management helps to deal with the enormous complexity of today’s IT environments by prioritising potential action according to its value in reducing risk. Risk management is about reducing complexity, not generating it […]. However, the retained number of scenarios still needs to accurately reflect business reality and complexity.

In addition to figuring out the “depth” of your risk assessment activities, determining an appropriate organizational scope will also help with defining how much effort to spend. If parts of an organization can be identified whose operations do not have an impact, or less of a potential impact, on the risk to assets under consideration, then it may be appropriate to exclude them from initial risk assessment efforts.

Lastly, keep in mind that risk assessments aren’t a one-time thing. You need to circle back on a regular basis and consider new technologies and evolving threat landscapes. Ideally you will also collect metrics allowing you to qualify the understanding of your risks and the effectiveness of your mitigation measures.

---

Conclusion

The purpose of performing risk assessments is to enable informed decisions about managing the risks that an enterprise faces. Consequently, the key to determining how much effort to spend on risk assessments is to figure out how much information is needed in order to allow for informed risk management and governance.

Regardless of an organization’s size, this requires a certain amount of understanding of the subject matter at hand. In these days, information (and IT) assets are often of significant value to a business, if not in and by itself then by being able to affect all other aspects of operations. Unless a dedicated information security manager is at hand, it is likely the job of the CIO (or equivalent) position to obtain a reasonable understanding of the associated risks in order to a) communicate them to the governance level of the organization and obtain direction on priorities and risk appetite/tolerance, and b) direct staff to both assess and treat risks at the technological level.

It may well be appropriate for a three-person shop to not spend much time on formally assessing IT risks and rather concentrating efforts on developing the technology that earns the company money. But not giving thought at all to what could go wrong might be catastrophic to the business if an unanticipated threat realizes.

In larger organizations, clearly defining risk ownership, metrics, and reporting relationships throughout the company hierarchy will help to ensure proper assessment and management of risks.

In either situation, performing a risk assessment on paper just to satisfy a compliance requirement and without actually gaining a clearer understanding of the organization’s exposure and an opportunity to address it (maybe even in a way that actually adds value to your business) seems like a waste of time.