Building a risk-scoring methodology for distributors and resellers

March 1, 2017

There are two main resources you will need to rate the risks of third parties: firstly, information about the entity (provided from any of a number of sources), and secondly, the people who will use that information to make decisions (such as the compliance team or staff from the business who perform part of the process). As there is only a finite amount that you can find out about your partners, there is an obvious desire to make the best use of that information. There also needs to be a balance between spreading the decision making across a large number of people whilst ensuring that the decisions that are made are consistent.

Here are things to think about when building a risk-scoring methodology to help you focus your controls and resources most efficiently.

1. What is the aim?

The first step in any risk-scoring exercise is to clearly define what you are trying to achieve. Generally a scoring process will be used to:

  • provide an automatic determination of the level of risk of a third party
  • help decide what the next step will be within a process (for example, a recommended level of due diligence)
  • help determine what to do next (for example, to determine whether the process remains with a business-level decision maker or is escalated to the compliance or legal review team)
  • provide broadly consistent results which represent a reliable best guess and can therefore be challenged or changed by an appropriately-trained member of your team
  • gather information on your partners, which may also be valuable to other parts of the business and can help justify the decision of whether or not to bring on a third party.

Considering the purpose of the scoring will have an impact later on in the process when deciding the types of questions to ask.

The second part of clarifying the aim is to build an expectation and means of measurement for what you think the outcome of the scoring programme will be. This will allow you to assess the validity of the programme during its working life and at later review stages. It is generally not recommended to try to fully automate the process until the scoring has been run and validated with a sufficient volume of partners.

2. Which information is relevant?

It is important to ascertain early on which information you will use to determine the overall score. Generally for third party corruption risk the main criteria are the location of the partner or where they will be providing their product or service, the type of service or product that partner is providing, and the value of that service.

Building a simple view of the world into high, medium and low risk helps to quickly determine a preliminary level of corruption risk based on a company’s location, although how you build on that view can be more difficult. Using openly-available lists can be a good start, but it is generally preferable to make your own list based on your knowledge of your own business. Think about where your company has had past or on-going compliance issues and where you have a strong presence to monitor third parties.

Establishing exactly which type of service a third party is providing is critical. The key to making this factor useful is to think about the risks you want to mitigate and consider what services might impact those risks.

  • Are they new or existing partners?
  • Are they touching end-users?
  • Are they selling to government customers?
  • Do you have contracts with them?
  • Do they obtain licences for selling products in that country on your behalf?
  • Do you provide market development funds to them?

Determining the value of the service is vital to ensure that your compliance objectives are still aligned to the business strategy by focusing on the high-value partners. Keep it relatively simple and stick to broad categories of contract value, for example $50,000 or $500,000 increments (depending on the type of contract).

Different compliance risks are likely to have different factors which require attention. There are also a number of other criteria which are sometimes included in a scoring scheme, such as sanctions and political exposure or previous compliance issues. Whilst there is some benefit in considering these factors, they are rarely uncovered through the questionnaire process so are best used as red flags rather than as determinative elements.

3. Where can I find the information?

The information which you can use to determine a perception of risk for a third party can be found from three main sources: what you know yourself about the third party, what the third party knows about itself, and what the world knows about the third party.

Throughout your organisation there is a vast amount of information about your third parties. Much of this information will be found in your ERP-type systems, but there is also plenty to be found from the institutional knowledge of your staff. Regardless of which source you use, you must firstly determine where the data is (i.e. which system or person holds the information) before deciding on how to go about accessing the information and which form you will gather it in. Internal information is generally the cheapest to gather, but might not provide all the data you need.

Often you can gain information about a third party’s internal structure and ownership, culture or past issues by asking that third party in a questionnaire or interview. Asking a partner for data is not generally expensive, although the time to manage the process needs to be considered. Whilst this can be a cost-effective means of gathering information, there may be questions about the veracity of what is provided where the third party doesn’t understand what you are asking, where you have asked the wrong person within the third party, or where it is not in the best interest of the other party to tell you.

Information gathered internally and from the third party should be verified at some point. External information might be able to help you do this, particularly if you have incomplete information. Certain factors, such as geographical risk and the value of the deal, could mean that external verification is needed to support the business decision to enter into the transaction. It may involve cross-checking the information you already have (i.e. from internally or from the partner) with a trusted source (like a corporate registry), or finding new information (such as the partner’s reputation).

4. Consider the questions

Based on the analysis of the criteria required and the possible sources to gain the information to support the criteria, there are further decisions to make:

  • Which is the most cost-effective source for the information? Based on the level of risk that you have determined for the third party based on their location and relationship with your organisation, which method of obtaining information will be proportionate to their risk and value?
  • What is the most accurate way of obtaining information? Often the same information can be found in multiple places, so decisions need to be made about which source is the most likely to be available for all partners, and which is likely to be most accurate. In some cases the same information can be sourced from multiple locations to improve accuracy.
  • Do you need to ask the question at all? Whenever a question is asked, there is always some cost incurred. The cost might be small (such as a need to translate or check accuracy) or potentially large (for example, asking for certain sensitive information in some jurisdictions might result in far greater data privacy concerns than are justified by the information gained).
  • How should the questions be worded to ensure the greatest efficiency in getting to the required answer? Given the lack of time and patience that most people have for answering questions it can be of great value to ensure that only relevant questions are asked and, where appropriate, only certain responses will lead to follow-up questions.
  • How do you write the questions to ensure the scores are usable? To use a scoring method the question will generally need to have one or more discrete responses (i.e. yes/no or multiple choice questions); however, discrete responses don’t allow for detailed answers, so it is normal to combine a discrete response with a follow up which allows more freedom (but can’t be scored).
    • Which questions and responses should be scored? Not all questions will provide relevant information for the key criteria, and not all responses will be interesting enough to warrant a score. Deciding which questions and answers warrant a score is vital to getting the overall scheme working.

Once the questions have been decided, other factors to consider are:

  • Who within that source will know the answers? If you ask your partner for information about themselves it is important that you consider who might have the relevant knowledge, as it is not always your primary contact. If asking internally, who might have access to IT systems where information will be available?
  • Does the person who knows the answers have the skills needed to respond accurately? If you are using internal resources to assess media reports, do they have the skills to read foreign-language media? Do they know how to assess the reputation of a media source? Can they assess the impact of a potential match on a sanctions list? Do they understand the analysis and research needed to assess a possible false-positive match? If they don’t have the skills, what training do they need to be able to respond accurately?

5. Are the responses accurate?

Once you have the answers you must ensure that they are accurate. This may involve:

  • sanity checks to ensure that the answers respond to the question and that the responder seems to have understood the question – this is especially useful when the questions have been translated into other languages
  • cross-checking against other information known about the partner
  • reviews by multiple people
  • audits and spot-checks to assess the accuracy and consistency of approvals.

Providing an accurate set of results from the process is important not only for your own use, but also because of the data protection and privacy obligations which you have to those who gave you the information.

6. What does it all mean?

Once all the base information has been collected and sufficiently reviewed you need to use it to make an intelligent decision. There are a number of methods that could be used to build the individual responses into a single, final score. These include:

  • averages (which could use the mean (arithmetic or geometric), median or the mode) – simple averages have a tendency to bunch around central figure and lose the impact of outliers, whilst unusual forms of average can be more difficult to understand and lead to unexpected results
  • additions – these are generally a safer form of analysis, but only when they are used with a small number of key criteria as they also have the tendency to be inaccurate when too many criteria have scores attached to them.

With the addition method, it is also possible to highlight clear red flags such as a sanctioned company or past criminal activities. In all cases it is strongly recommended to keep the scoring schemes as simple as possible, whilst still providing usable results.

As it is not always possible (or likely) that you will get the scheme correct the first time, it is important to design the overall process to allow changes to be made in the future as you learn more about the results.

7. What happens next?

Once the process has been run and a score has been generated for a particular partner, a number of questions will arise.

Firstly, how does the score fit into the overall workflow? As the score will generally be used to make decisions about the next steps it is important to decide:

  • who should make decisions (this might be a single person or a group, acting together or in sequence)
  • which decisions fall outside of the normal process and need to be escalated.

Secondly, how does the scoring and decision making get documented? In many cases recording the fact that a decision has been made and the reasons behind that decision can be just as important as the accuracy of the decision itself. When planning the documentation you should consider where it will be stored (whether it is in a database or in hardcopy format, and, if it is in hardcopy, its physical location) by deciding who might need to access the information. Access will generally be required by your internal compliance and legal teams, but your business teams might also utilise the information to establish partners’ capabilities, and internal auditors and investigators may need the information if an issue is discovered.

8. How will you carry out the review process?

At an appropriate interval it is important to stand back from the process and consider whether it met the aims set out in step one. Whether this is after a certain time period or after a particular number of partners will be dependent on the volume that you are expecting through the process. It is also advisable to have the review carried out by a team which is independent of the process, whether internal or external.

When reviewing the scoring methodology it is important to ask:

  • Did it accurately reflect the risk that you understood the partners posed?
  • Did it agree with what you would have decided yourself given the same information?
  • Were decisions made by the right people?
  • Were issues escalated to the right people?
  • Have the risks changed?
  • Can the process be changed, or has it been built into an inflexible technology or workflow?

Once the review is complete any necessary changes should be communicated to the staff involved in the process to ensure they know how their role is impacted.

 

Conclusion

Building a scoring model is a useful process and can be crucial to making the most out of the limited resources you have. It is important to remember:

  • don’t expect to use scoring to fully automate a process – the information available is generally not complete enough to provide an accurate model, so scoring is far better when used as a guide
  • don’t assume you will get it right first time (or second) – it is important to have a clear understanding of what you are aiming at, and to build regular review into the programme to recalibrate the scoring
  • keep the process and scoring as simple as possible – most of the relevant risk-related information can be found in a few key criteria
  • your perception of risk will change when new information comes to light, so remember to document the decision-making process so that you can justify the final risk outcome.
Previous Article
Ten soft skills of a compliance officer as a change agent
Ten soft skills of a compliance officer as a change agent

Introducing change also needs to rely on a cohesive company culture. Before launching any new programme, co...

Next Article
Improving your organisation’s compliance programme
Improving your organisation’s compliance programme

Compliance team may not be highly respected by the business or is generally regarded as dysfunctional. Glob...

Expand your team to overcome your compliance challenges

Tell me how