Blog: Vendor Management

"You need a pen test. This is a vulnerability assessment. Have you considered Red Team testing?"

You might have been told this by a regulatory examiner, your IT vendor, a senior partner, a Board member, or read it in an article. It's a common statement in the penetration testing space these days and with good reason. The scope and methodology of penetration tests is not standardized or regulated, so each provider can create their pen test service as they see fit. This puts more responsibility on you, the customer, to determine if they are meeting your needs.

In this article:

We covered these topics in our webinar Choosing Pen Tests & Real-Life Horror Stories. In this webinar we discussed:

  • The different types of penetration tests available
  • The pros and cons of different penetration tests
  • How to choose the best penetration test for your organization
  • Stories of real-world exploits
  • How to make the most of your penetration testing budget

What is Penetration Testing?

Many terms are used to describe penetration testing…

Red Team, Physical Intrusion, Gray Box Testing, Phishing, Reconnaissance, Privilege Escalation, Precision Strike, War Dialing, Black Box Testing, Social Engineering, Capture the Flag, Web Application Testing, Purple Team, White Box Testing, Blue Team, Pivoting, Internal Testing, External Testing

They overlap, they conflict, they can be misleading.

What is Penetration Testing?

So, what is pen testing?

Whether it's physical, logical, or human, pen testing shows you how an attacker would look at your organization. They look for holes and possibilities to disrupt the way you work.

Penetration testers look for holes and vulnerabilities that could disrupt the way you work.

According to the FFIEC IT Exam Handbook for financial institutions, there are many types of penetration tests . . . and management should determine the level and types of tests employed to ensure effective and comprehensive coverage.* The regulators are giving you the responsibility for figuring out what level and type of pen test you need.

So, what terms mean something, and what is terminology fluff? What should you look for when determining the pen test you need and what vendor to use?

Choosing and understanding penetration test vendors and knowing the terminology is important. Our aim in this article is to help remove some of the mystery around pen testing and help you feel confident in determine the best solution for your company.

What type of Pen Test do you need?

What is the type or level of penetration testing that you need?
Let's go back to the regulation:
"A penetration test subjects a system to real-world attacks selected and conducted by the testers." - FFIEC IT Exam Handbook, Information Security Booklet, Sep 2016 (emphasis added)

Essentially you are choosing what you want tested, you are not telling them how you want them to test it. That should be up to the penetration tester. The reason is, the penetration tester should have the expertise to know what the attackers are doing, and they are going to choose the real-world attacks that the attackers would choose. They have done the research, gone through the certifications and the training to know what types of methods to use. So it's important to know you're not telling them how to do the test, you're telling them what you want tested.

You figure out WHAT you need tested and the penetration tester will figure out HOW to test it.

Here is a simple definition of penetration testing that doesn't use any of the fancy marketing terms. Let's start here.

At its core, penetration testing should show you how an attacker would see you and describe specific attacks they would choose to use against you. That means you don't need to tell the pen testers HOW to do the testing. You just need to tell them WHAT to test and ensure their METHODOLOGY meets your needs and keeps you safe.

So, how do you determine WHAT you need?

Start with a Risk Assessment

When done correctly, a risk assessment will inform you of what you need to do for security testing.

Let's use an example to illustrate. Here is a simple risk assessment.

Simple Risk Assessment

  • What attacks do we hear about from IT, in the news, etc.?
    • Phishing!
    • Ransomware
    • Website attacks
  • What assets do those attacks target for us?
    • Employees
    • Corporate email and perimeter defenses
    • Web servers
  • What testing do we need?
    • Email social engineering for ALL employees
    • Internet perimeter testing for ALL of our public IP addresses

In this sample risk assessment, we are showing that as a company that offers services via the Internet our top concern is cybersecurity; but more specifically, we are concerned with phishing, ransomware, and website attacks.

So how do those threats apply to us? Phishing relies on our employees to click a link or open an attachment, so testing our people is important. Ransomware is typically introduced from the outside, so our corporate email defenses and perimeter defenses, such as firewalls, are important. Website attacks target our webservers directly, so those are also important targets.

Translate your Risk Assessment

After you perform a risk assessment, you will need to translate the results to answer the questions "What" to test and "Why" test it. The penetration test company will figure out "How" to test it.

For our sample risk assessment illustrated above, here is what we would choose:

  1. What to test? — Internet exposed systems
    Why test it? — These are our most exposed systems. They are what our attackers are going to hit first.
  2. What to test? — Employee responses to social engineering, because our employees are the ones who will protect us from phishing attacks.
    Why test it? — Phishing attacks are frequent and successful. They are hurting other people and happen all the time.

TIP: Test against common attacks. Attackers stick with what works. We're tempted to say "It's so common, let's look for the next big thing instead." Attackers will stick with what works as long as it works. So if you're hearing about things in the news, like ransomware and you're worried about it, then that is a valid threat to test against. It will continue to happen until it's no longer profitable for the attacker.

Test against common attacks. Attackers stick with what works as long as it works.

Whether your risk assessment is a formal or informal process is not as important as the results. You need to identify the areas where you are exposed, where your most critical assets are, and where attacks are most likely to occur. These decisions come from knowing your organization and knowing what types of attacks are common. We see these attacks in the news and other places all the time.

Determine the testing you need

Going back to our simple risk assessment example, we can take the assets we identified in the first step and translate them into the type of testing we need. In this case, we decided on email social engineering for all employees and Internet perimeter testing for all of our public IP addresses.

Be specific!

It is important to be very specific when communicating to your penetration testing company what type of testing you want performed. In a minute, we will look at what could happen if we don't communicate clearly.

Choosing a Pen Test Provider

Now that we know what we want to test, we need to define the scope of the testing. Do we want to test just our web servers or our entire Internet exposure? Do we want to test a specific group of employees or are we specifically concerned about a certain department that performs risky tasks – maybe they initiate wire transfers for our company.

In this example, we will define the scope as:

  • All public IP addresses
  • All employees

TIP: Include ALL external IP addresses, active and inactive. You never know when an inactive IP address might become active. Mistakes and misconfigurations happen and attackers are looking for them.

This scope can be modified over time. Maybe we want to test all employees the first time and then focus on a risky department the next time.

What can happen if you don't communicate clearly?

Check out what happened when pen testing services were requested by Iowa State Court Officials. As reported in this news article, the scope of testing was defined as "test the security of the court's electronic records . . . through various means." Not specific enough! It only defined an asset, not what the organization was worried about or what they wanted tested. The result: two pen testers were arrested in Adel, Iowa for attempting to physically break into the court house. The State's response: "[we] did not intend, or anticipate, those efforts to include the forced entry into a building."

In other words, if we want our employees tested against phishing and our Internet perimeter tested against a remote attacker, we don't want the pen testers to do this to our front door:

Unrelated to the Iowa incident, this gentlemen is well-known for his pen testing of physical controls, but would you want this sort of testing when you are concerned about ransomware coming in through phishing emails?

Set the Rules of Engagement

The final step in our pen test selection process is to set the boundaries for our pen testers to follow. Remember that we get to choose WHAT they target, but they get to choose the attacks that would be most effective – what the real attacker might choose. By settings boundaries on those attacks or rules of engagement, we can ensure the pen test won't go too far and cause harm.

In this step you will agree on:

  • What the pen testers WILL attempt
  • What the pen testers WON'T attempt

"The test mimics a threat source's search for and exploitation of vulnerabilities to demonstrate a potential for loss." - FFIEC IT Exam Handbook, Information Security Booklet, Sep 2016 (emphasis added)

Going back to our regulatory reference, these boundaries are stated as "demonstrating the potential for loss." Notice it doesn't say "demonstrating the loss," but rather the potential for loss. This is an important word to highlight because it identifies the need for pen test boundaries. We want our pen test firm to show us how an attack could happen and how it could negatively impact our organization, but we don't want them to go so far as to actually cause loss or have a significant negative impact.

Here's an example set of the rules of engagement that CoNetrix will use. Because we typically test live, operational systems, we put certain rules into place to make sure the testing stays within bounds.

Rules of Engagement
  • Do no harm. It's not a good penetration test if the penetration test company leaves you more vulnerable than you were when you started. What would that look like? If they went into a system and they installed malware and left that malware sitting there and didn't tell you they installed it. That malware may be opening a door to the internet that you didn't have before. If the pen testers create new exposure, then it would leave you worse off than when you started.
  • No significant customer impact. You don't want customers calling in saying: "I can't get into my internet banking account," "I can't log in to this website," "the website is down." The penetration test company needs to be very aware of who your customers are and how they use your systems. They need to make sure that everything they do does not impact the customer. Some tests can't be performed unless there is some customer impact. How would a penetration company handle that? The way it should go is: 1) Identify the vulnerability and the potential exploit for that vulnerability. 2) Contact the customer and let them know about it. If they would like us to perform the exploit, inform them it may cause some downtime and it might affect their customers. 3) Coordinate with the customer on when to perform the exploit testing. Maybe we do it after hours. Or it may be they are great with just knowing the vulnerability exists and don't need the exploit performed due to the possible negative impact.
  • No unplanned operational impact. A penetration test company may discover that your system or website is extremely vulnerable to a certain kind of denial of service attack. If exploited, it could be taken down for hours on end. That's great to know, but the penetration test company should not exploit that vulnerability without coordinating with you first. That would cause unplanned operational impact and now your IT department is scrambling and your customers or your internal staff are upset because they can't do their job or access your services.
  • Limited system recovery time/money. This is an important one and not always obvious. If you're not careful, a penetration tester that is not very good at what they're doing could say "well, we could do this and they would just have to restore from backup". That's easy for the penetration tester to say, but they aren't the ones who have to do the work to restore from backup. A lot of times restoring from backup can be a complicated process and can take a while. It might cost a lot of time and money, and that may not be obvious to a penetration tester. Just having a dedicated person to go recover a system can be a big impact to other projects and tasks you have going on. If an exploit is going to take significant effort or money to recover from, then that needs to be out of bounds, unless you have coordinated about it ahead of time.
  • Attempted exploits need to provide value. Penetration testing can be flashy and glamorous. It shows up in movies, TV shows, YouTube, etc. Make sure that the exploits that are being performed are providing value. Flashy and glamorous is fine for the movie screen, but real world pen testing must provide value to your company by helping improve your security posture.
  • If an exploit might break the rules, report the vulnerability. A penetration test company may say they found this vulnerability, but if exploiting it is going to break the rules, then it would be better to just report the vulnerability and say what needs to be fixed. It's better to do that than to cause your customers and staff a lot of grief. The vulnerability can be talked about and remediated without causing problems.

These are the rules of engagement we practice here at CoNetrix when performing penetration tests.

Evaluate a Pen Test Provider

So let's wrap up how to choose a pen test provider with some ways you can evaluate pen testers and then we'll move on to the fun stuff – the exploits!

  1. Know what you need tested and clearly communicate that in the project scoping and quoting process. We talked about identifying what you need tested. Make sure the pen test company's expertise matches with what you need tested. As we saw earlier, some pen testers excel in physical security testing. Others excel at Internet-based testing. Some are industry-specific such as SCADA pen testers. Make sure you find out what their strengths are.
  2. Make sure their report is understandable and well-organized. Many pen testers are great at successfully demonstrating attacks. After all, that's the fun part – the part that gets put into the movies. Buy many don't put as much effort in communicating what they did and how you can improve your defenses. Make sure the format of the report and the way information is communicated in the report is going to help you improve your security. Is it actionable for you and your IT/security staff?
  3. Check out their certifications on top of experience. Certifications are a great way of making sure a pen test company has the knowledge and skills necessary. Some examples include Certified Ethical Hacker (CEH), Offensive Security Certified Professional (OSCP), and any certification that shows they have knowledge of the area you need tested.
  4. Make sure the penetration test company will help you understand the report. We believe security testing is most useful when it is a partnership, not a stand-alone service. Ask how the pen test company will help you understand what the issue is and what you can do about it. Do they review their results with you? Do they provide remediation recommendations? Or do they drop a bunch of results on you and wish you good luck.

Phases of Pen Testing

So now let's talk about the fun side of pen testing—exploitation. It's good to know that exploitation is part of the process. Many times it is because of this process that a penetration test report may look like a vulnerability assessment. Every pen test process follows a similar process—identify how the target is exposed—identify vulnerabilities that could be attacked—exploit those vulnerabilities—report both the vulnerabilities and any successful exploits.
But what if no exploits were possible—at least not within the rules of engagement? In that case, the report would only identify vulnerabilities and could look a lot like a vulnerability assessment. But through your pen test selection process, you made sure the firm would exploit vulnerabilities when safe and possible and their documented methodology should reflect that approach.
Phases of Pen TestingOne of the best things you can do when evaluating a pen test company is ask them for examples of what they have done. See how they did it and whether you're comfortable with the things they are capable of doing.

One of the best ways to evaluate a pen test company is to ask for some examples of their successful exploits.

*FFIEC IT Exam Handbook, Information Security Booklet, Sep 2016
**This blog post is for information purposes only. Evaluate your risks before acting on any ideas presented in this article.


On April 2, 2019, the Federal Deposit Insurance Corporation (FDIC) released a new financial institution letter (FIL-19-2019) called "Technology Service Provider Contracts."

Why was this guidance published?

When FIL-19-2019 was published, it had been five years, almost to the date, since the last vendor management guidance was released by the FDIC (see FIL-13-2014, published on April 7, 2014). Presumably, it was a good time for a reminder about vendor management expectations.

In addition, the guidance stated FDIC examination findings recently noted some financial instruction contracts with Technology Service Providers (TSP) lack of sufficient detail around business continuity and incident response. 

What does it mean when the guidance states "contracts do not adequately" address some risks?

In recent exams, the FDIC was looking for a few key areas to be covered in TSP contracts, but the contracts did not always meet those expectations. Missing items included:

  • A Business Continuity Plan (BCP): Contracts should require TSPs to have BCP and acceptable recovery standards.
  • Remedies: Contracts should include assurance of compensation if a business disruption occurs and the TSP fails to restore services in the established timeframe.
  • Notification Requirements: Contracts should define who the TSP should contact (e.g., the financial institution, regulators, law enforcement, etc.) and in what timeframe, if an incident occurs.
  • Key Terms: Contracts should define what constitutes a "business disruption" or an "incident," since rights and responsibilities could be debatable without clear definitions.

How can you ensure TSP contracts are "adequate?"

It would be beneficial for you to review your TSP contracts again with these items in mind, especially if they are long-term or automatically renewing contracts. If your existing contracts are not sufficient in these areas, it is important to note that the financial institution is still responsible for assessing and applying controls to mitigate the risk.

What controls can you apply to ensure you are covered?

In vendor management, your primary control is performing adequate oversight, which is something you should already be doing. The FDIC seems to recognize this since a significant percentage of the FIL recaps guidance that already exists.

For more specific recommendations though, if your contract with a TSP does not clearly define business continuity and incident response requirements:

  • Request and Review Their BCP: Find out if your TSP actually has one and if they'd be willing to share it with you. You don't necessarily need their whole BCP; you just need to know that they have a plan and it is routinely tested.
  • Update Your BCP: If the TSP does not have a BCP or you find it inadequate, it is the financial institution's responsibility to compensate. Update your BCP to describe how you would continue to offer services to your customers or members if your TSP's services are unavailable.
  • Conduct More Frequent Reviews: Whatever the contract says, it is important to periodically confirm the TSP is holding up their end of the deal. You may want to assess this more often if the contract is weak in the areas of business continuity and incident response.
  • Renegotiate the Contract: Depending on the financial institution's risk tolerance, if the contract is deemed "inadequate," it may benefit the financial institution to consider renegotiation or an alternative TSP.

In Summary

Contracts with TSPs should address business continuity and incident response. The FDIC recommends financial institutions contractually require the TSP to have a BCP, as well as contractually define remedies, notification requirements, and key terms.

If existing TSP contracts do not stipulate these items, you should consider additional oversight options, such as requesting and reviewing their BCP documentation, updating your BCP, reviewing the TSP more frequently, or renegotiating the contract.

Does CoNetrix have anything that can help with this?

Absolutely. The Tandem Vendor Management software includes suggested significance questions, designed to help you determine if you need BCP documentation from your vendors. The module also includes a contract review template, designed with business continuity and incident response in mind. Learn more about Tandem Vendor Management.


Today the Federal Financial Institutions Examination Council (FFIEC) issued an update to the FFIEC IT Handbook, BCP Booklet.  The update included a new appendix entitled Strengthening the Resilience of Outsourced Technology Services.  The appendix highlights and expands on the BCP Booklet in four specific areas: third-party management, third-party capacity, testing with third-party technology service providers, and cyber resilience.  To learn more, visit  


CoNetrix is pleased to announce the release of Tandem, new security and compliance software. Tandem was developed to help financial institutions complete and maintain an Information Security Program (per GLBA and the Interagency Guidelines Establishing Information Security Standards).  While Tandem was designed as a complete solution from the ground up, it was fashioned into modules which allow for versatility.  The modules include risk assessment, policies, vendor management, and business continuity planning.  Each module was released as it was completed.

To read the full press release, visit