From Assumption to Assurance: Why Independent, Authenticated Audit Data Collection Matters

User access and system activity logs

This data shows who logged in, when, and from where, as well as what happened on a system, including background processes and events not directly related to user activity.

Issues with logging controls can easily go unnoticed for long periods of time, as systems may continue to operate effectively in every other way. An organization may not realize these critical logs are nonexistent until a system failure, security breach, or other incident reveals the gap. 

Group memberships and role assignments 

This data shows who has access to which systems or data including who has elevated privileges. 

Roles, groups, and privileges change regularly. It's easy for access levels to drift, especially when someone changes roles, leaves the company, or when a new vendor comes on board. Each of these changes can inadvertently result in incorrect group memberships or access rights if not carefully managed. 

Legitimate temporary changes can easily become not-so-temporary. For example, if an employee is having technical issues, the IT team might grant them elevated privileges to help with troubleshooting. But once the issue is resolved, those privileges don't always get removed as they should. 

Configuration settings 

This data shows operating system and application settings that impact security. Hardware and software vendors are constantly releasing updates, adding features, fixing bugs, and patching vulnerabilities, all of which can change configuration settings. 

New security settings introduced in these updates don't always get enabled. This is largely due to a lack of awareness about the changes or the existence of the new settings. 

Patch status reports 

This data shows information about missing or out-of-date security updates. 

When auditors find missing critical security patches, this should spark a helpful conversation about the root cause. In some cases, the patch management system may not have been configured to report on all types of updates. In others, certain systems or applications may have been unintentionally excluded from the reporting process altogether. 

Firewall and network device configurations 

This data shows the rules and policies that control how traffic flows into, out of, and through the network. 

Temporary firewall rules are sometimes left in place permanently. For example, a rule might be added that allows unrestricted internet access to a system during testing, but after the testing ends, the rule is not removed or restricted. 

Endpoint protection status 

This data shows whether endpoint protection systems (e.g., EDR, anti-malware, etc.) are installed and functioning. 

Individual workstations sometimes fail to check-in with the endpoint security server. These workstations often go unnoticed because they are not reporting an issue and the reporting software does not flag when the system has stopped communicating. As a result, these workstations can miss important updates or fall out of compliance. 

Vulnerability scan results 

This data shows the status of unresolved vulnerabilities. 

Managing vulnerabilities across IT systems can be overwhelming with hundreds (or even thousands) of issues continuously emerging. An effective vulnerability management program relies on risk-based decision-making: determining which vulnerabilities to fix, which to mitigate through alternative controls, and which to accept.  

The goal of running vulnerability scans during an audit isn't simply to point out vulnerabilities, but to assess whether the organization's risk-based decisions are resulting in the level of risk reduction they expect. Performing an independent vulnerability scan in an IT audit can bring valuable clarity to the effectiveness of what is often a complex and nuanced process. 

Other data types 

Depending on the nature and scope of the audit, other data types may be collected, as well. For example: 

• Audit trails from key applications which show logs of data access, changes, and user activity. 
• Backup records which show evidence that data is being backed up and would be recoverable, if needed. 
• Security system alerts which show key security alerts are configured and are being delivered, as expected, from applicable systems (e.g., SIEM systems, EDR / XDR systems, email systems, etc.). 

Data Types 

Why Authenticated Audit Data Collection Matters 

User access logs; group memberships and role assignments 

It goes beyond just looking at exported or reported values. It reveals real-time user permissions and roles. 

Configuration settings 

It goes beyond just looking at the procedures for setting up a new system. It allows direct access to system and application settings to verify actual configurations and find areas where configurations have changed. 

Patch status reports 

It goes beyond relying on reports that may not be complete. It verifies installed patches directly on a sample of systems selected by the auditor. 

System activity logs; firewall and network device configurations; audit trails from key applications; backup records; security system alerts 

It goes beyond reviewing a sample from the system or control owner. It ensures the logs are coming from trusted, original sources, they are being collected and stored properly, and they haven't been tampered with. 

Endpoint protection status 

It goes beyond reviewing the same reports your IT team, MSP, and/or MSSP are already reviewing. It ensures required endpoint security tools (e.g., EDR, anti-malware, etc.) are installed and function correctly on the systems your staff use daily. 

Risk 

Response 

Privileged access. Giving external auditors or tools privileged access can increase exposure. 

Grant least privilege access. Ideally, audit data collection should be done with read-only access, since auditors don't administer the system; they just need to collect data. However, the level of access required varies across systems. For example, Microsoft Active Directory only allows audit data collection with Domain Admin privileges. On the other hand, Microsoft's cloud products have made significant strides in allowing audit data collection through read-only access. The goal is to balance collecting high-quality audit data while granting the least privilege necessary to minimize risk. 

Use a dedicated account. The account used for data collection should be a dedicated account and should not be used for any other activities. This allows you to manage security settings of the account, enable/disable the account as needed, and maintain logs related to the account's activity. 

Disable accounts when not in use. Audit data collection is a relatively short-lived process, especially when CAATs speed up the process. This means the window of time privileged access is needed can be short, sometimes days or only hours. As soon as the data collection is complete, disable the accounts until the next audit. 

Do good vendor management. Independent testing should be done by reputable firms with experienced personnel. Assess the audit firm's qualifications, review their testing procedures, and negotiate a strong contract. Granting privileged access doesn't remove accountability. Instead, it provides the necessary visibility to ensure the audit delivers real value. 

Vulnerable technology. The tools used for testing might be vulnerable or require network changes to use. 

Don't relax your security posture. Authenticated testing may require privileged access and treating the audit tools as trusted, internal assets. But the audit should use credentials you control and follow your change management, authentication, and monitoring processes.

Do good vendor management. Just like with any vendor who connects to your network, make sure the vendor is reputable, and their testing tools are patched, secured, and subject to your due diligence process. 

Operational downtime. Testing could disrupt production systems or slow down operations. 

Coordinate testing with the audit firm. Testing should always be carefully planned and managed. This could include off-hours scheduling, controlled scan settings, and clear communication throughout the process. Reputable audit firms know how to test without disruption and know your priority is keeping the organization running. 

Vendor compromise. Audit firms are attractive cyber targets. If their tools or systems are compromised, they could become a conduit to your network. 

Do good vendor management. You can and should ask about the audit firm's own security posture. Just like any vendor, they should meet your due diligence requirements. Disable their access when it is not in use and perform ongoing monitoring to ensure continued compliance with contractual obligations. 

Realistic testing. Unauthenticated scans reflect more realistic attack conditions. Internal security tools may block testing or flag it as malicious. 

This should not be an either/or decision. Both unauthenticated and authenticated testing provide value. One shows what an outsider sees (internal and external penetration tests); the other shows what's happening inside the house, to continue our analogy (IT audits). Confusing the two decreases the value of both. 

Audit data collection needs to be thorough to verify control effectiveness. Blocking this data collection by treating it the same as malicious activity decreases the auditor's ability to give you what you need: independent verification. 

In contrast to an audit, internal and external penetration testing should be limited by all your controls and treated like malicious activity because the goal is to simulate an attack. 

Third-party testing coverage. Relying solely on MSP-provided SOC reports or scan results may leave gaps in assessing the security of the systems or controls they manage on your behalf. 

Get the full story. SOC reports and vulnerability scans from your MSP are a valuable part of the picture. They show how well the MSP secures their own infrastructure, but they don't always cover the security of your hosted systems and network devices. For example, a SOC report might assess how the MSP patches their own servers, but not necessarily the ones they manage for you. That's why it's important to ensure your own controls are independently audited. 

Do good IT audit scoping. To ensure complete coverage, clarify with your MSP which controls are being independently audited. Ask questions like: 

• Are admin and user accounts in my Active Directory domain being independently audited? 
• Are patches and updates for my servers, workstations, and network devices being independently audited? 
• Are endpoint protections on devices used by my employees (e.g., workstations, laptops, tablets, etc.) being independently audited? 
• If "Yes," how and by whom?