Securiment Blogs

Filtering Custom Logs and Optimizing Event Monitoring using DCR and XPath Query

In the domain of cybersecurity, effective data collection and log management are important for detecting and responding to potential threats. This blog post delves into the utilization of AMA for data collection in Azure Sentinel, focusing on filtering custom logs using XPath queries, sending logs to destinations using Data Collection Rules (DCR), and best practices for collecting crucial event IDs. My aim is to provide actionable tips, real-life examples, and expert insights that will aid readers in optimizing their security operations while managing costs effectively.

Understanding Data Collection with AMA

What is AMA?

Benefits of Using AMA

• Centralized Management: AMA allows for the centralized management of data collection policies across multiple machines.

• Flexibility: It supports various configurations tailored to specific monitoring needs and environments.

• Security: Enhanced security features ensure that data is collected and transmitted securely.

Filtering Custom Logs with XPath Query

XPath Query Basics

XPath queries are used to filter logs based on specific criteria, ensuring that only relevant data is collected. This not only optimizes performance but also reduces costs by minimizing the volume of data stored and processed.

Examples of an XPath Query

Creating and Testing XPath Queries

• Event Viewer:

Use Windows Event Viewer to create and test XPath queries. This tool allows you to specify the event IDs and other criteria, and it generates the corresponding XPath query Open the Event Viewer and select the log file. Choose Filter Current log and enter the Event IDs you want to collect. Click on XML for opening the Xpath structure.

• PowerShell:

Test the validity of an XPath query locally using the Get-WinEvent cmdlet with the FilterXPathparameter.In PowerShell, you can filter logs further based on specific criteria. This capability becomes particularly useful when you want to refine your data collection for subsequent processing. For instance:

Sending Logs to Destination Using DCR

What is a DCR?

A Data Collection Rule (DCR) in Azure defines how data should be collected, transformed, and sent to a destination, such as a Log Analytics workspace.

Creating a DCR

Define the Rule: Specify what data to collect, which can include logs from Windows Events, Syslog, or performance counters.

• Open Microsoft Sentinel

• In the menu select Data connectors (1)

• Select the Windows Security event via AMA connector (2) 

• Open the connector page (3)

Now from the connector page configure the new data sources. Make sure you have read and write permissions. For collecting security events from Windows agents and installing the AMA agent. Start with creating a new data collection rule (DCR). For creating the new rule click the button Create data collection rule

The Data Collection Rule determines the destination for data transmission. In this blog, we utilize the Microsoft Sentinel Log Analytics workspace for this purpose. Add the following details and click create:

• Rule Name: Name for the specific Data Collection Rule

• Subscription: Choose the appropriate subscription with region.

• Resource Group: Select the relevant resource group & platform type.

For the remaining steps, you can follow a step-by-step approach for how to configure a data collection rule (DCR) from Microsoft documentation(Configure DCR). However, today we are more focused on custom logs filtering using DCR, so let's skip ahead to the last step where we tell the DCR how to collect and filter logs using an XPath query.

For collecting events select one of the event groups:

• All Events: This category encompasses all Windows security and AppLocker events.

• Common Events: This standard collection includes essential events for auditing purposes, providing a comprehensive user audit trail. It covers activities such as user sign-in and sign-out (event IDs 4624, 4634), security group modifications, key domain controller Kerberos operations, and other events aligned with best practices. While primarily designed to streamline event volume, it still ensures complete audit trail capability, even including less frequent events.

• Minimal Events: This minimalistic set focuses on potential threat indicators without providing a full audit trail. It concentrates solely on events signaling a possible breach and other critical occurrences with low frequency. For instance, it includes successful and failed user logons (event IDs 4624, 4625) but excludes sign-out information (4634), which, though significant for auditing, isn't as pertinent for breach detection and tends to generate higher volume. Most data within this set pertains to sign-in events and process creation events (event ID 4688).

• Custom Events: This category enables users to define their own event sets through a data collection rule utilizing XPath queries. Users can tailor this set to specific needs and preferences.

Create Custom Collection:

Now this is the step where we can finally implement XPath queries to create custom collections and the event of interests. After finalizing the setup, you can change the rule with more events of different types. The selection custom enables the custom event configuration.

• Collect only event 4625 (failed sign-in): Filters to include only security events with Event ID 4625, which indicates a failed sign-in attempt.

Query: Security!*[System[(EventID=4625)]

• Collect event 4625 (failed sign-in) and 4624 (Successfully logged on): Filters to include security events with Event IDs 4624 and 4625, representing successful and failed sign-in attempts, respectively.

Query: Security!*[System[(EventID=4624) or (=4625)]

• Collect event 1, 299, 4624, 4625, 4661, 4662, 4663, 4664, 4665: Filters to include security events with any of the specified Event IDs, covering a range of system and security-related activities

Query: Security!*[System[(EventID=1) or (=299) or (=4624) or (=4625) or (=4661) or (=4662) or (=4663) or (=4665)]

• Collect all critical, Error, Warning, and Information events from the system log except Event ID 6: Filters to include all events from the system log with severity levels of critical, error, warning, and information, excluding those with Event ID 6.

Query: System!*[[(Level=1or Level23) and (EventID!=6)]]

For validating rules before adding in Sentinel, PowerShell can be used for validating XPath queries. use the Get-WinEvent cmdlet to validate the XPath query.

$XPathQuery = "*[System[(EventID=4625)]]"Get-WinEvent -LogName Security -FilterXPath$

Best Practices for Collecting Event IDs

To optimize costs and performance in Azure Sentinel, it's essential to collect only the necessary events that are relevant to your monitoring objectives. This can be achieved by using XPath queries to specify the events precisely.

• Collect Only Necessary Events

To collect only necessary events, you should focus on specific Event IDs that are relevant to your security monitoring objectives. For example, if you're interested in monitoring for failed logins, you can use an XPath query like this: Security*[System[EventID=4625]]. This query collects only events with 4625, which represents a failed sign-in attempt.

By collecting only these specific events, you can reduce the volume of data and optimize your costs. While the specific event IDs to monitor may vary depending on your organization's environment and security requirements, there are certain critical events that should be prioritized for collection and analysis. These include, but are not limited to:

• Failed logon attempts (Event ID 4625): Monitor for potential brute-force attacks or unauthorized access attempts.

• Successful logon events (Event ID 4624): Track user activity and identify potential account compromises.

• Process creation events (Event ID 4688): Detect the execution of suspicious or malicious processes.

• File creation/modification events (Event IDs 4663, 4660): Monitor for potential data exfiltration or malware activity.

• Firewall and network events: Capture events related to network connections, blocked traffic, and firewall policy changes.

By focusing on these critical event IDs, you can gain valuable insights into potential security incidents, enabling timely detection and response.

Regular Review and Update

Regularly reviewing and updating your collection policies and XPath queries is crucial to adapt to changing security landscapes and operational needs. This ensures that your monitoring setup remains effective and efficient over time.

For instance, if you initially collect events for failed logins (EventID 4625), you might need to update your query to include additional Event IDs as new threats emerge. You could add more Event IDs to the query like this:Security*[System[(=4624 or =4625 or =4626)]]

This updated query collects events for successful logins (EventID 4624), failed logins ( 4625), and other relevant events ( 4626). By regularly reviewing and updating your queries, you can ensure that your monitoring setup remains effective and efficient in response to changing security threats.

Conclusion

Effective data collection and log management are critical components of a robust security strategy. By leveraging Azure Sentinel's capabilities, such as XPath queries and Data Collection Rules, organizations can optimize their log collection processes, reduce operational costs, and enhance the efficiency of their security monitoring efforts. By following best practices and focusing on critical event IDs, security teams can gain valuable insights and improve their ability to detect and respond to potential threats. Remember, the key to success lies in striking the right balance between comprehensive log coverage and selective filtering to ensure that only the most relevant data is collected and analyzed. By implementing the techniques and best practices outlined in this blog post, you can embark on a journey towards a more efficient and effective security operations center, empowering your organization to stay ahead of evolving cyber threats.