Securiment Blogs

Developing ASimCEFAuthenticaion Parser for Custom Application Sending Events in CEF Format

Introduction

To begin with the Parser development firstly, our goal here is to standardize the logging format for Custom Developed Applications.

Most network and security systems support either Syslog or CEF (which stands for Common Event Format) over Syslog as means for sending data to a SIEM. Azure Sentinel provides the ability to ingest data from an external solution. If your appliance or system enables you to send logs over Syslog using the Common Event Format (CEF), the integration with Azure Sentinel enables you to easily run analytics and queries across the data. This makes Syslog or CEF the most straightforward way to stream security and networking events to Azure Sentinel.

The advantage of CEF over Syslog is that it ensures the data is normalized making it more immediately useful for analysis using Sentinel.

Custom Application Monitoring via CEF

We have defined CEF-based application logs monitoring solution by standardizing fields and values. For example, an Account Created Event should include the fields and values shown below.

Creating Sample Events to Ingest in MS Sentinel Environment

To ingest events we’ve created a python script to generate events that we defined earlier in the exact format which simulates events coming from an actual log source.

Python script to generate Login Success event.

#!/usr/bin/python

# Python script designed to write to the local Syslog file in CEF format on an Azure Ubuntu 18.04 VM.

# Laraib Khan Nov 2022

# Importing the libraries used in the script

import random

import syslog

import time

from datetime import datetime

# Simple list that contains usernames that will be randomly selected and then otput to the "duser" CEF field.

usernames = ['Frank', 'John', 'Joe', 'Tony', 'Mario', 'James', 'Chris', 'Mary', 'Rose', 'Jennifer', 'Amanda', 'Andrea', 'Lina']

outcomes = ['success', 'failure']

dhost = 'scals.securiment.com'

#Device Time

now = datetime.now()

current_time = now.strftime("%H:%M:%S")

# Endless loop that will run the below every five minutes.

while True:

# Assigning a random value from the above lists to the two variables that will be used to write to the Syslog file.

selected_user = random.choice(usernames)

event_outcome = random.choice(outcomes)

# Assigning a random integer value from 1-255 that will be appended to the IP addresses written to the Syslog file.

ip = str(random.randint(1,255))

ip2 = str(random.randint(1,255))

# The User Authentication Succeeded Syslog message that will be written.

syslog_message6 = "CEF:0|Securiment|SCALS|1.0|1005|User Authentication Succeeded|10|src=10.0.10.1 dst=10.0.0.19 duser=farhan Outcome=success cat=Application Generic act=LoginSuccess dhost=scals.securiment.com rt=" + current_time + " dvc=10.1.1.19 dtz=utc dvchost=scals.securiment.com spriv=User"

# Writing the event to the Syslog file.

syslog.openlog(facility=syslog.LOG_LOCAL7)

syslog.syslog(syslog.LOG_INFO, syslog_message6)

# Pausing the loop for x second.

time.sleep(300)

# End of script

Once the script is executed on a Linux machine. OMS agent sends the event log to MS Sentinel where it populates in the CommonSecurityLog table.

Sample LoginSuccess event:

Python script to generate Login Failure event.

#!/usr/bin/python

# Python script designed to write to the local Syslog file in CEF format on an Azure Ubuntu 18.04 VM.

# Laraib Khan Nov 2022

# Importing the libraries used in the script

import random

import syslog

import time

from datetime import datetime

# Simple list that contains usernames that will be randomly selected and then otput to the "duser" CEF field.

usernames = ['Frank', 'John', 'Joe', 'Tony', 'Mario', 'James', 'Chris', 'Mary', 'Rose', 'Jennifer', 'Amanda', 'Andrea', 'Lina']

outcomes = ['success', 'failure']

dhost = 'scals.securiment.com'

#Device Time

now = datetime.now()

current_time = now.strftime("%H:%M:%S")

# Endless loop that will run the below every five minutes.

while True:

# Assigning a random value from the above lists to the two variables that will be used to write to the Syslog file.

selected_user = random.choice(usernames)

event_outcome = random.choice(outcomes)

# Assigning a random integer value from 1-255 that will be appended to the IP addresses written to the Syslog file.

ip = str(random.randint(1,255))

ip2 = str(random.randint(1,255))

# The User Authentication Failed Syslog message that will be written.

syslog_message7 = "CEF:0|Securiment|SCALS|1.0|1006|User Authentication Failed |10|src=10.0.10.1 dst=10.0.0.19 duser=farhan Outcome=failure cat=Application Generic act=LoginFailure dhost=scals.securiment.com rt=" + current_time + " dvc=10.1.1.19 dtz=utc dvchost=scals.securiment.com spriv=User"

# Writing the event to the Syslog file.

syslog.openlog(facility=syslog.LOG_LOCAL7)

syslog.syslog(syslog.LOG_INFO, syslog_message7)

# Pausing the loop for 6 minute.

time.sleep(10)

# End of script

Sample LoginFailure event:

Now we have both Authentication Events so it’s time to create an ASim function.

I named the function ASimCEFAuthentication, and that can be further called in a union parser like ASimAuthenticaion to aggregate.

CommonSecurityLog

| where DeviceEventClassID has_any (1005, 1006)

| extend

EventVendor = 'Securiment'

, EventProduct = 'SCALS'

, EventCount=int(1)

, EventSchemaVersion='0.1.0'

, EventResult = EventOutcome

, EventOriginalResultDetails = coalesce(SimplifiedDeviceAction, Activity)

, EventStartTime = TimeGenerated

, EventEndTime= TimeGenerated

, EventType= 'Logon'

, SrcDvcId=DeviceVersion

, SrcDvcHostname =DeviceName

, SrcDvcIpAddr=SourceIP

| project-rename

EventOriginalUid =DeviceEventClassID

, TargetUserId = SourceUserID

, TargetUsername=SourceUserName

, TargetUserType=SourceUserPrivileges

| project-reorder

TimeGenerated

,EventProduct

, EventOriginalUid

, EventResult

, EventOriginalResultDetails

, EventStartTime

, EventEndTime

, TargetUserId

, TargetUsername

, SrcDvcId

, SrcDvcHostname

, SrcDvcIpAddr

As we can see events are filtered prior to parsing using the DeviceEventClassID value, which we already set in the Custom Application Security Log Monitoring Solution. i.e. 1005 for LoginSuccess and 1006 for LoginFailure.

Conclusion

Using the DeviceEventClassID values for login events, we can filter events that are coming from other applications having different values in the same parser.