In the first part of this series, I highlighted the critical role of the Linux Audit Daemon (auditd) in maintaining system security by recording detailed logs of system events. This essential functionality makes auditd a primary target for attackers who seek to obscure their malicious activities.

While my previous discussion focused on ensuring auditd’s continuous operation, this article delves into detecting the deletion of auditd rules and changes to its configurations. Ensuring the integrity of these settings is crucial for effective security monitoring.

In this article, I will cover the importance of detecting unauthorized changes to auditd rules and configurations. I’ll introduce methods and tools, including auditd rules and Splunk queries, that can help identify these changes and ensure the reliability of your audit logs.

By the end, you’ll be equipped with the knowledge and tools necessary to enhance your system’s security by detecting tampering with auditd rules and configurations.

The problem we aim to solve

Attackers often delete auditd rules or modify configurations to disable logging and hide their activities. Detecting these unauthorized changes is challenging due to the various methods attackers use.

Our goal is to ensure the detection of any unauthorized deletions or modifications to auditd rules and configurations. By implementing effective detection mechanisms, we can maintain the reliability of audit logs and enhance system security.

The Solution: Detecting Deletion and Monitoring Changes in Auditd Rules and Configurations

Our solution involves two main components to address the challenges posed by attackers targeting auditd:

1. Detecting Deletion of Auditd Rules: Utilizing audit record types logged by auditd, such as CONFIG_CHANGE, we will implement monitoring to detect any deletions of auditd rules. These record types provide built-in logging of rule deletions, allowing us to proactively identify and respond to unauthorized changes that could compromise system security.
2. Monitoring Changes in Auditd Configurations and Critical Files: We will implement new auditd rules to monitor changes to critical auditd configurations. These rules will capture modifications or deletions of essential configuration files, ensuring that any unauthorized changes are promptly flagged for investigation.

Understanding the Log Activities

Auditd logs CONFIG_CHANGE events whenever there is a modification to the audit system configuration, including changes to audit rules. This record type captures details such as the timestamp and nature of the modification, making it ideal for monitoring and detecting unauthorized deletions of auditd rules.

Reviewing the logs

To validate our correlation search for detecting the deletion of auditd rules, I first conducted a rule deletion simulation to generate the necessary logs in Splunk.

It’s very easy to spot the pattern here. We should look for type=CONFIG_CHANGE and op=remove_rule.

Sample audit logs generated by the rule deletion activity

Developing the detection for deleted auditd rules in Splunk

Disclaimer: The logged activities to test the detection rules were performed on a real Linux server, simulating common methods used by malicious actors to delete auditd rules or modify auditd configurations.

While this approach covers typical tactics, other methods may also exist. In large corporate environments, diverse configurations might cause false positives or negatives, necessitating fine-tuning of the detection rule.

The detection rule and the correlated event

Explanation:

1. Search Criteria (index, sourcetype, type, op, res):

   •  Searches the linux_audit index for linux:audit events where type is CONFIG_CHANGE, op is remove_rule, and res is 1 (indicating success).

2. Field Evaluation (eval key):

   •  Sets key to “unknown” if it is “(null)”, ensuring field consistency.

3. Aggregation (stats values(key) AS deleted_rules):

   •  Aggregates events by _time, host, type, auid, ses, and op, collecting all key values (names of the deleted auditd rules) into deleted_rules.

4. Time Conversion (convert ctime(_time)):

   •  Converts the Unix timestamp in _time to a human-readable date and time format.

5. Counting Deleted Rules (eval count=mvcount(deleted_rules)):

   •  Counts the number of deleted_rules occurrences.

6. Event Description (eval event_info):

   •  Constructs event_info based on the count of deleted_rules.
   •  Specifies the number of auditd rules deleted and the timestamp.
   •  Recommends investigation due to the unexpected nature of rule deletions.

7. Final Output (table _time, host, type, auid, ses, op, deleted_rules, event_info):

   •  Presents results in a structured table format, including columns for timestamp (_time), host, event type (type), audit user ID (auid), session ID (ses), operation (op), deleted rules (deleted_rules), and event details (event_info).

This detection rule effectively identifies and reports instances where auditd rules have been deleted, providing timely alerts for potential security incidents.

Developing the detection for auditd configuration modifications/deletion

Step 1: Create auditd rules for monitoring configuration modifications

The auditd rules we are setting up are strategically designed to monitor key configuration files essential to the Linux auditing infrastructure. These files include /etc/audit/auditd.conf, /etc/audit/rules.d/test.rules, /etc/audisp/audispd.conf, and /etc/libaudit.conf. Here’s why each is crucial:

   •  /etc/audit/auditd.conf: Controls global settings for the audit daemon (auditd), including log location, retention policies, and system-wide audit configurations.
   •  /etc/audit/rules.d/test.rules: Houses audit rules defining specific events and conditions for monitoring.
   •  /etc/audisp/audispd.conf: Configures the audit event dispatcher (audispd), responsible for processing and forwarding audit events.
   •  /etc/libaudit.conf: Manages library-level settings for the audit framework (libaudit), influencing its behavior and functionality.


Auditd rules monitoring for configuration modifications

The logic of these auditd rules remains consistent across the different configuration files:

   •  Each rule uses the -a always,exit option to ensure that audit events are generated whenever a relevant action occurs.
   •  They all specify -F path= to define the path to a specific configuration file being monitored.
   •  -F perm=wa specifies that events are triggered for write (w) and attribute change (a) permissions.
   •  -F success=1 ensures events are logged only when the operation is successful.
   •  -k parameter specifies the name of the auditd rule.
   •  The difference lies solely in the paths specified (-F path=), which point to different configuration files (/etc/audit/auditd.conf, /etc/audit/rules.d/test.rules, /etc/audisp/audispd.conf, /etc/libaudit.conf).

Once auditd rules are configured to monitor key configuration files, they generate logs for any modifications detected. These logs serve as inputs for creating detection rules in Splunk.

By correlating and analyzing these audit logs, Splunk enables proactive monitoring and alerts for unauthorized changes.

Step 2: Develop the correlation search in Splunk

I tested the new audit rules to ensure they function correctly by simulating various modifications to auditd.conf.

The first test involved deleting auditd.conf using the rm command. In the second test, I moved the configuration file to another directory, and in the third test, I modified it using the vim text editor.

The detection rule and the _raw logs showing the malicious activities described above

Explanation:

1. Search Criteria (index, sourcetype, type, key):
   

   •  Searches the linux_audit index for events with linux:audit sourcetype and type as SYSCALL.
   •  Filters events where the key field matches any of the specified audit rules we created (auditrule_modification, auditd_conf_modification, audispd_conf_modification, libauditd_conf_modification).

2. Aggregation (transaction host maxpause=1s):

   •  Groups consecutive events (SYSCALL entries) from the same host within a 1-second timeframe (maxpause=1s).
   •  This aggregation helps to analyze sequences of system calls which could give us additional context about the specific actions of the threat actor.

3. Statistical Summary (stats count by _time, host, key, comm, exe, uid, gid, _raw):

   •  Calculates counts grouped by various fields: _time (timestamp), host (machine name), key (audit key), comm (command name), exe (executable path), uid (user ID), gid (group ID), and _raw (raw log entry). This summary indicates the frequency of each distinct combination of these attributes within the transaction.

The crucial role of SYSCALL events

SYSCALLs (system calls) play a crucial role in detecting and understanding events within a Linux system’s audit logs. Here’s how they contribute to event detection:

Fundamental System Operations: SYSCALLs are essential because they represent fundamental operations performed by processes on a Linux system. These operations include file system accesses (like open, read, write, unlink), process management (such as fork, execve, exit), and network communications (e.g., socket, connect, sendmsg).

Audit Logging: Linux systems use auditing mechanisms to log SYSCALL events. Each SYSCALL event recorded in audit logs includes details such as the type of operation (syscall), its success or failure (success=yes/no), associated process IDs (pid, ppid), user and group IDs (uid, gid), and executable paths (exe).

Breakdown of the Raw Log Entry:

   •  type=SYSCALL: Indicates that the log entry is related to a system call event.
   •  msg=audit(1715771488.665:75612745): Timestamp and sequence number of the audit event.
   •  syscall=263: Specifies the system call number.
   •  success=yes: Indicates that the system call was successful (yes).
   •  a0, a1, a2, a3: These fields in the audit log entry represent the arguments (a0 through a3) passed to the system call (syscall=263) at the time of execution. They provide extra context about the specific details of the system call operation.
   •  exit=0: Exit status of the system call (0 indicates success).
   •  auid=*****: Audit user ID.
   •  uid=0, gid=0, euid=0, suid=0, fsuid=0, egid=0, sgid=0, fsgid=0: User and group IDs associated with the process (uid for user ID; gid for group ID).
   •  comm=”rm”: Command name (rm in this case) executed by the process.
   •  exe=”/usr/bin/rm”: Executable path (/usr/bin/rm) associated with the command.
   •  key=”auditd_conf_modification”: Identifies the audit key associated with this event (auditd_conf_modification indicates a modification related to auditd configuration).

Conclusion

In this two-part series on detecting Linux audit logs tampering, we’ve explored essential strategies to safeguard the integrity of auditd rules and configurations. Initially, we emphasized the critical role of auditd in security monitoring and its susceptibility to malicious tampering. This led us to focus on detecting deletions of auditd rules and changes in configurations, vital for maintaining robust security practices.

Through practical implementations using auditd rules and Splunk queries, we’ve demonstrated effective methods to monitor and detect unauthorized modifications. By leveraging audit record types like CONFIG_CHANGE and SYSCALLs, we’ve developed detection mechanisms capable of flagging suspicious activities promptly. These efforts ensure the reliability of audit logs, crucial for incident response and compliance.

Next Steps:

Moving forward, the next steps involve:

1. Continuous Monitoring: Implementing ongoing monitoring of auditd rules and configurations to detect any future unauthorized changes.
2. Enhanced Alerting: Fine-tuning alert mechanisms in Splunk to provide real-time notifications of suspicious activities related to auditd.
3. Incident Response Readiness: Preparing response actions to swiftly investigate and mitigate any identified security breaches or policy violations.
4. Periodic Review: Regularly reviewing and updating auditd configurations and rules to adapt to evolving security threats and organizational needs.
   
   By integrating these practices into your security operations, you’ll strengthen your Linux environment’s defenses against tampering and ensure the integrity of your audit logs for effective security monitoring and compliance.

Written by Aleksandar Matev