Monitor and graph file creation rates

Monitoring and graphing file creation rates is crucial for systems administration, performance tuning, and detecting anomalies like malware or runaway processes. This article explores methods for tracking file creation rates in real-time or over intervals, using command-line tools, custom scripts, and visualization techniques.

Why Monitor File Creation Rates?

File creation rates indicate system activity levels. Sudden spikes might signal malicious behavior (e.g., ransomware generating files rapidly), excessive logging, or application bugs. Consistently high rates might also lead to performance issues, fill up storage, or affect inode usage. Monitoring allows:

• Proactive performance tuning.

• Anomaly detection.

• Trend analysis for storage planning.

• Debugging application behavior.

Methods to Monitor File Creation

1. Auditd (Linux Auditing System)

Auditd logs file-related events by monitoring system calls. It is ideal for capturing detailed file activity.

Configuration Steps:

1. Install auditd:

   bash
   CopyEdit
   sudo apt install auditd audispd-plugins
   

2. Add a rule to monitor directories:

   bash
   CopyEdit
   sudo auditctl -w /path/to/dir -p wa -k file_creation_monitor
   

3. View logs:

   bash
   CopyEdit
   ausearch -k file_creation_monitor
   

4. Count file creation events:

   bash
   CopyEdit
   ausearch -k file_creation_monitor | grep "name=" | wc -l
   

Auditd logs can be parsed periodically and fed into graphing tools.

2. Inotifywait (inotify-tools)

For lightweight monitoring of directory changes, inotifywait from the inotify-tools package is effective.

Installation:

bash
CopyEdit
sudo apt install inotify-tools

Monitor script:

bash
CopyEdit
#!/bin/bash
LOGFILE="creation_log.txt"
inotifywait -m -e create --format '%T %w%f' --timefmt '%Y-%m-%d %H:%M:%S' /path/to/dir >> $LOGFILE

This script logs timestamps and file names of newly created files. You can then parse this file to aggregate events by minute, hour, or day for plotting.

3. Filesystem Events via auditd on macOS

macOS users can use the fs_usage or OpenBSM framework for auditing file-level events:

bash
CopyEdit
sudo fs_usage -w | grep -i create

This provides a real-time feed of file-related system calls.

4. Using find and Timestamp Filters

If real-time monitoring isn’t required, periodic scans can identify newly created files:

bash
CopyEdit
find /path/to/dir -type f -cmin -1

This lists files created in the last minute. Combine with a cron job or script to log and count:

bash
CopyEdit
#!/bin/bash
TIMESTAMP=$(date +%Y-%m-%d:%H:%M)
COUNT=$(find /path/to/dir -type f -cmin -1 | wc -l)
echo "$TIMESTAMP,$COUNT" >> file_creation_rate.csv

5. SystemTap (Advanced Users)

SystemTap allows kernel-level probing. Example script to track file creation:

bash
CopyEdit
global creations
probe syscall.open {
    if (flags & O_CREAT)
        creations[execname()]++
}
probe end {
    foreach (p in creations)
        printf("%s created %d filesn", p, creations[p])
}

This logs which process is creating files, offering deep visibility into system behavior.

Graphing File Creation Rates

Once data is collected (usually as a timestamp and count), you can visualize it using:

1. Gnuplot

Simple and powerful for time-series data.

Example usage:

bash
CopyEdit
gnuplot -e "set xdata time; set timefmt '%Y-%m-%d:%H:%M'; set format x '%H:%M'; plot 'file_creation_rate.csv' using 1:2 with lines title 'File Creations'"

2. Python + Matplotlib

For customized and interactive plots.

python
CopyEdit
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("file_creation_rate.csv", names=["timestamp", "count"])
df["timestamp"] = pd.to_datetime(df["timestamp"])
df.set_index("timestamp", inplace=True)
df.plot()
plt.title("File Creation Rate Over Time")
plt.ylabel("Files Created")
plt.xlabel("Time")
plt.grid()
plt.show()

3. Grafana + Prometheus

For real-time dashboards, use Grafana with Prometheus or Telegraf/InfluxDB.

• Export file creation count as metrics.

• Push via node_exporter custom script or Telegraf.

• Set up time-series visualization in Grafana.

Automation and Alerts

To make monitoring actionable:

• Set up Cron Jobs: Run monitoring scripts every minute/hour.

• Threshold-Based Alerts: Use tools like monit, Nagios, or Prometheus AlertManager to notify on spikes.

• Email/SMS Notifications: Integrate alert tools with messaging APIs to inform admins of anomalies.

Best Practices

• Monitor only relevant directories to reduce overhead.

• Use log rotation and archiving for long-term tracking.

• Secure audit logs to prevent tampering.

• Regularly review and tune thresholds based on system behavior.

• Combine file monitoring with CPU, memory, and disk usage data for complete insights.

Use Cases in Real-World Environments

1. Web Servers: Track upload directories for spikes in user uploads.

2. CI/CD Pipelines: Monitor build artifact directories for abnormal growth.

3. Security: Detect ransomware by flagging high file creation rates.

4. DevOps: Analyze deployment activity by watching container volume mounts or shared storage.

Conclusion

Monitoring and graphing file creation rates can provide deep insights into system health, performance, and security. By leveraging tools like auditd, inotifywait, and visualization platforms, administrators can proactively detect issues, understand usage patterns, and improve operational reliability. With proper automation, alerts, and analysis, file creation metrics become a vital part of a robust monitoring strategy.