Monitor Python script execution

Monitoring the execution of a Python script can be essential for debugging, performance analysis, or ensuring that a process runs smoothly. Below is a comprehensive guide covering several methods and tools you can use to monitor Python script execution:

1. Basic Print Statements and Logging

• Use print() statements to track the flow of the script or variable values.

• More professional: Use the logging module to log messages with different severity levels (DEBUG, INFO, WARNING, ERROR, CRITICAL).

python
CopyEdit
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logging.info('Script started')

for i in range(5):
    logging.debug(f'Processing item {i}')
    
logging.info('Script finished')

2. Time Execution Duration

• Use the time module or timeit to measure how long parts of your script take to run.

python
CopyEdit
import time

start_time = time.time()

# Your code here
time.sleep(2)

end_time = time.time()
print(f"Execution time: {end_time - start_time} seconds")

• For more detailed profiling, use cProfile (built-in profiler).

3. Profiling with cProfile

• cProfile helps identify slow functions and bottlenecks.

Run your script with:

bash
CopyEdit
python -m cProfile -s time your_script.py

Or programmatically:

python
CopyEdit
import cProfile

def main():
    # Your script code here
    pass

cProfile.run('main()')

4. Using Watchdog for File/System Event Monitoring

• If your script needs to react or monitor files or directories, use watchdog to track file system events.

python
CopyEdit
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import time

class MyHandler(FileSystemEventHandler):
    def on_modified(self, event):
        print(f'File modified: {event.src_path}')

observer = Observer()
observer.schedule(MyHandler(), path='.', recursive=False)
observer.start()

try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    observer.stop()
observer.join()

5. Real-time Script Output Monitoring

• You can run your Python script and watch its output live in the terminal.

• Or, redirect output to a log file:

bash
CopyEdit
python your_script.py > output.log 2>&1
tail -f output.log

6. Resource Usage Monitoring (CPU, Memory)

• Use psutil to monitor CPU and memory usage during script execution.

python
CopyEdit
import psutil
import os
import time

process = psutil.Process(os.getpid())

for i in range(10):
    print(f'CPU usage: {process.cpu_percent()}%')
    print(f'Memory usage: {process.memory_info().rss / (1024 * 1024):.2f} MB')
    time.sleep(1)

7. Exception Handling and Reporting

• To monitor errors, wrap your main code in try-except and log or print exceptions.

python
CopyEdit
import traceback

try:
    # your script logic
    pass
except Exception as e:
    print("Exception occurred:", e)
    traceback.print_exc()

8. Remote Monitoring

• Use tools like Sentry or custom dashboards with Flask and socket.io to monitor script execution remotely.

• For production, container monitoring (Docker + Prometheus + Grafana) can be integrated.

Summary

• For simple tracking, use print() or logging.

• For performance, use cProfile or timeit.

• For resource monitoring, use psutil.

• For file monitoring, use watchdog.

• For error monitoring, use try-except and logging.

• For remote monitoring, use dedicated tools or build a custom dashboard.

If you want, I can provide a tailored script example combining these tools for your use case—just let me know!