Copy Files with Progress Indicators in Python

Copying files with progress indicators is a practical task in Python, especially when dealing with large files or bulk operations where user feedback enhances experience. This article explores several approaches using Python libraries such as shutil, os, tqdm, and concurrent.futures to implement file copying with visible progress tracking.

Basic File Copy in Python

Python’s built-in shutil module provides basic file copying capabilities.

python
CopyEdit
import shutil

shutil.copy2('source_file.txt', 'destination_file.txt')

This method is efficient but doesn’t provide progress indicators. For enhanced functionality, a more manual approach with byte-by-byte copying is required.

Manual Copy with Progress

To create a custom copy function with progress, read and write in chunks while tracking the total size.

python
CopyEdit
import os
import sys

def copy_file_with_progress(src, dst, buffer_size=1024*1024):
    total_size = os.path.getsize(src)
    copied = 0

    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
        while True:
            buf = fsrc.read(buffer_size)
            if not buf:
                break
            fdst.write(buf)
            copied += len(buf)
            progress = copied / total_size * 100
            sys.stdout.write(f"rProgress: {progress:.2f}%")
            sys.stdout.flush()
    print("nCopy completed.")

This function reads the file in 1MB chunks and displays real-time percentage completion in the console.

Using tqdm for Better Progress Bar

The tqdm library provides a neat progress bar in the terminal with minimal setup.

python
CopyEdit
from tqdm import tqdm

def copy_with_tqdm(src, dst, buffer_size=1024*1024):
    total_size = os.path.getsize(src)
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst, tqdm(total=total_size, unit='B', unit_scale=True, desc=src) as pbar:
        while True:
            buf = fsrc.read(buffer_size)
            if not buf:
                break
            fdst.write(buf)
            pbar.update(len(buf))

tqdm automatically manages progress display and formatting, making it user-friendly for developers.

Copy Multiple Files with Progress

When dealing with multiple files, tracking individual file progress and overall progress improves user clarity.

python
CopyEdit
import glob

def copy_files_with_overall_progress(src_folder, dst_folder):
    files = glob.glob(os.path.join(src_folder, '*'))
    total_bytes = sum(os.path.getsize(f) for f in files)

    with tqdm(total=total_bytes, unit='B', unit_scale=True, desc='Total Progress') as pbar:
        for file in files:
            dst_path = os.path.join(dst_folder, os.path.basename(file))
            with open(file, 'rb') as fsrc, open(dst_path, 'wb') as fdst:
                while True:
                    buf = fsrc.read(1024*1024)
                    if not buf:
                        break
                    fdst.write(buf)
                    pbar.update(len(buf))

This function computes the total size of all files and updates the overall progress bar as each file is copied.

Asynchronous Copy with concurrent.futures

For better performance in copying large sets of files, parallelism can be introduced using ThreadPoolExecutor.

python
CopyEdit
from concurrent.futures import ThreadPoolExecutor

def async_copy(src, dst):
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
        while True:
            buf = fsrc.read(1024*1024)
            if not buf:
                break
            fdst.write(buf)

def copy_files_concurrently(src_folder, dst_folder):
    files = glob.glob(os.path.join(src_folder, '*'))
    with ThreadPoolExecutor(max_workers=4) as executor:
        for file in files:
            dst_path = os.path.join(dst_folder, os.path.basename(file))
            executor.submit(async_copy, file, dst_path)

Although this version omits progress indicators for simplicity, it provides a foundation to integrate tqdm progress tracking with thread-safe updates.

Integrating Progress with Logging

In production scripts, logging progress instead of printing it to the console may be preferable.

python
CopyEdit
import logging

logging.basicConfig(filename='copy.log', level=logging.INFO)

def copy_with_logging(src, dst):
    total_size = os.path.getsize(src)
    copied = 0
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
        while True:
            buf = fsrc.read(1024*1024)
            if not buf:
                break
            fdst.write(buf)
            copied += len(buf)
            progress = copied / total_size * 100
            logging.info(f"{src} -> {dst}: {progress:.2f}%")

This approach is useful in headless environments or background services where console access isn’t available.

Using rich for Stylish Terminal Output

The rich library offers aesthetically pleasing terminal outputs including progress bars.

python
CopyEdit
from rich.progress import Progress

def copy_with_rich_progress(src, dst):
    total_size = os.path.getsize(src)
    with Progress() as progress:
        task = progress.add_task(f"Copying {os.path.basename(src)}", total=total_size)
        with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
            while True:
                buf = fsrc.read(1024*1024)
                if not buf:
                    break
                fdst.write(buf)
                progress.update(task, advance=len(buf))

rich enhances user experience significantly and supports nested tasks or live updates in complex CLI tools.

Error Handling and Safety Tips

While copying files, especially large or sensitive ones, consider the following best practices:

• File existence check: Avoid overwriting unless explicitly allowed.

• Atomic writes: Write to a temporary file first, then rename.

• Permissions: Ensure read/write permissions for source and destination.

• Exception handling: Gracefully handle IOErrors, disk full, etc.

Example:

python
CopyEdit
import shutil

def safe_copy(src, dst):
    try:
        if os.path.exists(dst):
            print(f"File {dst} already exists. Skipping.")
            return
        shutil.copy2(src, dst)
    except Exception as e:
        print(f"Failed to copy {src} to {dst}: {e}")

When to Use What

Conclusion

Python’s standard and third-party libraries offer flexible solutions for copying files with user-friendly progress indicators. Depending on your requirements—from simple scripts to advanced multi-threaded applications—you can choose from shutil, tqdm, rich, or even parallelization libraries to create efficient and informative file operations. Combining these tools with good practices ensures your file management tasks are both robust and user-aware.