Python File Handling Guide: Read, Write, Process Any File Type

Python File Handling Guide: Read, Write, Process Any File Type

Files are how programs persist data between runs. They're how you share data between systems. They're how you automate working with documents, logs, exports, and reports.

Python's file handling is straightforward once you understand three things: the with statement, the pathlib module, and file modes. This guide covers all of that, plus handling the specific file types you'll encounter most often.

The with Statement — Always Use It

The with statement handles closing files automatically, even if an error occurs:

# Old way — don't do this:
f = open("data.txt", "r")
content = f.read()
f.close()  # Easy to forget; won't run if an error occurs

# Modern way — always do this:
with open("data.txt", "r") as f:
    content = f.read()
# File is automatically closed when the with block ends

Why it matters: Files hold system resources. If you open a file and your program crashes before .close() runs, the file stays open. The with statement guarantees cleanup.

File Modes

open("file.txt", "r")   # Read (default) — file must exist
open("file.txt", "w")   # Write — creates file, overwrites if exists
open("file.txt", "a")   # Append — creates file, adds to end if exists
open("file.txt", "x")   # Exclusive create — fails if file exists
open("file.txt", "rb")  # Read binary
open("file.txt", "wb")  # Write binary

Reading Text Files

# Read entire file as a string
with open("notes.txt", "r", encoding="utf-8") as f:
    content = f.read()

# Read as a list of lines
with open("notes.txt", "r", encoding="utf-8") as f:
    lines = f.readlines()  # Includes \n at end of each line
    lines = [line.rstrip() for line in lines]  # Remove newlines

# Read line by line (memory efficient for large files)
with open("large_log.txt", "r", encoding="utf-8") as f:
    for line in f:
        process(line.strip())

Always use encoding="utf-8" for text files to avoid encoding errors on different systems.

Writing Text Files

# Write (overwrites existing content)
with open("output.txt", "w", encoding="utf-8") as f:
    f.write("Hello, World!\n")
    f.write("Second line\n")

# Append to existing file
with open("log.txt", "a", encoding="utf-8") as f:
    f.write(f"Log entry: {message}\n")

# Write multiple lines at once
lines = ["Line 1", "Line 2", "Line 3"]
with open("output.txt", "w", encoding="utf-8") as f:
    f.writelines(line + "\n" for line in lines)

pathlib — The Modern Way

pathlib is built into Python (no installation needed) and makes file operations cleaner:

from pathlib import Path

# Building paths (cross-platform — works on Windows AND Mac/Linux)
home = Path.home()
documents = home / "Documents"
my_file = documents / "notes" / "python.txt"

# Simple read/write (for small files)
content = my_file.read_text(encoding="utf-8")
my_file.write_text("New content", encoding="utf-8")

# Append (pathlib doesn't have append — use open())
with my_file.open("a") as f:
    f.write("More content\n")

# Check existence
if my_file.exists():
    print("File found")

# File info
print(my_file.name)        # "python.txt"
print(my_file.stem)        # "python"
print(my_file.suffix)      # ".txt"
print(my_file.parent)      # Path to containing folder
print(my_file.stat().st_size)  # File size in bytes

# Create directories
(home / "new_folder").mkdir(parents=True, exist_ok=True)

# Find files
python_files = list(Path("src").rglob("*.py"))     # Recursive
txt_files = list(Path(".").glob("*.txt"))           # Current dir only

CSV Files

Using the csv Module

import csv

# Write CSV
data = [
    {"name": "Alice", "age": 30, "city": "London"},
    {"name": "Bob",   "age": 25, "city": "Paris"},
]

with open("people.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "age", "city"])
    writer.writeheader()
    writer.writerows(data)

# Read CSV
with open("people.csv", "r", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row["name"], row["age"])  # Row is a dict

Using pandas (For Data Analysis)

import pandas as pd

# Read
df = pd.read_csv("people.csv")

# Filter, transform, analyze
adults = df[df["age"] >= 18]

# Write back
adults.to_csv("adults.csv", index=False)

For a complete guide to pandas and data manipulation, see our Python data science roadmap.

JSON Files

import json

# Write JSON
data = {
    "user": "Alice",
    "settings": {"theme": "dark", "notifications": True},
    "scores": [95, 87, 92]
}

with open("config.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)

# Read JSON
with open("config.json", "r", encoding="utf-8") as f:
    loaded = json.load(f)

print(loaded["user"])        # Alice
print(loaded["settings"])    # {'theme': 'dark', ...}

# Shorthand with pathlib
from pathlib import Path
Path("config.json").write_text(json.dumps(data, indent=2))
loaded = json.loads(Path("config.json").read_text())

Binary Files

# Read binary (images, PDFs, executables)
with open("photo.jpg", "rb") as f:
    binary_data = f.read()

# Write binary
with open("copy.jpg", "wb") as f:
    f.write(binary_data)

# Using pathlib for binary
data = Path("photo.jpg").read_bytes()
Path("copy.jpg").write_bytes(data)

Working with Multiple Files

from pathlib import Path
import shutil

# Copy a file
shutil.copy("source.txt", "destination.txt")

# Move a file
shutil.move("old_location.txt", "new_location.txt")

# Delete a file
Path("unwanted.txt").unlink(missing_ok=True)  # missing_ok prevents error if not found

# Delete a directory
shutil.rmtree("old_folder")  # Careful — deletes recursively!

# Rename
Path("old_name.txt").rename("new_name.txt")

# Iterate over files in a folder
for file in Path("data").iterdir():
    if file.is_file() and file.suffix == ".csv":
        process_csv(file)

Error Handling

from pathlib import Path

def safe_read(filepath: str) -> str | None:
    path = Path(filepath)
    
    if not path.exists():
        print(f"File not found: {filepath}")
        return None
    
    try:
        return path.read_text(encoding="utf-8")
    except PermissionError:
        print(f"Permission denied: {filepath}")
        return None
    except UnicodeDecodeError:
        # Try with a different encoding
        try:
            return path.read_text(encoding="latin-1")
        except Exception as e:
            print(f"Cannot read {filepath}: {e}")
            return None

Practical Pattern: Configuration Files

A real-world pattern for loading app configuration from a JSON file:

import json
from pathlib import Path
from dataclasses import dataclass

@dataclass
class AppConfig:
    api_key: str
    debug: bool = False
    max_retries: int = 3
    output_dir: str = "./output"

def load_config(config_path: str = "config.json") -> AppConfig:
    path = Path(config_path)
    
    if not path.exists():
        # Create default config
        default = {"api_key": "", "debug": False, "max_retries": 3}
        path.write_text(json.dumps(default, indent=2))
        print(f"Created default config at {path}")
    
    data = json.loads(path.read_text())
    return AppConfig(**data)

config = load_config()

Frequently Asked Questions

Best way to read files in Python?

with open("file.txt", "r", encoding="utf-8") as f: — the with statement ensures the file closes. Use pathlib.Path.read_text() for simple one-line reads.

What is pathlib?

Python's modern built-in file path library. Cleaner and more Pythonic than os.path. Use it for all file system operations.

How to handle file errors?

Wrap in try/except for FileNotFoundError, PermissionError, and UnicodeDecodeError. Use Path.exists() for simple existence checks.

How to read/write CSV?

Use the csv module for simple reading/writing, pandas for data analysis on CSV files.

Final Thoughts

File handling is one of those Python skills that's used in almost every project — configuration, logging, data processing, automation, and more. The patterns here (with statements, pathlib, encoding="utf-8") apply across all file types.

The most important habit: always use encoding="utf-8" explicitly. Files that work on your Mac may break on Windows without it.

For applying these file skills in automation scripts that process and organize files, see our Python automation scripts guide for 20 ready-to-use examples. And for understanding the OOP patterns that make file-handling code cleaner and reusable, our Python OOP tutorial covers classes and objects with practical examples.