Reading and Writing Text Files

Always use with

with open("notes.txt") as handle:
    content = handle.read()

print(content)
print(handle.closed)          # True - closed automatically

The with statement closes the file however the block ends: normally, through a return, or because an exception was raised. The manual alternative needs a try and a finally to be equally safe, so there is no reason to write it.

# Do not do this
handle = open("notes.txt")
content = handle.read()
handle.close()                # skipped entirely if read() raises

An unclosed file holds an operating system handle, and on Windows it may keep the file locked so nothing else can write to it. Data written to a file that is never closed can also be lost, because it may still be sitting in a buffer.

Writing

with open("notes.txt", "w", encoding="utf-8") as handle:
    handle.write("First line\n")
    handle.write("Second line\n")

lines = ["Third line\n", "Fourth line\n"]
with open("notes.txt", "a", encoding="utf-8") as handle:
    handle.writelines(lines)          # note: writelines adds NO newlines itself

• "w" truncates the file to empty before writing. Anything that was there is gone.
• "a" appends to the end, creating the file if it does not exist.
• write does not add a newline. You must include \n yourself.
• writelines is badly named: it writes a sequence of strings with no separator at all.

rows = ["alpha", "beta", "gamma"]

with open("out.txt", "w", encoding="utf-8") as handle:
    handle.write("\n".join(rows) + "\n")      # the usual way

with open("out.txt", "w", encoding="utf-8") as handle:
    for row in rows:
        print(row, file=handle)               # print can write to a file too

Reading, four ways

with open("notes.txt", encoding="utf-8") as handle:
    everything = handle.read()          # one string, the whole file

with open("notes.txt", encoding="utf-8") as handle:
    first = handle.readline()           # one line, including its newline

with open("notes.txt", encoding="utf-8") as handle:
    all_lines = handle.readlines()      # a list of lines, newlines included

with open("notes.txt", encoding="utf-8") as handle:
    for line in handle:                 # one line at a time, lazily
        print(line.rstrip())

Iterating the file object is the default choice. It reads lazily, so a ten gigabyte log file uses no more memory than a ten line one.

Newlines

with open("notes.txt", encoding="utf-8") as handle:
    for line in handle:
        print(repr(line))              # 'First line\n' - the newline is included

with open("notes.txt", encoding="utf-8") as handle:
    for line in handle:
        print(line.rstrip("\n"))       # strip only the newline
        # line.strip() would also remove meaningful leading whitespace

Always pass encoding

with open("data.txt", encoding="utf-8") as handle:      # correct
    content = handle.read()

with open("data.txt") as handle:                        # risky
    content = handle.read()

Without encoding=, Python uses whatever the operating system happens to prefer. A file written on one machine can then fail to read on another with a UnicodeDecodeError. Write encoding="utf-8" every time; it costs eighteen characters and removes an entire class of bug.

# When a file may contain undecodable bytes
with open("messy.txt", encoding="utf-8", errors="replace") as handle:
    content = handle.read()          # bad bytes become the replacement character

with open("messy.txt", encoding="utf-8", errors="ignore") as handle:
    content = handle.read()          # bad bytes are silently dropped

Handling a missing file

def read_config(path):
    try:
        with open(path, encoding="utf-8") as handle:
            return handle.read()
    except FileNotFoundError:
        print(f"{path} not found, using defaults")
        return ""
    except PermissionError:
        print(f"no permission to read {path}")
        raise

from pathlib import Path

path = Path("config.txt")
if path.exists():
    content = path.read_text(encoding="utf-8")

The check and the open are two separate steps, so the file could disappear between them. For anything that matters, prefer try over exists().

Reading and writing in one pass

def add_line_numbers(source, destination):
    with open(source, encoding="utf-8") as reader, \
         open(destination, "w", encoding="utf-8") as writer:
        for number, line in enumerate(reader, start=1):
            writer.write(f"{number:4}  {line}")


add_line_numbers("notes.txt", "numbered.txt")

One with can open several files. Both are closed when the block ends, in reverse order.

Worked examples

Counting lines, words and characters

def count(path):
    lines = words = characters = 0
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            lines += 1
            words += len(line.split())
            characters += len(line)
    return lines, words, characters


print(count("notes.txt"))

Filtering a log file

def extract_errors(source, destination):
    kept = 0
    with open(source, encoding="utf-8") as reader, \
         open(destination, "w", encoding="utf-8") as writer:
        for line in reader:
            if "ERROR" in line:
                writer.write(line)
                kept += 1
    return kept

Reading a simple settings file

def load_settings(path):
    settings = {}
    with open(path, encoding="utf-8") as handle:
        for number, line in enumerate(handle, start=1):
            line = line.strip()
            if not line or line.startswith("#"):
                continue
            key, separator, value = line.partition("=")
            if not separator:
                print(f"line {number} ignored: {line!r}")
                continue
            settings[key.strip()] = value.strip()
    return settings

Finding the most common words

from collections import Counter


def top_words(path, n=5):
    counts = Counter()
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            counts.update(word.strip(".,!?;:").lower() for word in line.split())
    return counts.most_common(n)

Note that the file is never loaded into memory as a whole. This function works the same on a file of any size.

Appending safely

from datetime import datetime


def log(message, path="app.log"):
    stamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(f"{stamp}  {message}\n")


log("service started")
log("first request handled")

Opening in append mode for each message is simple and safe. For a real application, the standard library logging module does this better, with levels, formatting and rotation.

Common mistakes

• Opening with "w" when "a" was meant, and erasing the file.
• Forgetting encoding="utf-8".
• Forgetting that write does not add a newline.
• Expecting writelines to insert newlines.
• Calling read() on a very large file.
• Using line.strip() where line.rstrip("\n") was meant, losing indentation.
• Reading a file twice from the same handle without seeking back; the second read returns nothing.

Best practices

• Always use with.
• Always pass encoding="utf-8".
• Iterate the file object rather than calling readlines().
• Handle FileNotFoundError rather than checking exists() first.
• Write to a temporary file and rename it when replacing important data, so a crash cannot leave a half written file.

Practice

1. Write a program that copies a file, converting every line to uppercase.
2. Count how many lines in a file are blank, comments, or content.
3. Read a file, reverse the order of its lines, and write the result to a new file.
4. Explain what happens when the same file is opened twice, once in "r" and once in "w".
5. Write a function that appends to a log file and prove the file is closed after each call.

Conclusion

Open with with, always pass encoding="utf-8", and iterate the file object one line at a time. Those three habits cover almost all text file work and keep it correct for files of any size.