Binary Files, seek and tell

bytes and str

text = "café"
data = text.encode("utf-8")

print(type(text), len(text))          # <class 'str'> 4
print(type(data), len(data))          # <class 'bytes'> 5
print(data)                            # b'caf\xc3\xa9'
print(data.decode("utf-8"))            # café

A str is a sequence of characters; bytes is a sequence of numbers from 0 to 255. Files, sockets and hardware deal in bytes. Text mode does the encoding for you; binary mode does not.

data = b"hello"

print(data[0])           # 104 - indexing gives an INTEGER
print(data[0:2])         # b'he' - slicing gives bytes
print(list(data))        # [104, 101, 108, 108, 111]
print(bytes([104, 105])) # b'hi'

print(data.upper())      # b'HELLO' - many str methods exist on bytes too
# print(data + "x")      # TypeError: cannot concatenate bytes and str

Reading and writing binary

with open("data.bin", "wb") as handle:
    handle.write(b"\x89PNG\r\n")
    handle.write(bytes([0, 1, 2, 255]))

with open("data.bin", "rb") as handle:
    content = handle.read()

print(content)                # b'\x89PNG\r\n\x00\x01\x02\xff'
print(len(content))           # 10
print(content.hex())          # 89504e470d0a000102ff

Copying a file

def copy(source, destination, chunk_size=64 * 1024):
    with open(source, "rb") as reader, open(destination, "wb") as writer:
        while chunk := reader.read(chunk_size):
            writer.write(chunk)


copy("image.png", "image-copy.png")

Reading in chunks keeps memory use constant no matter how large the file is. The walrus operator handles the loop condition cleanly: read returns b"" at the end of the file, which is falsy.

Identifying a file by its first bytes

SIGNATURES = {
    b"\x89PNG\r\n\x1a\n": "PNG image",
    b"\xff\xd8\xff": "JPEG image",
    b"%PDF": "PDF document",
    b"PK\x03\x04": "ZIP archive",
    b"GIF87a": "GIF image",
    b"GIF89a": "GIF image",
}


def identify(path):
    with open(path, "rb") as handle:
        head = handle.read(16)
    for signature, name in SIGNATURES.items():
        if head.startswith(signature):
            return name
    return "unknown"

Most binary formats begin with a fixed marker, often called a magic number. Reading the first few bytes is far more reliable than trusting a file extension.

tell

with open("notes.txt", "rb") as handle:
    print(handle.tell())          # 0 - at the start
    handle.read(5)
    print(handle.tell())          # 5
    handle.read()
    print(handle.tell())          # the file size

tell() reports the current position, in bytes from the start.

seek

handle.seek(offset, whence)

whence = 0   from the start        (the default)
whence = 1   from the current position
whence = 2   from the end

with open("data.bin", "rb") as handle:
    handle.seek(4)            # to byte 4
    print(handle.read(2))

    handle.seek(0)            # back to the start
    print(handle.read(4))

    handle.seek(-3, 2)        # three bytes before the end
    print(handle.read())

    handle.seek(2, 1)         # two bytes forward from here

In text mode, only seek(0) and seeking to a position previously returned by tell() are supported, and relative seeks are not allowed. Variable width encodings make arbitrary character offsets meaningless. Do position arithmetic in binary mode.

Reading a file twice

with open("notes.txt", encoding="utf-8") as handle:
    first = handle.read()
    second = handle.read()          # '' - the position is at the end

    handle.seek(0)
    third = handle.read()           # the content again

print(len(first), len(second), len(third))

This catches everyone once. A file object has one position, and reading consumes it.

Reading the last lines of a file

def tail(path, lines=5, block=4096):
    """Read the last few lines without loading the whole file."""
    with open(path, "rb") as handle:
        handle.seek(0, 2)
        size = handle.tell()
        data = b""
        while size > 0 and data.count(b"\n") <= lines:
            step = min(block, size)
            size -= step
            handle.seek(size)
            data = handle.read(step) + data
    return data.decode("utf-8", errors="replace").splitlines()[-lines:]


for line in tail("app.log", 3):
    print(line)

This is what a tail command does: jump to the end, walk backwards in blocks until enough newlines are found. Reading the whole file would work too, and would be unusable on a multi gigabyte log.

Fixed width records

RECORD_SIZE = 32


def read_record(path, index):
    with open(path, "rb") as handle:
        handle.seek(index * RECORD_SIZE)
        return handle.read(RECORD_SIZE)


def write_record(path, index, data):
    padded = data.ljust(RECORD_SIZE, b"\x00")[:RECORD_SIZE]
    with open(path, "r+b") as handle:
        handle.seek(index * RECORD_SIZE)
        handle.write(padded)

When every record is the same size, any record can be reached in one seek without reading the ones before it. This is the idea behind every indexed file format and, ultimately, behind database storage.

Numbers as bytes

value = 1000

big = value.to_bytes(4, "big")
little = value.to_bytes(4, "little")

print(big)                                   # b'\x00\x00\x03\xe8'
print(little)                                # b'\xe8\x03\x00\x00'
print(int.from_bytes(big, "big"))            # 1000
print(int.from_bytes(little, "little"))      # 1000

import struct

packed = struct.pack(">IH4s", 1000, 25, b"data")
print(packed)
print(struct.unpack(">IH4s", packed))        # (1000, 25, b'data')
print(struct.calcsize(">IH4s"))              # 10

struct converts between Python values and a fixed binary layout. The format string names each field: I is a four byte unsigned integer, H a two byte one, 4s a four byte string, and > means most significant byte first. This is how binary file headers and network protocols are read.

bytearray

data = bytearray(b"hello")
data[0] = 72                        # bytes are immutable, bytearray is not
data.extend(b" world")
print(data)                         # bytearray(b'Hello world')
print(bytes(data))                  # b'Hello world'

Common mistakes

• Mixing str and bytes in one expression.
• Expecting data[0] to be a one byte bytes object; it is an integer.
• Using relative seek in text mode.
• Reading a file twice without seeking back to zero.
• Assuming a file's extension tells you its format.
• Reading an entire large binary file instead of processing it in chunks.

Best practices

• Use binary mode for anything that is not text, and decode explicitly when you need text.
• Process large files in chunks with a fixed buffer size.
• Do all seeking in binary mode.
• Use struct rather than slicing bytes by hand for structured binary data.
• Check magic numbers rather than file extensions.

Practice

1. Write a function that reports the size of a file using only seek and tell.
2. Copy a large file in 64 KB chunks and confirm the copy is identical.
3. Identify the format of five files from their first bytes.
4. Implement a fixed width record store supporting read and update by index.
5. Pack three numbers with struct, write them to a file, and read them back.

Conclusion

Binary mode hands you the exact bytes, and seek with tell turns a file from a stream into something you can navigate. Together they let you read the last lines of a huge log, jump straight to record number 5000, and read file headers without loading anything else.