Serialization and pickle

What serialization is

Serialization turns an object in memory into a sequence of bytes that can be stored or sent. Deserialization turns those bytes back into an object. JSON does this for a small set of types in a language neutral format; pickle does it for almost anything, but only for Python.

import pickle

data = {"name": "Meera", "scores": [92, 78], "point": (1, 2), "tags": {"a", "b"}}

encoded = pickle.dumps(data)
print(type(encoded), len(encoded))          # <class 'bytes'>

restored = pickle.loads(encoded)
print(restored == data)                      # True
print(type(restored["point"]))               # tuple - preserved exactly
print(type(restored["tags"]))                # set   - preserved exactly

pickle compared with JSON

Files

import pickle

data = {"cache": list(range(1000)), "version": 3}

with open("cache.pkl", "wb") as handle:        # binary mode, always
    pickle.dump(data, handle)

with open("cache.pkl", "rb") as handle:
    loaded = pickle.load(handle)

print(loaded["version"], len(loaded["cache"]))

Pickle files are binary. Opening one in text mode raises or corrupts the data.

Custom objects

import pickle


class Account:
    def __init__(self, owner, balance):
        self.owner = owner
        self.balance = balance
        self.history = []

    def deposit(self, amount):
        self.balance += amount
        self.history.append(("deposit", amount))
        return self

    def __repr__(self):
        return f"Account({self.owner!r}, {self.balance}, {len(self.history)} entries)"


account = Account("Meera", 1000).deposit(500).deposit(250)

encoded = pickle.dumps(account)
restored = pickle.loads(encoded)

print(restored)
print(restored.history)
print(type(restored))                # the class, fully restored

Pickle stores the class's name and module, not its code. Unpickling imports the class and rebuilds the instance. If the class has been renamed, moved or deleted, unpickling fails.

Controlling what is pickled

import pickle


class Connection:
    def __init__(self, host):
        self.host = host
        self.socket = "an open socket"       # cannot and should not be pickled
        self.cache = {}

    def __getstate__(self):
        """Return what should be stored."""
        state = self.__dict__.copy()
        del state["socket"]                  # drop the unpicklable part
        state["cache"] = {}                  # do not persist the cache
        return state

    def __setstate__(self, state):
        """Rebuild from what was stored."""
        self.__dict__.update(state)
        self.socket = None                   # reconnect later
        print("  restored, socket must be reopened")

    def __repr__(self):
        return f"Connection({self.host!r}, socket={self.socket!r})"


c = Connection("localhost")
c.cache["key"] = "value"

restored = pickle.loads(pickle.dumps(c))
print(restored)
print(restored.cache)                        # {} - deliberately not stored

What cannot be pickled

import pickle

# These all raise
# pickle.dumps(lambda x: x)               # a lambda
# pickle.dumps(open("f.txt"))             # an open file
# pickle.dumps((n for n in range(3)))     # a generator
# pickle.dumps(threading.Lock())          # a lock


def named(x):
    return x


print(len(pickle.dumps(named)))            # a module level function IS picklable
                                            # - only its name is stored

• Lambdas and nested functions: no name to store.
• Open files, sockets, database connections, locks: operating system resources.
• Generators and running frames.
• A module level function or class: fine, because the reference can be stored by name.

The security problem

import pickle


class Innocent:
    def __reduce__(self):
        """Pickle calls this to decide how to rebuild the object."""
        import os
        return (os.system, ("echo this could have been any command",))


payload = pickle.dumps(Innocent())

# Unpickling RUNS the command
pickle.loads(payload)

Unpickling executes code. A crafted pickle can run any command with your program's permissions. Never unpickle data from a network request, a file upload, a user supplied path, or any source you do not control. This is not a theoretical risk; it is a standard attack.

import hmac
import hashlib
import pickle

SECRET = b"a key kept out of source control"


def sign(data):
    payload = pickle.dumps(data)
    signature = hmac.new(SECRET, payload, hashlib.sha256).digest()
    return signature + payload


def verify_and_load(blob):
    signature, payload = blob[:32], blob[32:]
    expected = hmac.new(SECRET, payload, hashlib.sha256).digest()
    if not hmac.compare_digest(signature, expected):
        raise ValueError("signature does not match; refusing to unpickle")
    return pickle.loads(payload)


blob = sign({"a": 1})
print(verify_and_load(blob))

try:
    verify_and_load(blob[:32] + b"tampered")
except ValueError as error:
    print(error)

If pickled data must cross a boundary, sign it and verify the signature before unpickling. Better still: use JSON, and accept the smaller type mapping.

Protocol versions

import pickle

data = {"values": list(range(1000))}

print(pickle.HIGHEST_PROTOCOL)
print(pickle.DEFAULT_PROTOCOL)

for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
    print(f"protocol {protocol}: {len(pickle.dumps(data, protocol))} bytes")

Higher protocols are smaller and faster but cannot be read by older Python versions. The default is a sensible compromise; specify one explicitly only when compatibility matters.

Practical uses

Caching an expensive result

import pickle
import time
from pathlib import Path


def expensive_computation(n):
    time.sleep(1)
    return {i: i * i for i in range(n)}


def cached(n, path="result.pkl"):
    cache = Path(path)
    if cache.exists():
        with open(cache, "rb") as handle:
            stored = pickle.load(handle)
        if stored["n"] == n:
            print("  loaded from cache")
            return stored["result"]

    result = expensive_computation(n)
    with open(cache, "wb") as handle:
        pickle.dump({"n": n, "result": result}, handle)
    return result


start = time.perf_counter()
cached(1000)
print(f"first:  {time.perf_counter() - start:.2f}s")

start = time.perf_counter()
cached(1000)
print(f"second: {time.perf_counter() - start:.2f}s")

Saving program state

import pickle
from dataclasses import dataclass, field


@dataclass
class GameState:
    level: int = 1
    score: int = 0
    inventory: list = field(default_factory=list)
    visited: set = field(default_factory=set)


def save(state, path="save.pkl"):
    with open(path, "wb") as handle:
        pickle.dump(state, handle)


def load(path="save.pkl"):
    try:
        with open(path, "rb") as handle:
            return pickle.load(handle)
    except (FileNotFoundError, pickle.UnpicklingError, EOFError):
        return GameState()


state = GameState(level=3, score=1200, inventory=["key"], visited={"a", "b"})
save(state)

restored = load()
print(restored)
print(type(restored.visited))         # set - preserved, unlike with JSON

Other formats

import csv
from io import StringIO

rows = [
    {"name": "Meera", "score": 92},
    {"name": "Arun", "score": 78},
]

buffer = StringIO()
writer = csv.DictWriter(buffer, fieldnames=["name", "score"])
writer.writeheader()
writer.writerows(rows)
print(buffer.getvalue())

buffer.seek(0)
for row in csv.DictReader(buffer):
    print(row)

import csv

# Always pass newline="" - the csv module handles line endings itself
with open("scores.csv", "w", newline="", encoding="utf-8") as handle:
    writer = csv.writer(handle)
    writer.writerow(["name", "score"])
    writer.writerows([["Meera", 92], ["Arun", 78]])

with open("scores.csv", newline="", encoding="utf-8") as handle:
    for row in csv.reader(handle):
        print(row)

import sqlite3

connection = sqlite3.connect(":memory:")
connection.execute("CREATE TABLE notes (id INTEGER PRIMARY KEY, title TEXT)")
connection.executemany(
    "INSERT INTO notes (title) VALUES (?)",          # parameters, never formatting
    [("first",), ("second",)],
)
connection.commit()

for row in connection.execute("SELECT id, title FROM notes"):
    print(row)

connection.close()

Common mistakes

• Unpickling data from an untrusted source.
• Opening a pickle file in text mode.
• Pickling a lambda, an open file or a lock.
• Renaming or moving a class and then failing to load old pickles.
• Using pickle for data another language must read.
• Using pickle for long term storage, where a class change breaks every stored file.
• Forgetting newline="" when writing CSV.

Best practices

• Use JSON by default; use pickle only for Python-only data you created.
• Never unpickle anything you did not write, and sign it if it must travel.
• Open pickle files in binary mode.
• Store a version number alongside pickled data so old files can be detected.
• Use __getstate__ to exclude resources and caches.
• Prefer SQLite or JSON for anything meant to last.

Practice

1. Round trip a structure containing a tuple, a set and a custom class, and compare with JSON.
2. Write a class that excludes an unpicklable attribute using __getstate__.
3. Build a disk cache that stores a computed result and reuses it.
4. Explain in two sentences why unpickling untrusted data is dangerous.
5. Convert the same records to JSON, CSV and pickle and compare the file sizes.

Conclusion

Pickle preserves Python objects exactly, which makes it ideal for caches and checkpoints and unsuitable for anything crossing a trust boundary. Unpickling runs code. Use JSON whenever the data leaves your program.