Multiprocessing

Why processes

import time
from multiprocessing import Pool


def heavy(n):
    return sum(i * i for i in range(n))


if __name__ == "__main__":
    values = [3_000_000] * 4

    start = time.perf_counter()
    [heavy(n) for n in values]
    print(f"sequential: {time.perf_counter() - start:.2f}s")

    start = time.perf_counter()
    with Pool(4) as pool:
        pool.map(heavy, values)
    print(f"parallel:   {time.perf_counter() - start:.2f}s")

sequential: 4.20s
parallel:   1.15s        on a machine with four or more cores

Threads cannot do this, because only one thread executes Python bytecode at a time. Each process has its own interpreter and its own lock, so they genuinely run at once.

The if __name__ == "__main__": guard is required on Windows and macOS. Child processes import the main module to find the target function, and without the guard they would re-run the process creation code, forking endlessly.

Creating a process

import os
from multiprocessing import Process


def worker(name):
    print(f"{name} running in process {os.getpid()}")


if __name__ == "__main__":
    print(f"main process {os.getpid()}")

    processes = [Process(target=worker, args=(f"w{i}",)) for i in range(3)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print("all finished")

Nothing is shared

from multiprocessing import Process

counter = 0


def increment():
    global counter
    for _ in range(100_000):
        counter += 1
    print("child sees:", counter)


if __name__ == "__main__":
    processes = [Process(target=increment) for _ in range(2)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()
    print("parent sees:", counter)      # 0 - each child had its own copy

Each process has its own memory. A global changed in a child is invisible to the parent and to every sibling. Everything that crosses a process boundary must be sent explicitly, and must be picklable.

Pool

from multiprocessing import Pool


def square(n):
    return n * n


def add(pair):
    a, b = pair
    return a + b


if __name__ == "__main__":
    with Pool(4) as pool:
        print(pool.map(square, range(10)))               # ordered results
        print(pool.starmap(add, [(1, 2), (3, 4)]))       # unpacks each tuple

        for result in pool.imap_unordered(square, range(5)):
            print(result, end=" ")                        # as they arrive
        print()

        handle = pool.apply_async(square, (12,))
        print(handle.get(timeout=5))                      # a single async call

ProcessPoolExecutor

from concurrent.futures import ProcessPoolExecutor, as_completed


def heavy(n):
    if n < 0:
        raise ValueError("n must not be negative")
    return sum(i * i for i in range(n))


if __name__ == "__main__":
    inputs = [1_000_000, 2_000_000, -1, 500_000]

    with ProcessPoolExecutor(max_workers=4) as pool:
        futures = {pool.submit(heavy, n): n for n in inputs}
        for future in as_completed(futures):
            n = futures[future]
            try:
                print(f"{n}: {future.result()}")
            except Exception as error:
                print(f"{n}: failed - {error}")

The interface is identical to ThreadPoolExecutor. Swapping one for the other is a one word change, which makes it easy to test whether a workload is bound by computation or by waiting.

Sending data between processes

A queue

from multiprocessing import Process, Queue


def producer(q, count):
    for i in range(count):
        q.put(i * i)
    q.put(None)                     # a sentinel


def consumer(q):
    total = 0
    while True:
        item = q.get()
        if item is None:
            break
        total += item
    print("total:", total)


if __name__ == "__main__":
    q = Queue()
    p1 = Process(target=producer, args=(q, 10))
    p2 = Process(target=consumer, args=(q,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

A pipe

from multiprocessing import Process, Pipe


def child(connection):
    connection.send({"status": "done", "value": 42})
    connection.close()


if __name__ == "__main__":
    parent_end, child_end = Pipe()
    p = Process(target=child, args=(child_end,))
    p.start()
    print(parent_end.recv())
    p.join()

Shared memory

from multiprocessing import Process, Value, Array, Lock


def increment(counter, lock, times):
    for _ in range(times):
        with lock:
            counter.value += 1


if __name__ == "__main__":
    counter = Value("i", 0)          # a shared integer
    numbers = Array("d", [1.0, 2.0]) # a shared array of doubles
    lock = Lock()

    processes = [Process(target=increment, args=(counter, lock, 10_000))
                 for _ in range(4)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print(counter.value)             # 40000
    print(list(numbers))

Shared memory still needs a lock, for exactly the same reason threads do. It is faster than a queue for simple values and much more awkward for anything structured.

A manager

from multiprocessing import Process, Manager


def worker(shared_dict, shared_list, name):
    shared_dict[name] = name.upper()
    shared_list.append(name)


if __name__ == "__main__":
    with Manager() as manager:
        d = manager.dict()
        l = manager.list()

        processes = [Process(target=worker, args=(d, l, f"w{i}"))
                     for i in range(3)]
        for p in processes:
            p.start()
        for p in processes:
            p.join()

        print(dict(d))
        print(list(l))

A Manager runs a server process that owns the real objects; every access is sent to it. That makes ordinary dictionaries and lists usable across processes, at the cost of speed.

Everything sent must be picklable

from multiprocessing import Pool


def top_level(n):                    # fine
    return n * n


if __name__ == "__main__":
    with Pool(2) as pool:
        print(pool.map(top_level, [1, 2, 3]))

    # These all fail, because they cannot be pickled:
    # pool.map(lambda n: n * n, [1, 2])          - a lambda
    # pool.map(open("f").read, [1])              - an open file
    # a locally defined function inside another function

Arguments and return values are serialised to be sent between processes. Lambdas, nested functions, open files, sockets, locks and database connections cannot cross a process boundary. Use a module level function, and pass plain data.

Start methods

import multiprocessing

print(multiprocessing.get_start_method())
print(multiprocessing.get_all_start_methods())

if __name__ == "__main__":
    multiprocessing.set_start_method("spawn", force=True)

With spawn, the child re-imports your module. That is why the main guard matters, and why module level code with side effects causes strange behaviour.

Choosing a worker count

import os

cores = os.cpu_count()
print(f"{cores} logical processors")

cpu_workers = cores                # computation: about one per core
io_workers = cores * 4             # waiting: more than cores is fine

More processes than cores does not help CPU bound work, and each one costs memory and startup time. Measure rather than guess.

The overhead is real

import time
from multiprocessing import Pool


def tiny(n):
    return n + 1


if __name__ == "__main__":
    data = list(range(100_000))

    start = time.perf_counter()
    [tiny(n) for n in data]
    print(f"sequential: {time.perf_counter() - start:.3f}s")

    start = time.perf_counter()
    with Pool(4) as pool:
        pool.map(tiny, data)
    print(f"parallel:   {time.perf_counter() - start:.3f}s")   # SLOWER

Starting processes and pickling data costs far more than the work here. Multiprocessing pays only when each task takes meaningfully longer than the round trip - use chunksize to batch small tasks, or do not parallelise at all.

with Pool(4) as pool:
    pool.map(tiny, data, chunksize=10_000)      # far fewer round trips

Threads, processes and asyncio

Common mistakes

• Omitting the if __name__ == "__main__": guard.
• Expecting a global changed in a child to be visible in the parent.
• Passing a lambda or a nested function to a pool.
• Parallelising work that is too small to be worth it.
• Creating far more processes than cores.
• Sharing a value without a lock.
• Forgetting to join(), or not using the pool as a context manager.

Best practices

• Use ProcessPoolExecutor or Pool rather than raw Process objects.
• Always guard the entry point with the main check.
• Keep worker functions at module level and pass plain data.
• Set chunksize when tasks are small and numerous.
• Measure before and after; the overhead is often decisive.
• Use threads for waiting, processes for computing.

Practice

1. Compute a CPU heavy result sequentially and with a pool, and report the speedup.
2. Show that a global variable is not shared between processes.
3. Pass results back from four workers using a Queue.
4. Demonstrate a case where multiprocessing is slower than sequential code, and explain why.
5. Swap ProcessPoolExecutor for ThreadPoolExecutor on the same workload and compare.

Conclusion

Processes are how Python uses more than one core. They cost startup time and cannot share memory, so send plain data, keep tasks large enough to be worth the round trip, and always guard the entry point.