First-Class and Higher-Order Functions

Functions are objects

def double(n):
    return n * 2


print(type(double))          # <class 'function'>
print(double.__name__)       # double
print(double.__doc__)        # None

double.category = "maths"    # you can even attach attributes
print(double.category)

"First class" means a function is treated like any other value. There is no special category for it in the language, and everything below follows from that.

Storing a function in a variable

def double(n):
    return n * 2


twice = double               # NOT double() - no call, just the object
print(twice(5))              # 10
print(twice is double)       # True

The single most common error here is writing twice = double(). The brackets call the function; without them you refer to it. callback = handler() stores the result, which is usually None.

Storing functions in collections

operations = {
    "add": lambda a, b: a + b,
    "subtract": lambda a, b: a - b,
}


def multiply(a, b):
    return a * b


operations["multiply"] = multiply

print(operations["multiply"](6, 7))      # 42

for name in sorted(operations):
    print(f"{name:<10}{operations[name](10, 3)}")

A dictionary of functions is a dispatch table. It replaces a long elif ladder, and new behaviour is added by inserting a key rather than editing a chain of conditions.

def handle_add(payload):
    return f"adding {payload}"


def handle_delete(payload):
    return f"deleting {payload}"


HANDLERS = {"add": handle_add, "delete": handle_delete}


def dispatch(command, payload):
    handler = HANDLERS.get(command)
    if handler is None:
        return f"unknown command: {command}"
    return handler(payload)


print(dispatch("add", "note-1"))
print(dispatch("archive", "note-1"))

Passing a function as an argument

A function that takes or returns another function is a higher-order function.

def apply_twice(func, value):
    return func(func(value))


print(apply_twice(lambda n: n * 3, 2))      # 18
print(apply_twice(str.upper, "ab"))         # AB

def transform_all(items, func):
    return [func(item) for item in items]


print(transform_all([1, 2, 3], lambda n: n ** 2))     # [1, 4, 9]
print(transform_all(["a", "b"], str.upper))           # ['A', 'B']

The higher-order functions you already use

words = ["banana", "kiwi", "apple"]

print(sorted(words, key=len))                    # key is a function
print(max(words, key=len))
print(list(map(str.upper, words)))
print(list(filter(lambda w: "a" in w, words)))
print(any(map(str.isupper, words)))

Returning a function

def multiplier(factor):
    def multiply(n):
        return n * factor
    return multiply                 # returning the function, not calling it


triple = multiplier(3)
tenfold = multiplier(10)

print(triple(7))        # 21
print(tenfold(7))       # 70

multiplier is a factory: it builds and hands back a new function configured with factor. The returned function remembers factor even though multiplier has already finished. That memory is a closure, and it has its own note.

Callbacks

def process(items, on_success=None, on_error=None):
    for item in items:
        try:
            value = int(item)
        except ValueError:
            if on_error:
                on_error(item)
            continue
        if on_success:
            on_success(value)


process(
    ["10", "abc", "30"],
    on_success=lambda v: print("ok:", v),
    on_error=lambda raw: print("bad:", raw),
)

Passing a function in lets the caller decide what happens, without process knowing anything about printing, logging or storing.

functools.partial

from functools import partial


def power(base, exponent):
    return base ** exponent


square = partial(power, exponent=2)
cube = partial(power, exponent=3)
two_to_the = partial(power, 2)

print(square(7), cube(3), two_to_the(10))     # 49 27 1024


def log(level, message):
    print(f"[{level}] {message}")


warn = partial(log, "WARNING")
warn("disk almost full")

partial fixes some arguments and returns a new callable. It is the tidy alternative to writing a wrapper lambda purely to bake in a value.

Sorting keys built at runtime

import operator

records = [
    {"name": "Meera", "dept": "eng", "salary": 90000},
    {"name": "Arun", "dept": "design", "salary": 75000},
    {"name": "Sara", "dept": "eng", "salary": 82000},
]


def sort_by(field, descending=False):
    return sorted(records, key=operator.itemgetter(field), reverse=descending)


for row in sort_by("salary", descending=True):
    print(row["name"], row["salary"])

Introspection

import inspect


def net_price(amount, tax_rate=0.18):
    """Return the amount including tax."""
    return amount * (1 + tax_rate)


print(net_price.__name__)                        # net_price
print(net_price.__doc__)                         # the docstring
print(inspect.signature(net_price))              # (amount, tax_rate=0.18)
print(list(inspect.signature(net_price).parameters))   # ['amount', 'tax_rate']
print(callable(net_price), callable(42))         # True False

callable(x) answers "can this be called?" for functions, classes, methods and any object defining __call__.

Making an object callable

class Multiplier:
    def __init__(self, factor):
        self.factor = factor

    def __call__(self, n):
        return n * self.factor


triple = Multiplier(3)
print(triple(7))                 # 21
print(callable(triple))          # True
print(list(map(triple, [1, 2, 3])))   # [3, 6, 9]

A class with __call__ behaves like a function while also holding state you can inspect and change. It is the object oriented alternative to a closure.

Common mistakes

• Writing callback = handler() instead of callback = handler.
• Returning inner() from a factory instead of inner.
• Forgetting that map and filter are lazy and must be wrapped in list() to see them.
• Passing a bound method and being surprised that it carries its object with it.
• Building a dispatch table with {"add": handle_add()}, calling every handler at definition time.
• Using a lambda to fix one argument where partial would be clearer.

Best practices

• Use a dispatch dictionary instead of a long elif ladder over one value.
• Accept a function argument when the caller should decide part of the behaviour.
• Use partial and operator.itemgetter rather than trivial lambdas.
• Give factory functions names that describe what they build.
• Use a callable class when the behaviour needs state you want to inspect.

Practice

1. Build a dispatch table for five text operations and drive it from user input.
2. Write apply_n_times(func, value, n) and use it to compound a value.
3. Write a factory that returns a validator function for a given minimum length.
4. Rewrite three trivial lambdas using partial or operator.
5. Write a callable class that counts how many times it has been called.

Conclusion

Functions are values. Store them in dictionaries to replace conditionals, pass them in to let callers choose behaviour, and return them to build configured functions at runtime. Every advanced feature ahead - closures, decorators, generators - is built on that one idea.