Descriptors: The Machinery Behind Attributes

The protocol

A descriptor is any class defining one or more of three methods:

The descriptor object is stored on the class, and it intercepts access on every instance.

class Traced:
    def __set_name__(self, owner, name):
        self.name = name                  # Python tells us the attribute name

    def __get__(self, obj, objtype=None):
        print(f"  reading {self.name}")
        return obj.__dict__.get(self.name)

    def __set__(self, obj, value):
        print(f"  writing {self.name} = {value!r}")
        obj.__dict__[self.name] = value


class Product:
    name = Traced()
    price = Traced()

    def __init__(self, name, price):
        self.name = name
        self.price = price


p = Product("Notebook", 250)
print(p.name)
p.price = 300

  writing name = 'Notebook'
  writing price = 250
  reading name
Notebook
  writing price = 300

__set_name__

class Field:
    def __set_name__(self, owner, name):
        """Called automatically when the class body is executed."""
        print(f"Field assigned to {owner.__name__}.{name}")
        self.public_name = name
        self.private_name = "_" + name


class Example:
    title = Field()
    subtitle = Field()

Before Python 3.6 you had to pass the name in by hand: title = Field("title"). __set_name__ removes that duplication, and it is what makes descriptors pleasant to write.

Data and non-data descriptors

class NonData:
    def __get__(self, obj, objtype=None):
        return "from the descriptor"


class Data:
    def __get__(self, obj, objtype=None):
        return "from the descriptor"

    def __set__(self, obj, value):
        obj.__dict__["data"] = value


class Example:
    non_data = NonData()
    data = Data()


e = Example()

e.__dict__["non_data"] = "from the instance"
print(e.non_data)          # from the instance  <- the instance wins

e.__dict__["data"] = "from the instance"
print(e.data)              # from the descriptor <- the descriptor wins

The full lookup order for obj.x is: data descriptor on the class, then the instance dictionary, then non-data descriptor or plain class attribute, then __getattr__. This is why cached_property works - it is a non-data descriptor that writes into the instance dictionary on first access, and is never consulted again.

A validating descriptor

class Validated:
    """Base class for descriptors that check a value before storing it."""

    def __set_name__(self, owner, name):
        self.name = "_" + name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self                       # accessed on the class itself
        return getattr(obj, self.name)

    def __set__(self, obj, value):
        self.validate(value)
        setattr(obj, self.name, value)

    def validate(self, value):
        raise NotImplementedError


class Number(Validated):
    def __init__(self, minimum=None, maximum=None):
        self.minimum = minimum
        self.maximum = maximum

    def validate(self, value):
        if not isinstance(value, (int, float)):
            raise TypeError(f"expected a number, got {type(value).__name__}")
        if self.minimum is not None and value < self.minimum:
            raise ValueError(f"{value} is below the minimum {self.minimum}")
        if self.maximum is not None and value > self.maximum:
            raise ValueError(f"{value} is above the maximum {self.maximum}")


class Text(Validated):
    def __init__(self, min_length=0, max_length=None):
        self.min_length = min_length
        self.max_length = max_length

    def validate(self, value):
        if not isinstance(value, str):
            raise TypeError("expected a string")
        if len(value) < self.min_length:
            raise ValueError(f"must be at least {self.min_length} characters")
        if self.max_length is not None and len(value) > self.max_length:
            raise ValueError(f"must be at most {self.max_length} characters")


class Product:
    name = Text(min_length=2, max_length=40)
    price = Number(minimum=0)
    discount = Number(minimum=0, maximum=90)

    def __init__(self, name, price, discount=0):
        self.name = name
        self.price = price
        self.discount = discount

    def __repr__(self):
        return f"Product({self.name!r}, {self.price}, {self.discount}%)"


print(Product("Notebook", 250, 10))

for bad in [("N", 250, 0), ("Notebook", -5, 0), ("Notebook", 250, 95)]:
    try:
        Product(*bad)
    except (TypeError, ValueError) as error:
        print(f"{bad}: {error}")

The validation rules are declared once, at the top of the class, and reused across every field and every class in the program. Doing the same thing with properties would mean writing a getter and a setter per field.

Descriptors versus properties

# With properties: one pair of methods per attribute
class Product:
    def __init__(self, price, cost, tax):
        self.price = price
        self.cost = cost
        self.tax = tax

    @property
    def price(self):
        return self._price

    @price.setter
    def price(self, value):
        if value < 0:
            raise ValueError("price cannot be negative")
        self._price = value

    # ... and the same twelve lines again for cost, and again for tax


# With a descriptor: the rule is written once
class Product:
    price = Number(minimum=0)
    cost = Number(minimum=0)
    tax = Number(minimum=0)

A property is a data descriptor. Reaching for a custom one is worthwhile only when the same rule repeats.

What is already a descriptor

class Example:
    attribute = 42

    def method(self):
        return "a method"

    @property
    def prop(self):
        return "a property"

    @classmethod
    def cls_method(cls):
        return "a class method"

    @staticmethod
    def static_method():
        return "a static method"


for name in ["attribute", "method", "prop", "cls_method", "static_method"]:
    value = Example.__dict__[name]
    print(f"{name:<16}{type(value).__name__:<16}get={hasattr(value, '__get__')}")

attribute       int             get=False
method          function        get=True
prop            property        get=True
cls_method      classmethod     get=True
static_method   staticmethod    get=True

A plain function is a non-data descriptor. When you write obj.method, its __get__ runs and returns a bound method with self already supplied. That is how self gets passed, and it is descriptors doing the work.

Lazy loading with a descriptor

class LazyLoad:
    """Compute on first access, then replace itself in the instance dictionary."""

    def __init__(self, factory):
        self.factory = factory

    def __set_name__(self, owner, name):
        self.name = name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        print(f"  computing {self.name}")
        value = self.factory(obj)
        obj.__dict__[self.name] = value        # non-data, so this now wins
        return value


class Report:
    def __init__(self, rows):
        self.rows = rows

    total = LazyLoad(lambda self: sum(self.rows))
    average = LazyLoad(lambda self: sum(self.rows) / len(self.rows))


r = Report([10, 20, 30])
print(r.total)        # computing total -> 60
print(r.total)        # 60, no recomputation
print(r.average)

This is functools.cached_property in miniature, and it shows exactly why the data versus non-data distinction matters.

Unit conversion

class Unit:
    """Store one canonical value, expose several units."""

    def __init__(self, factor, name):
        self.factor = factor
        self.name = name

    def __set_name__(self, owner, name):
        self.attribute = "_base"

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return getattr(obj, self.attribute) / self.factor

    def __set__(self, obj, value):
        setattr(obj, self.attribute, value * self.factor)


class Distance:
    metres = Unit(1, "m")
    kilometres = Unit(1000, "km")
    miles = Unit(1609.344, "mi")

    def __init__(self, metres=0):
        self.metres = metres

    def __repr__(self):
        return f"Distance({self.metres:.1f}m)"


d = Distance(5000)
print(f"{d.metres:.0f} m")
print(f"{d.kilometres:.2f} km")
print(f"{d.miles:.2f} mi")

d.miles = 1
print(f"{d.metres:.2f} m")        # 1609.34

One stored value, three views, no possibility of them disagreeing.

Common mistakes

• Storing the value on the descriptor itself, so every instance shares it:

class Broken:
    def __set__(self, obj, value):
        self.value = value            # WRONG - one descriptor, many instances

    def __get__(self, obj, objtype=None):
        return self.value


class Thing:
    field = Broken()


a, b = Thing(), Thing()
a.field = 1
b.field = 2
print(a.field)          # 2 - a and b share the descriptor

• Forgetting to handle obj is None, so accessing the attribute on the class raises.
• Naming the private attribute the same as the descriptor, causing infinite recursion.
• Putting a descriptor on an instance rather than the class; it will not be invoked.
• Using a descriptor where a property would be shorter and clearer.

Best practices

• Store per instance data in obj.__dict__ or under a name derived in __set_name__.
• Always handle obj is None by returning self.
• Use __set_name__ rather than passing the name in.
• Reach for a descriptor only when the same rule applies to several attributes.
• Give the descriptor class a name that reads well in a class body: Number, Text, Positive.

Practice

1. Write a descriptor enforcing that a value is a non-empty string, and use it on three attributes.
2. Demonstrate the shared state bug and fix it.
3. Show that a data descriptor beats the instance dictionary and a non-data descriptor does not.
4. Write a descriptor that logs every read and write of an attribute.
5. Explain, using descriptors, how obj.method() gets self passed automatically.

Conclusion

Descriptors are how attribute access is customised in Python, and they are already everywhere - every method, property, class method and static method is one. Write your own when the same validation or computation applies across many attributes; otherwise a property is plenty.