Decorators Are Fun

Author:	Matt Wilson <matt@tplus1.com>

What are decorators?

Decorators are functions that wrap around other functions. Sometimes a decorator monkeys with the stuff going into or out of the decorated function.

Decorators depend on two fun facts...

Functions can be parameters

map applies a function to every element of a sequence.

>>> map(len,  ['a', 'bb', 'ccc'])
[1, 2, 3]
>>> from string import upper
>> map(upper, ['a', 'bb', 'ccc'])
['A', 'BB', 'CCC']

Functions can make functions

>>> def add_n_to_x(n):
...     def add_to_x(x):
...         return n + x
...     return add_to_x
...
>>> add3_to_x = add_n_to_x(3)

What is add3_to_x?

>>> type(add3_to_x)
<type 'function'>

Functions making functions

What will add3_to_x(8) return?

>>> add3_to_x(8)
11

What about add_n_to_x(7)(3)?

>>> add_n_to_x(7)(3)
10

Partial function application

Ruby won't get this until 1.9, so feel smug.

>>> def f(a, b):
...     "Returns a minus b"
...     return a - b
...
>>> from functools import partial
>>> subtract_one = partial(f, b=1)

People argue over whether this is currying or not.

Usage

functools.partial returns a new object based on the original function, except one or more parameters are set to fixed values.

>>> subtract_one(5)
4

>>> subtract_one_from_three = \
... partial(subtract_one, a=3)

>>> subtract_one_from_three()
2

Stop for discussion

Simple decorator examples

"What you do at Initech is, you take the specifications from the customers and you bring them down to the software engineers."

smykowsky

>>> def smykowsky(f):
...
...     def h(*args, **kwargs):
...         return f(*args, **kwargs)
...
...     return h

Every time smykowsky is called, it makes a new function h and then returns it.
h doesn't get called when smykowsky gets called.

Worse than useless

smykowsky mangles the function signature of the function he decorates.

f requires exactly two parameters:

>>> f(3, 1)
2
>>> f(1, 2, 3) # kaboom!
Traceback (most recent call last):
    ...
TypeError: f() takes exactly 2 arguments (3 given)

Worse than useless

We can inspect the function f to find out the parameters it requires:

>>> import inspect
>>> inspect.getargspec(f)
(['a', 'b'], None, None, None)

Worse than useless

Now see what happens when smykowski gets involved:

>>> h = smykowsky(f)
>>> inspect.getargspec(h)
([], 'args', 'kwargs', None)

The @ symbol

>>> @smykowsky
... def g():
...     return True

Is a shortcut for:

>>> def g():
...     return True
...
>>> g = smykowsky(g)

The downside of @

The inner function is trapped forever!

Less ridiculous

>>> def as_string(f):
...     "convert results of function f to string."
...
...     def g(*args, **kwargs):
...         return str(f(*args, **kwargs))
...     return g

What will as_string(f)(3, 1) + "abc" return?

>>> as_string(f)(3, 1) + "abc"
'2abc'

Use a decorator for logging

Panicked developers sometimes do this:

>>> from decoratortalk import log
>>> def gross(a, b):
...     log.debug("a is %s and b is %s." % (a, b))
...     y = f(a, b)
...     log.debug("y is %s." % y)
...     return y

That logging can be moved into a decorator instead...

Use a decorator for logging

>>> from decoratortalk import debuglogger
>>> f(3, 1)
2
>>> logging_f = debuglogger(f)
>>> logging_f(3, 1)
args is (3, 1) and kwargs is {}.
results from calling f is 2.
2

Why use decorators?

Reuse code
Is a gateway drug to the larger world of functional programming
Separate code into layers, in other words, aspects
You tell me

Build a decorator factory

add1

I'll revise this in the next slide:

>>> def add1(f):
...     "Return a new function returns 1 + f(...)"
...
...     def g(*args, **kwargs):
...         return 1 + f(*args, **kwargs)
...
...     return g

add1 revisited

Instead of writing add1 from scratch, it would be much nicer to be able to do something like:

>>> from decoratortalk import add_n
>>> add1 = add_n(1)

We could reuse add_n to make other decorators too:

>>> subtract5 = add_n(-5)

Introducing add_n

>>> def add_n(n):
...     "Return a decorator that adds n."
...     def dec(f):
...         def h(*args, **kwargs):
...             return n + f(*args, **kwargs)
...         return h
...     return dec

The decorator module

Two problems with decorators so far:

They mangle the function signature.
Writing decorator factories requires too much thinking.

Smykowsky part two

You have to write your decorator with a parameter f.

>>> from decorator import decorator
>>> def smykowsky2(f, *args, **kwargs):
...     return f(*args, **kwargs)

Trying it out

>>> dec = decorator(smykowsky2)
>>> f2 = dec(f)
>>> from inspect import getargspec as g
>>> g(f) == g(f2)
True

We get the docstring too!

>>> print f.__doc__
Returns a minus b
>>> f3 = smykowsky(f)
>>> print f3.__doc__
None
>>> print f2.__doc__
Returns a minus b

Less silly example

Functions often coerce parameters into certain types, sort of like this:

>>> def j(x):
...     if isinstance(x, (list, tuple)):
...         xlist = x
...     else:
...         xlist = [x]
...
...     # Now do the real stuff

Use a decorator

>>> def as_list(f, x):
...     "Assure the arg x is a list."
...     if isinstance(x, (list, tuple)):
...         xlist = x
...     else:
...         xlist = [x]
...     return f(xlist)

Test drive

>>> from decorator import decorator
>>> @decorator(as_list)
... def j2(x):
...     print type(x), x
...

>>> j2(99)
<type 'list'> [99]

>>> j2([1, 2])
<type 'list'> [1, 2]

Better decorator factories

The decorator module version also offers a better approach for decorator factories:

>>> class AddX(object):
...    def __init__(self, x):
...        self.x = x
...    def call(self, f, *args, **kwargs):
...        return self.x + f(*args, **kwargs)
...
>>> AddX = decorator(AddX)

Better decorator factories

>>> @AddX(13)
... def f(a, b):
...    return a + b

What will f(1, 1) return?

>>> f(1, 1)
15

Some real-world examples

Developers working on Python 3k

classmethod

Both classes and instances can call methods decorated as classmethods.

>>> class C(object):
...     @classmethod
...     def spam(cls):
...         return "delicious!"
...

classmethod

Now classes and instances can call these methods.

>>> C.spam()
'delicious!'
>>> c1 = C()
>>> c1.spam()
'delicious!'

Alternate constructors

>>> d = {}
>>> d.fromkeys('abc', 99)
{'a': 99, 'c': 99, 'b': 99}
>>> dict.fromkeys('def', 98)
{'e': 98, 'd': 98, 'f': 98}

An instance made other new instances.

staticmethod

If you decorate with @staticmethod, your method won't get an extra parameter like self or cls. Some reasons to use staticmethod:

Make Java programmers feel less homesick
Put associated functions inside the relevant class, so we don’t pollute the module namespace
Makes it obvious to the reader that a method won't alter the object.

staticmethods for DI

staticmethods useful when you want to add functions to a class:

>>> class C(object):
...     g1 = f
...     g2 = staticmethod(f)
>>> i = C()

What will happen if I call i.g1(3, 1)?

staticmethods for DI

>>> i.g1(3, 1)
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
TypeError: f() takes exactly 2 arguments (3 given)

However, g2 will work fine:

>>> i.g2(9, 8)
1

Properties

They look like attributes, but are really methods.

>>> class Rectangle(object):
...     def calculate_area(self):
...         return self.height * self.width
...
...     area = property(calculate_area)

Properties

>>> r = Rectangle()
>>> r.height, r.width = 3, 4
>>> r.calculate_area()
12
>>> r.area
12

>>> getattr(r, 'area')
12

property can be a decorator

>>> class Square(object):
...     @property
...     def area(self):
...         return self.side ** 2
...
>>> sq = Square()
>>> sq.side = 3
>>> sq.area
9

properties in 2.6

Specify setters and deleters as properties:

class C(object):
    @property
    def x(self):
        return self._x

    @x.setter
    def x(self, value):
        self._x = value

properties in 2.6

Subclasses can tweak getters:

class D(C):
    @C.x.getter
    def x(self):
        return self._x * 2

    @x.setter
    def x(self, value):
        self._x = value / 2

Decorators can just watch

A decorator doesn't HAVE to tweak inputs or return values.

You can use a decorator to tell another system about the presence of a function.

>>> functions = list()
>>> def register_function(f):
...     functions.append(f)
...     return f
...

Generic functions

Dispatching on parameters rather than types

Pet groomer example

Groom dogs and cats differently.
Offer a standard and a premium package.

Four possible procedures total {A, B, C, D}

...	Dog	Cat
Standard	A	C
Premium	B	D

First version of groom

>>> def groom1(pet, package):
...     "Groom according to pet and package types"
...     if isinstance(pet, Dog):
...         if isinstance(package, Standard):
...             A()
...         else:
...             B()

First version of groom

...     elif isinstance(pet, Cat):
...         if isinstance(package, Standard):
...             C()
...         else:
...             D()

Same idea, without the elsifs

>>> def groom2(pet, package):
...     dispatcher = {(Dog, Standard):A,
...                   (Dog, Premium):C,
...                   (Cat, Standard):B,
...                   (Cat, Premium):D}
...     key = (type(pet), type(package))
...     f = dispatcher[key]
...     f()

Why is this bad?

We have one function that is going to keep growing longer and longer and longer as we add more branches.

Using classes helps

We can make groom a method rather than a function. Dog.groom and Cat.groom still need to test the type of package.

>>> class Cat(Pet):
...
...     def groom(self, package):
...         if isinstance(package, Standard):
...             C()
...         else:
...             D()

Single Dispatching

When I call mittens.groom(x), Python runs Cat.groom(mittens, x).

mittens is effectively a parameter that python dispatches on.

Since mittens's type is the only parameter python looks at in order to find the right method to run, this is called "single dispatch".

Ancestral Wisdom

In 1987, common lisp introduced generic methods. Generic methods can dispatch based on the types and values of any of the parameter.

Common Lisp version of groom

(defgeneric groom (pet grooming-package))

(defmethod groom ((pet dog)
                  (grooming-package standard-groom))
    "cheapskate dog owner"
    (A))

But you came here for python instead...

PEAK-Rules

>>> from peak.rules import abstract, when, after
>>> @abstract
... def groom3(pet, package):
...     "Like defgeneric"
...
>>> @when(groom3, (Cat, Premium))
... def snootycat(x1, x2):
...     D()

In use

>>> groom3(Cat(), Premium())
wash
dry
hairball medicine
A ridiculous haircut

Why is this better than subclassing?

Now the vet wants to know if your dog is young or old. Young dogs get different treatments than old dogs.

Will you subclass by this dimension too?

The @after rule

Imagine that the premium package for dogs is really just the standard package plus something else:

>>> def B():
...     A()
...     print 'toothbrushing'

PEAK Rules can first run one rule, then run another, so this is not necessary.

The @after rule

First make a rule that applies to dogs getting the standard or premium package:

>>> @when(groom3, (Dog, Package))
... def everydog(x1, x2):
...     A()
...

Remember Package is the superclass of Standard and Premium. So this rule will match any grooming.

The @after rule

Now make a rule that runs AFTER the first rule.

>>> def B2():
...     "Like B, but doesn't call A first"
...     print 'toothbrushing'

>>> @after(groom3, (Dog, Premium))
... def fancydog(x1, x2):
...     B2()

In use

>>> groom3(Dog(), Standard())
wash
flea dip
dry

>>> groom3(Dog(), Premium())
wash
flea dip
dry
toothbrushing

More fancier

You can do more than test on types! Anything goes!

>>> @abstract
... def f(i):
...     "abstract!"
...

>>> @when(f, "i % 2")
... def f(i):
...     print "%d must be an odd number." % i
...

Disambiguation

PEAK-Rules isn't magic. If you give it a parameter that matches two "when" rules, it will fuss.

If you give it a parameter that matches NO rules, it will fuss.

Apocrypha

Stuff that I can talk about if time permits.

doctests are great

I can test all the code samples in the text file because of the doctest module.

Doctests in a function's docstring guarantee that the function works and provide examples of intended use.

You can store really long doctests in a separate file.

Multiple Decorators can be tricky

>>> @as_string
... @add1
... def g(a, b):
...     return a + b
...
>>> g(3, 4)
'8'

Now switch the order

>>> @add1
... @as_string
... def g(a, b):
...     return a + b
...
>>> g(3, 4)
Traceback (most recent call last):
    ...
TypeError: unsupported operand type(s) for +: 'int' and 'str'

Class decorators

Apply that debuglogger to every method in a class:

>>> from inspect import getmembers, ismethod
>>> from decoratortalk import debuglogger
>>> def debug_everything(C):
...     "danger -- this alters the C param."
...     for name, value in getmembers(C, ismethod):
...         setattr(C, name, debuglogger(value))
...     return C
...

Class decorators

In 2.5, you have to apply it like this:

>>> debug_everything(C) #2.5

In 2.6, you can do this:

>>> @debug_everything
... class C(object):
...     def f1(self, a):
...         return a + 1
...

Class decorators

Here's how it works:

>>> c = C()
>>> c.f1(99)
args is (<__main__.C object at 0x8366fec>, 99) and kwargs is {}.
results from calling f1 is 100.
100

Class decorators vs metaclasses

both allow clever postprocessing.
metaclasses do their magic on subclasses.
metaclasses may block the creation of a class or substitute something else in place of the original class.
both can make your code gorgeous or horrifying.