Advanced Python Constructs

A simple description of a couple of more advanced, not wildly used, Python constructs like list comprehension, dictionary comprehension, map, and filter.

All examples are using Python 3.5.

List Comprehension

This is a very nice way to create a list. Much shorter, more clear, unless you don’t know the syntax. This can be used instead of creating a list with a multilevel for loop.

Let’s get a list of 100 numbers from 1 to 100. For all smaller than 11 let’s get the value squared. So basically I need to get a list like:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

The simple way of implementing such a thing would be the following code:

res = []
for i in range(1,101):
    if i < 11:
        res.append(i*i)

However, using a list comprehension it can be:

[i*i for i in range(1, 101) if i < 11]

Let’s read it from the middle:

• range(1, 101) - the function range is a generator in Python 3 (in Python 2 it was a function returning a list, xrange was the generator). It means that we can iterate through it like it was a list, but in reality, the function returns an object which returns the list values one by one.
• for i in range() - is a loop getting the value i from the generator, one at a time, that’s why generators are better than creating huge lists
• if i < 11 - limits the set of values from the generator, if the generator returns a value for which the if part returns False, than it behaves like there would be continue in the loop
• i*i - is the final value that will be stored in the result list

So basically, there is a generator which returns one value at a time, which is stored in the i variable, then the if part is applied, if returns True, then the i*i is stored in the list, if returns False, then we take the next value from the generator.

Dictionary Comprehension

This is a similar construct to the above one, but it creates a dictionary. The syntax is similar, the only difference is: different brackets and the key:value part instead of a single value.

Let’s create a dictionary using the previous list comprehension. The key would be the number, the value would be the number squared:

{i:i*i for i in range(1, 101) if i < 11}

the returned value is:

{1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81, 10: 100}

As you can see there are just two differences:

• {...} - different brackets, for the list there were [...]
• key:value - for the dictionary we need to have a mapping, not a single value

Using Filter

In Python there is a couple of functions widely used in functional languages. One of them is the filter function. The basic syntax is:

filter(function, iterable)

This function takes two objects as arguments: a function and an object which we can iterate on. It returns a generator object which returns values from the iterable argument, for which the function returns True.

Let’s now use it for filtering the values 1..100 from the generator to get the first 10:

f = filter(lambda x: x < 11, range(1, 101))
list(f)
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

• lambda x: x < 11 - is an anonymous function taking one argument, it returns boolean value (True/False)
• range(1, 101) - is a generator returning values from 1 to 100
• list(f) - the simplest way of creating a list from a generator

The above code can be also written like this:

def check_smaller_than_11(x):
    return x < 11

f = filter(check_smaller_than_11, range(1, 100))
[x for x in f]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Using Map

The map function is similar to the filter one. The arguments are the same:

map(function, iterable)

This function returns a generator, which returns exactly the same number of values as were in the the iterable object. The returned collection values are function(value).

So if we have a list: [1, 2, 3] and a function fun, then map(fun, [1, 2, 3]) returns a generator, which returns the values: [fun(1), fun(2), fun(3)].

Let’s make the same list as in the list comprehension example. Instead of this:

[i*i for i in range(1, 101) if i < 11]

It can also be created with the map and the filter:

m = map(lambda x: x*x, filter(lambda x: x<11, range(1, 101)))
list(m)
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

So basically:

• filter(...) - returns a generator, which returns all values from range(...), which are smaller than 11
• map(...) - returns a generator, which returns all values from filter squared
• list(m) - converts the generator to a list, only for displaying the values

Final Notes

Which solution is better? I don’t know. Pyflakes is not happy about map/filter saying This could be changed to list comprehension.

One of the huge cons of using lambdas in the following examples is that lambda can be just a one liner in Python. You can always write a separate function and just use it instead of the lambda. Functions in Python are objects, so you can just pass them to another function.

On the other hand if you need to use the same condition in a couple of list comprehensions, it is always better to move them into one separate function, so the code won’t be duplicated.

This function then can be used in a list comprehension like:

def fun(x):
    return True if x < 1 else return False

[x for x in gen if fun(x)]

but you can also use the same function in filter:

filter(fun, gen)

As usually all the solutions have pros and cons, the real life choice usually depends on many different aspects.