01: Python Iterators, Generators & Decorators Tutorial

Assumes that Python3 is installed as described in Getting started with Python.

1. Iterators

Iterators don’t compute the value of each item when instantiated. They only compute it when you ask for it. This is known as lazy evaluation. Lazy evaluation is useful when you have a very large data set to compute.

Python Iterators have to implement two methods: 1) __next__(self) 2) __iter__(self)

Let’s create a test file with a Main() method.

Finally, run “test_counter.py“.

Output:

python3 REPL
dir(…) to print the object

You can see that the Counter object has the “__iter__” & “__next__” methods.

2. Generators

Generator functions allow you to declare a function that behaves like an iterator. They allow programmers to make an iterator in a fast, easy, and clean way. It is an easier way to create iterators with a keyword yield from a function. It works a bit like return because it returns a value.

Unlike iterators, generators save the state of the function. The next time the function is called, execution continues from where it left off, with the same variable values it had before yielding.

A generator is a more Pythonic way to create an iterator. Use iterator only for very large data sets.

Using a generator expression
Using a list comprehension
Without generator expression

Note: set(xs) removes duplicates from each list as in [‘a’,’b’,’a’,’c’], [‘a’,’b’,’c’,’d’], etc within a list, hence you get a,b,c from the first list and a,b,c,d from the second list within the list.

Q: When should you use generator expressions and when should you use list comprehensions in Python?
A: List comprehensions and generator expressions (short form: “listcomps” and “genexps”) are a concise notation for such operations, borrowed from the functional programming language Haskell. With a list comprehension, you get back a Python list. Generator expressions return an iterator that computes or yield the values as necessary, not needing to materialise all the values at once.

Use list comprehensions when the result needs to be iterated over multiple times, or where speed is paramount. Generator expressions are preferable in situations where the iterators that return an infinite stream or a very large amount of data.

Say you have a 1TB log file called “hugefile.log”, and you want the content and length for all the lines that start with the word “INFO”. If you use a list comprehension as below the underlying array could therefore contain up to 1TB of content, which is a lot of RAM, and not practical for this purpose.

If use a generator to apply a “filter” to your content, no data will be read until we start iterating over the result.

Q: Can you iterate conditionally?
A: Yes, by using if conditions.

3. Decorators

Before looking at Decorators, lets have a quick look at a Closure in Python. In Python, a Function can be assigned to a variable and passed around like objects.

Closure

A Closure is nothing but a function that is returned by another function. For example:

adder is a closure which adds a given number to a pre-defined one. For example, in “add5“, the function returned has “5” closed over. The “num1 = 5”, and then when you call add(6), the “num2 = 6”.

Now, back to a decorator:

Decorator is a way to dynamically add some new behaviour to some objects. You can use a Closure to decorate a given object. Decorators define reusable building blocks you can apply to a callable to modify its behaviour without permanently modifying the callable itself. In the example below “add” is the callable function, and “logging” is reusable building block.

@logging is a syntactic sugar to do something as shown below:


300+ Java & Big Data Interview FAQs

16+ Java Key Areas Interview Q&As

800+ Java Interview Q&As

300+ Java & Big Data Tutorials