"Python Booster Pack" is probably the most clickbait-y title you've read for a post that documents a few tips and tricks in Python. Believe me, the content isn't as disappointing as the title. kek. Here I want to explain some of the lesser-known features that Python and some of its modules have to offer, the kind that can change how you code and debug, or bump up your productivity a bit. Not claiming much though. Some of these are hidden in plain sight and call for a proper Eureka! moment.
I didn't discover or develop any of these. Most of them showed up on my Twitter feed at 2AM, my Reddit feed at 3AM, or a friend's chat at 4AM. Let's begin.
Introducing lru_cache()
from functools import lru_cache
@lru_cache(maxsize=None)
def fibonacci_with_lru(n):
return n if n in [0, 1] else (fibonacci_with_lru(n - 1) + fibonacci_with_lru(n - 2))
def fibonacci(n):
return n if n in [0, 1] else (fibonacci(n - 1) + fibonacci(n - 2))Go on. Give the code above a whirl in your Jupyter notebook.
The lru_cache decorator in functools caches, or auto-memoizes, repeated calls using a least-recently-used eviction policy. The maxsize argument decides the max cache size (usually a power of 2). It also has a typed argument which, if set to True, treats 3 and 3.0 as two separate numbers.
In the snippet above, you'll notice a speedup of almost 10-3 even though the input increased by 10x. This isn't always advisable, and it's no replacement for a faster non-recursive approach, but it's a neat trick during prototyping and has its own use cases, like caching file fetches.
binding.pry in Python? Hmmm.
I've seen countless Ruby developers gush about binding.pry. It's a debugging routine that lets you step through code line by line and inspect variables at each stage. Turns out it exists in Python too (pip install ipdb)!
Consider a binary search routine we want to examine from the inside.
import ipdb; debug = ipdb.set_trace
def binary_search(needle, haystack):
left, right = 0, len(haystack) - 1
while left <= right:
mid = left + (right - left) // 2
debug();
if haystack[mid] == needle:
return True
elif haystack[mid] > needle:
right = mid - 1
else:
left = mid + 1
return False
binary_search(2, [1, 2, 3, 4])
binary_search(5, [1, 2, 3, 4])Notice how we've sneakily placed the debug() call right after the algorithm decides the midpoint.
The program pauses at debug() and drops you into an interactive session. n steps to the next line, c continues execution. There are other debuggers you can try, but ipdb gives you a proper IPython shell with syntax highlighting and tab-completion, which makes it my favorite.
Introducing the LineProfiler
Ever wondered why your routine takes so long despite writing what feels like efficient code? Spending hours figuring out which part is eating up all the time is injurious to the hair on your head.
Anyway, if you're on Python 2.7, start with a
pip install line_profilerPython 3 users need to follow,
pip3 install Cython
git clone https://github.com/rkern/line_profiler
cd line_profiler; sudo python3 setup.py installLet's take an arbitrary mathematical function as an example. We import the line_profiler and atexit modules and place the @profile decorator over the function we want to profile.
import numpy as np
import line_profiler
from atexit import register
profile = line_profiler.LineProfiler()
register(profile.print_stats)
@profile
def math_function(n):
x = np.random.normal(size=n)
y = np.power(x, 4)
z = np.sqrt(y)
return np.sum(z)
math_function(7123681)Run your script and voilà: you get a line-by-line breakdown of hits per line (more than 1 for loops), total time, time per hit, and percentage of the function's total time.
Images == Numbers
An image is nothing but a collection of numbers, so we can use a handy lambda to turn a matrix of numbers into an image. It's fast, lightweight, and gets the job done, which makes it useful for prototyping or poking at a dataset.
import numpy as np
from PIL import Image
get_image = lambda x: Image.fromarray(np.uint8(255 * (x - x.min()) / x.ptp()))
get_image(np.random.rand(200, 200))In the snippet above, we pass a random 200 x 200 matrix to get_image and get an noise image.
Better Python REPL
Most Pythonistas may already know this. For those who don't, run
pip install ptpythonPtpython is a replacement for the traditional Python REPL. It has autocompletion, syntax highlighting, mouse support, multi-line editing, and auto-tabbing. What a package!
You could also level up your setup by aliasing it to your specific Python distribution.
Cython is <3
We often run into scripts doing something simple but crunching a lot of data, where getting any real speedup is a pain. Let's see how far we can push this, starting with a plain Python program.
n = 100000000
arr = [0.0] * n
for i in range(n):
arr[i] = i % 3
print(arr[:10])Time taken:
13.26s user 0.31s system 99% cpu 13.636 totalUsing modifications inspired by Cython, let's convert this program to something like the one below (also change the file extension to .pyx),
from array import array
cdef int n = 100000000
cdef object arr = array('d', [0.0]) * n
cdef double[:] mv = arr
cdef int i
for i in range(n):
mv[i] = i % 3
print(arr[:10])Execute it using,
pip install Cython
cythonize -b -i filename.pyx
python -c "import <filename>"Time taken:
0.53s user 0.31s system 86% cpu 0.866 totalHoly smokes! That's almost 25 times faster than our non-Cython code.
This happens because Cython creates a native binary that's linked at runtime. I've heard of a few use cases where it gives a speedup of almost 100x. If a few type declarations and giving up some of Python's expressiveness doesn't bother you much, this is a great option for compute-heavy scripts.
Named Tuple
This is one of the lesser-known data structures in the collections module. Here's how it works.
from collections import namedtuple
Candidate = namedtuple('Candidate', 'name age gender')
candidate_1 = Candidate(name='John Doe', age=35, gender='M')
# Output
# >>> candidate_1
# Candidate(name='John Doe', age=35, gender='M')
# >>> candidate_1.name
# 'John Doe'
# >>> candidate_1.age
# 35
# >>> candidate_1.gender
# 'M'namedtuple is a great option if you want struct-like functionality in Python without reaching for a full class just to hold a few fields.
Avoid data structure initializing functions
Have a look at the snippet below and think about what it's telling you.
>>> from timeit import timeit
>>> timeit("x = dict()", number=10000000)
1.3071075379999968
>>> timeit("x = {}", number=10000000)
0.37984672199999636
>>> timeit("x = list()", number=10000000)
1.0812364029999912
>>> timeit("x = []", number=10000000)
0.285002046999999Here we timed the data-structure initializing functions, dict and list, against their symbolic initializers (idk what to call them), running each 10,000,000 times. Using the symbolic initializers gives a 4x to 5x speedup. Worth pointing out next time you're reviewing or writing code.
Better function aliases
Here we look at functools.partial, which lets you create a new version of a function with one or more arguments already filled in. To convert a number in a different base to decimal, we use int with the base argument. Writing that out as separate functions looks like this.
def from_binary(x):
return int(x, base=2)
def from_octal(x):
return int(x, base=8)
def from_hex(x):
return int(x, base=16)
print(from_binary('10010')) # prints 18
print(from_octal('34472')) # prints 14650
print(from_hex('7e8c9')) # prints 518345Using partial instead gets us the same result, more tersely.
from functools import partial
from_binary = partial(int, base=2)
from_octal = partial(int, base=8)
from_hex = partial(int, base=16)
print(from_binary('10010')) # prints 18
print(from_octal('34472')) # prints 14650
print(from_hex('7e8c9')) # prints 518345This example is trivial, but partial earns its keep in real API design.
Heard of for..else?
The snippet below checks whether an odd number exists in a list, using a boolean flag to do it. Can you find a way to do this without a flag, and without making the code more complex? Think about it.
flag = False
for i in [2, 3, 4, 5, 6]:
if i % 2 != 0:
flag = True
break
if flag:
print("An odd number exists in the array.")
else:
print("An odd number does not exist in the array.")Whether or not you came up with something, an elegant solution is Python's for..else syntax. The else block runs only if the loop finishes without hitting a break.
for i in [2, 2, 4, 4, 6]:
if i % 2 != 0:
print("An odd number exists in the array.")
break
else:
print("An odd number does not exist in the array.")We've elegantly avoided the flag, and the code reads a lot cleaner. Another one for the code review pile.
Manage your contexts
There will be situations in your coding life where you need something to run before and after your actual logic.
Think closing a file after you're done reading it, releasing a lock, or having a thread release resources once a task finishes. Plenty of use cases like this exist.
Python's with statement is built for exactly this.
with open('filename.txt', 'r') as f:
...That snippet automatically closes the file once the with block finishes. This pattern is called a context manager, and Python gives you a few ways to build one.
The first is a plain class with __enter__ and __exit__ dunder methods. Run this to see the flow.
class Demo:
def __init__(self):
print("__init__")
def __enter__(self):
print("__enter__")
return "whatever __enter__ returns"
def __exit__(self, *args):
print("__exit__")
with Demo() as x:
print(x)
print("hello")
## Output
__init__
__enter__
whatever __enter__ returns
hello
__exit__x ends up being whatever __enter__ returns, and the order of execution is __init__, __enter__, the code inside the with block, then __exit__. Anything you want to run before your logic goes in __enter__; anything after goes in __exit__.
The second method is useful when you don't want the overhead of a class, or have very little to do before and after your logic. Just use a function instead: add the @contextmanager decorator, and anything before yield behaves like __enter__, anything after behaves like __exit__.
from contextlib import contextmanager
@contextmanager
def Demo():
print("__enter__")
yield "hello"
print("__exit__")
with Demo() as x:
print(x)
## Output
__enter__
hello
__exit__Notice how x here stores the value returned by yield.
The third way generalizes this further. If you're repeating the same before-and-after logic a lot, use a ContextDecorator instead of copy-pasting it. Below, we write a class like the one in the first method, but have it inherit from ContextDecorator.
from contextlib import ContextDecorator
class Demo(ContextDecorator):
def __enter__(self):
print("__enter__")
def __exit__(self, *args):
print("__exit__")
@Demo()
def DemoFunction():
print("hello")
DemoFunction()Using this class as a decorator on any function lets us reuse the same before-and-after logic everywhere. Write it once, decorate as many functions as you want. Notice the with keyword doesn't even show up in this method.
Generator expressions
IPython's %timeit magic is great for figuring out what's actually fast. Below, we find all the even numbers up to 1000 and square them, two ways: a generator expression and a plain list comprehension. Then we iterate over each and print the values.
>> %timeit it = (x**2 for x in range(1000) if x % 2 == 0)
642 ns ± 18.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
>> %timeit non_it = [x**2 for x in range(1000) if x % 2 == 0]
228 µs ± 2.95 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
>> %timeit for el in it: print(el) #ignoring the output
44.9 ns ± 0.769 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>> %timeit for el in non_it: print(el) #ignoring the output
3.77 ms ± 167 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)Generator expressions come out roughly 103 times faster to create and 106 times faster to iterate over. Generators are a genuinely useful concept, and swapping your list comprehensions for them can be a real speed win. Most of the time, you can just swap [] for () wherever a list comprehension is generating a series. ¯\_(ツ)_/¯
Invoke class routines using strings
Here's a fun one. Say you have a class like the one below, with 4 private members (using the classic double-underscore trick) and getter functions for each. Since the members are private, you can't access them directly with object.__member, but the getters work fine: object.getter_for_member().
class Demo():
def __init__(self):
self.__name = 'John Doe'
self.__age = 18
self.__gender = 'Male'
self.__location = 'EU'
def name(self):
return self.__name
def age(self):
return self.__age
def gender(self):
return self.__gender
def location(self):
return self.__location
d = Demo()
print(d.__name) # fails
print(d.name()) # returns the nameBut what if you're handed the attribute names as strings instead? That's what getattr is for.
getattr takes the object as its first argument and the member name, as a string, as its second.
Notice how this calls every getter function, using nothing but strings.
attrs = ['name', 'age', 'gender', 'location']
d = Demo()
for attr in attrs:
print(getattr(d, attr)())It's a handy way to keep a list of attribute names as plain strings and not worry about how they get invoked.
That's the grab bag for now. I'm always on the hunt for stuff like this. Until next time, adios!