Having programmed for a significant portion of my career in Python, I too share your love for this language. It is an incredibly popular programming language and there are often multiple ways to accomplish the same task. However, not all of them are equally efficient. For example, using certain data structures or algorithms may have a significant impact on the speed and memory usage of your program. It's crucial to understand which techniques can optimize your code's performance. In this article, based on my practical experiences, we will examine several of these techniques with comparative examples, illustrating how they can help boost the speed and efficiency of your Python code.

List Comprehension

List comprehensions are a concise and fast way to create lists in Python. They are a powerful tool for data processing and can often replace longer more complicated loops. Here's an example:

# using for loop
my_list = []
for i in range(1000):
    if i % 2:
        my_list.append(i ** 2)
    else:
        my_list.append(i ** 0)
print(my_list)

# using list comprehension
my_list = [i ** 2 if i % 2 else i**0 for i in range(1000)]
print(my_list)

In the above example, list comprehension is faster as it does not require the use of a append() method, which results in better performance. Additionally, list comprehension can make code more readable and concise.

The nested list comprehension is also supported but it can be avoided if it compromises the readability.

Nested list comprehension example:

matrix = [[i*j for j in range(3)] for i in range(3)]
print(matrix)

Note: There may be situations where list comprehension and traditional for loops offer similar performance. Nonetheless, it is a good idea to use them wherever we can.

Bonus:

The append() the method in Python has an amortized O(1) complexity, which means that the time it takes to append an element to a list is constant on average. It can be called o(1)+ but overall it's a constant time. You can read more on this complexity in the linked article.

Generators

Generators are a type of iterable in Python that allow for the lazy evaluation of data. Unlike lists or other sequences, which are stored entirely in memory, generators produce data on the fly, only as needed. This makes them a powerful tool for working with large datasets, or with datasets sequences that are otherwise difficult to store in memory.

Example of how generators can be used in Python:

def generate_numbers(start, end):
    current = start
    while current <= end:
        yield current
        current += 1

# create a generator object that generates numbers from 1 to 10
numbers = generate_numbers(1, 10)

# iterate over the numbers using a for loop
for num in numbers:
    print(num)

In this example, the generate_numbers the function is a generator that produces a sequence of numbers starting from start and ending at end. The yield statement is used to return each number in the sequence as it is generated, and the generator function continues from where it left off each time it is called.

As it yields only the required data for each iteration you do not need to load all the data into memory at once and then process it.

The strong use case for this is whenever you have to generate a huge sequence of data, you can use generators.

Another useful application of this could be batch processing .

def batch_generator(items, batch_size):
    for i in range(0, len(items), batch_size):
        yield items[i:i+batch_size]

# Example usage
my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
batch_size = 5

for batch in batch_generator(my_list, batch_size):
    # Process the current batch
    print("Processing batch:", batch)

Bonus :

The difference between a Python list and an iterator

• list contains a number of objects in a specific order - so you can, for instance, pull one of them out from somewhere in the middle

• Whereas, an iterator yields a number of objects in a specific order, often creating them on the fly as requested as we discussed in the above examples

Numpy for Math Operations

If you are working with mathematical operations, you should consider using the Numpy library. Numpy is a Python library that provides support for large, multi-dimensional arrays and matrices, as well as a large collection of high-level mathematical functions to operate on these arrays. Numpy is optimized for fast array operations. The reason for this being faster

• Numpy arrays are homogenous compare to python lists which can support nonhomogenous data type is a single list - [1, "string", 2.01]

• Numpy operations are implemented in C lang

Use Set for Membership Test

If you need to test whether an item is in a collection, using a set can be much faster than using a list or a tuple. Sets are implemented as hash tables in Python. Thus they have a constant time complexity for checking an item in a collection

membership testing much faster than in a list or a tuple. Here's an example:

# using a list
my_list = [1, 2, 3, 4, 5]
if 3 in my_list:
    print("found")

# using a set
my_set = {1, 2, 3, 4, 5}
if 3 in my_set:
    print("found")

In this example, the set approach is much faster as its o(1) compare to o(n) of the first case.

Use the "Join" method for String Concatenation

# using +
my_string = ''
for i in ['str1', 'str2', 'str3']:
    my_string += i
print(my_string)

# using join()
my_list = ['str1', 'str2', 'str3']
my_string = ''.join(my_list)
print(my_string)

Each time the "+" operator is called a new buffer is allocated to add the concatenated string and in the next "+" call again previous result string plus the new string is allocated a new buffer.

In other words, if you have N strings to be joined, you need to allocate approximately N temporary strings, and the first substring gets copied ~N times. The last substring only gets copied once, but on average, each substring gets copied ~N/2 times.

While in join python can figure out how much buffer is needed to be allocated and it can create in one time.

Use builtin function as much as possible

Python has a number of built-in functions that are optimized for speed and can save you a lot of time compared to writing your own functions from scratch.

Understanding and using the right data structure

Python provides a variety of built-in data structures, including lists, tuples, sets, and dictionaries. While lists and dictionaries are the most commonly used, it's important to consider the specific needs of your program when choosing a data structure.

One example of this is the use of tuples. Tuples are more performant than lists in situations where you have an immutable data sequence. This is because tuples are implemented as a contiguous block of memory, which allows for faster access times than a list.

Do not Prefer "dot" Operation

Try to avoid dot operation. See the below program.

import math
val = math.sqrt(60)

Instead of the above style write code like this:

from math import sqrt
val = sqrt(60)

Because when you call a function using . (dot) it first calls __getattribute()__ or __getattr()__ which then uses dictionary operation which costs time. So, try using from module import function. If dot notation is used inside the loop it can create a significant difference.

To conclude, by using these techniques developers can significantly improve the speed and efficiency of their Python programs.

I hope this would help you in your day-to-day programming and thank you for reading this article. As Python is my primary programming language, I will be sharing more of my learnings on it. If you would like to read more, subscribe to my blog.