In this article, we'll take a tour of the fundamental data structures and implementations built into Python and its standard library.

As a Python developer, understanding and utilizing various data structures is a fundamental part of writing efficient and effective code. Each data structure provides a particular way of organizing data so it can be accessed efficiently, and learning when to use each one can greatly improve the performance and readability of your code. This information will also help you shine in Python coding interviews.

Table of Contents

• Dictionaries
• Lists
• Tuples
• Sets and Multisets
• Stacks
• Queues
• Priority Queues

Dictionaries

In Python, a dictionary (often referred to as a dict) is a data structure that stores a collection of items, where each item is a key-value pair. It is an unordered collection, which means that the items are not stored in a particular order, and the elements are accessed using keys rather than indexes.

dict - General-purpose Dictionaries

Dictionaries are defined using curly braces {}, with keys and values separated by colons : – here's an example of how to work with dictionary:

# Creating a dictionary
my_dict = {'name': 'John Doe', 'age': 30, 'gender': 'male'}
# Or with the dict constructor
my_dict = dict(name='John Doe', age=30, gender='male')

# Accessing elements of dictionary
print(my_dict['name']) # 'John Doe'

# Return default if key is not found
print(my_dict.get('age')) # 30
print(my_dict.get('job', 'Not Found')) # 'Not Found'

# Get the dictionary members
print(my_dict.keys()) # ['name', 'age', 'gender', 'job']
print(my_dict.values()) # ['Jane Doe', 30, 'male', 'software engineer']
print(my_dict.items()) # [('name', 'Jane Doe'), ('age', 30), ('gender', 'male'),

Note: while standard dict instances preserve the insertion order of keys in CPython 3.6 and above; this is just a side effect of the CPython implementation and is not defined in the language spec.

OrderedDict - Preserve Insertion Order of Keys

An OrderedDict is a subclass of the built-in dict class in the collections module of python, that remembers the order in which items were added. It is similar to a regular dictionary, but it maintains the order of the items, so that when you iterate over the items, they are returned in the order they were added. Here's an example of how to use an OrderedDict:

from collections import OrderedDict
d = OrderedDict()
d['task1'] = 'complete'
d['task2'] = 'pending'
d['task3'] = 'in-progress'
# Returns the items in order of insertion
print(list(d.items()))

OrderedDict is useful when you want to maintain the order of elements in the dictionary and you need to access the elements in the order they were added. This can be useful in situations where the order of elements is important for the functionality of your code. For example, if you want to keep track of the order in which items were added to a shopping cart, you would use an OrderedDict.

defaultdict – Return Default Values for Missing Keys

A defaultdict works just like a regular dictionary, but it has a default value for items that do not exist yet. This can make your code simpler because you do not have to check whether a key is in the dictionary before adding or retrieving a value. They are especially useful when working with dictionaries where the keys are unknown beforehand or when you want to do some operation on keys that doesn't exist yet.

from collections import defaultdict

dd = defaultdict(list)
dd['fruits'].append('apple')
dd['fruits'].append('banana')
dd['vegetables'].append('potato')

print(dd)
# {'fruits': ['apple', 'banana'], 'vegetables': ['potato']}

ChainMap – Join Multiple Dictionaries

In Python, the collections module provides the ChainMap class, which is a type of data structure that allows you to group multiple dictionaries together into a single, logical view. This can be useful in certain situations where you want to access multiple dictionaries as if they were one dictionary while still keeping them separate.

One common use case for ChainMap is to handle situations where you want to provide defaults for a configuration or settings but also allow users to override them with their own settings.

from collections import ChainMap
dict1 = {'one': 1, 'two': 2}  
dict2 = {'three': 3, 'four': 4}
chain = ChainMap(dict1, dict2)
# ChainMap({'one': 1, 'two': 2}, {'three': 3, 'four': 4})

Lists

A list is an ordered, mutable collection of items that can be of any type. Lists are defined by enclosing the elements in square brackets [], separated by commas. Here's an example of how to create and use a list in Python:

# Creating a list
my_list = [1, "hello", 3.14, True]

# Accessing elements of list
print(my_list[0]) # 1
print(my_list[1]) # 'hello'
print(my_list[-1]) # True

# iterating over the list
for item in my_list:
    print(item)

# Modifying the list
my_list[0] = "world"
# ["world", 'hello', 3.14, True]

my_list.append(4)
# ["world", 'hello', 3.14, True, 4]

my_list.remove("world")
# [hello', 3.14, True, 4]

Lists are mutable, which means that the elements of a list can be modified after it is created. This is the main difference between lists and tuples.

Tuple

In Python, a tuple is an immutable, ordered collection of items that can be of any type. Tuples are defined by enclosing the elements in parentheses (), separated by commas. For example:

# Creating a tuple
t = (1, "hello", 3.14, True)

# Accessing elements of tuple
print(t[0])  # 1
print(t[1])  # 'hello'
print(t[-1]) # True

# iterating over tuple
for item in t:
    print(item)

Tuples are useful when you want to group a set of related items together, like coordinates or a name and an age, or when you want to ensure that the data remains constant throughout the program execution. You can also use them as keys in dictionaries and as elements of sets because they are immutable.

NamedTuple

The NamedTuple is a subclass of a tuple that allows accessing the elements of the tuple using human-readable names rather than index numbers:

class Car(NamedTuple):
    make: str
    color: str
    mileage: float

car1 = Car('Jeep', 'blue', 9047.3)
# Car(make='Jeep', color='blue', mileage=9047.3)

# Accessing fields:
car1.mileage # 9047.3

# Fields are immutable:
car1.mileage = 12 # AttributeError: "can't set attribute"

NamedTuples are useful when working with tuples that have a clear meaning and intent, as they make it easier to understand the code and reduce the likelihood of bugs caused by using the wrong index numbers.

NamedTuples make the code more readable and understandable, and they reduce the need for comments or variable names to explain the purpose of the tuple.

Sets and Multisets

A set is an unordered collection of objects that does not allow duplicate elements. Typically, sets are used to quickly test a value for membership in the set, to insert or delete new values from a set, and to compute the union or intersection of two sets.

vowels = {'a', 'e', 'i', 'o', 'u'}
print('i' in vowels) # True
letters = set('alice')
letters.intersection(vowels) # {'a', 'e', 'i'}

frozenset – Immutable Sets

The frozenset class implements an immutable version of a set that cannot be changed after it has been constructed (no inserts or deletions.) Because frozen sets are static and hashable, they can be used as dictionary keys or as elements of another set, something that isn’t possible with regular (mutable) set objects.

vowels = frozenset({'a', 'e', 'i', 'o', 'u'})
vowels.add('p')  
# AttributeError:  
# "'frozenset' object has no attribute 'add'"

Counter – Multisets

The collections.Counter class in the Python standard library implements a multiset (or bag) type that allows elements in the set to have more than one occurrence.

This is useful if we need to keep track of how many times an element is included in the set:

from collections import Counter

inventory = Counter()

loot = {'gold': 1, 'bread': 3}
inventory.update(loot) 
# Counter({'bread': 3, 'gold': 1})

len(inventory) # 2
# Note: only unique elements, not total

Stacks

A useful real-world analogy for a stack data structure is a stack of plates – similar to a stack of plates, adding or removing is only possible at the top. To access plates lower in the stack, each plate must be removed one by one (Last In, First Out). Whenever a new plate is added, it is placed on top of the stack.

Both stacks and queues are linear collections of items. However, the way in which the items are accessed differs. In a queue, the item that was added first is the first to be removed (First In, First Out, or FIFO). Whereas in a stack, the item which was most recently added is the one that is removed first (Last In, First Out, or LIFO).

Stack using a list

The built-in list type can be used as a stack, but be careful to only append and remove items with append() and pop() in order to avoid slow performance from having to move all the items in the list.

# Create an empty stack
stack = []

# Push elements onto the stack
stack.append(1)
stack.append(2)
stack.append(3)

# Print the stack
print(stack) # [1, 2, 3]

# Pop an element from the stack
x = stack.pop()
print(x) # 3

# Print the remaining stack
print(stack) # [1, 2]

deque – General Stacks

Even though we may be able to use a list as a stack, it may not be obvious to someone else reading our code. It's best to be explicit about the intent of the code.

We should use the collections.deque class instead for a safe and fast general-purpose stack implementation. Deques are actually double-ended queues, which can maintain a stack from the left or right side:

from collections import deque

# Create a new deque
d = deque()

# Add elements to the deque
d.append(1)
d.appendleft(2)
d.extend([3, 4, 5])

# Remove elements from the deque
x = d.pop()
y = d.popleft()

print(x) # 5
print(y) # 2
print(d) # deque([1, 3, 4])

# Remove all element 
d.clear()
print(d) # deque([])

LifoQueue – Locking Semantics for Parallel Computing

The LifoQueue (Last In, First Out) implementation in the Python standard library is synchronized and provides locking behavior to support multiple concurrent producers and consumers.

from multiprocessing import Process, LifoQueue

def worker(q):
    # Get an item from the queue
    item = q.get()
    # Do something with the item
    print(f'Worker: {item}')
    # Signal that the task is done
    q.task_done()

# Create a LIFO queue
q = LifoQueue()

# Create a worker process
p = Process(target=worker, args=(q,))

# Start the process
p.start()

# Put some items on the queue
for i in range(5):
    q.put(i)

# Wait for all the tasks to be completed
q.join()

# Exit the process
p.join()

In this example, a LifoQueue is created, and a single worker process is started that runs the worker function. The worker function is passed the LifoQueue as an argument, so the process can access it. The main process then puts five items on the queue and waits for them to be processed by the worker process using q.join(). The worker function takes an item from the queue, does something with it (e.g. print it out), signals that the task is done, and exits. The main process will wait on p.join() until the worker process finishes its task. The output will be as follows:

Worker: 4
Worker: 3
Worker: 2
Worker: 1
Worker: 0

Queues

Queues are similar to stacks, and the difference between them lies in how items are removed:

With a stack, you remove the item most recently added (Last In, First Out, or LIFO). With a queue, you remove the item least recently added (First In, First Out, or FIFO).

collections.deque – Double Ended Queues

The deque class implements a double-ended queue that supports adding and removing elements from either end in O(1) time. Because deques support adding and removing elements from either end equally well, they can serve both as queues and as stacks.

queue.Queue – Locking Semantics for Parallel Computing

This queue implementation in the Python standard library is synchronized and provides locking semantics to support multiple concurrent producers and consumers.

import threading
import queue

def worker(q):
    while True:
    	# Get an item from the queue
        item = q.get()
        if item is None:
            break
        # Process the item
        print(item)
        # Signal that task is done
        q.task_done()

q = queue.Queue()

# Put some items to the queue
for item in range(5):
    q.put(item)

# Create worker threads
t1 = threading.Thread(target=worker, args=(q,))
t2 = threading.Thread(target=worker, args=(q,))

# Wait for all the tasks to be completed
q.join()

t1.join()
t2.join()

Priority Queues

Ideally, high-priority tasks on the system (e.g., playing a real-time game) should take precedence over lower-priority tasks (e.g., downloading updates in the background). By organizing pending tasks in a priority queue that uses the task urgency as the key, the task scheduler can quickly select the highest-priority tasks and allow them to run first.

heapq

Python provides a built-in module that implements a binary heap based on a list, which maintains the minimum element from the list in the first spot. Removing repeatedly from a min heap produces results in sorted order:

import heapq

items = []
heapq.heappush(items, (2, 'code'))
heapq.heappush(items, (1, 'eat'))
heapq.heappush(items, (3, 'sleep'))

while q:  
    next_item = heapq.heappop(items)
    print(next_item)

# Result:
#   (1, 'eat')
#   (2, 'code')
#   (3, 'sleep')

PriorityQueue

The PriorityQueue class is a thread-safe implementation of a heap that uses locks behind the scenes. This is useful when multiple threads need access to a priority queue simultaneously.

from queue import PriorityQueue

q = PriorityQueue()
q.put((2, 'code'))
q.put((1, 'eat'))
q.put((3, 'sleep'))

while not q.empty():
    next_item = q.get()
    print(next_item)

# Result:
#   (1, 'eat')
#   (2, 'code')
#   (3, 'sleep')

References

Python Data Structures Docs

Leetcode Data Structures Crash Course

Common Python Data Structures Guide

Grokking Algorithms - An Illustrated Guide

***

Did you find this helpful?
I’d love to hear about it. Please let me know in the comments.

Do you have any questions?
Leave your question in a comment below, and we'll answer it with our best advice.

This little-known feature in Python could change the entire way you think about programming in a few sentences and help you come up with better programs.

We can thank Tim Peters for putting this easter egg into the language. I’ve benefited from revisiting it over the years, and I think Tim’s words have made me a better programmer.

>>> import this

The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren’t special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one—and preferably only one—obvious way to do it.
Although that way may not be obvious at first unless you’re Dutch.
Now is better than never.
Although never is often better than right now.
If the implementation is hard to explain, it’s a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea—let’s do more of those!

***

This applies beyond Python to most other programming languages, frameworks, technologies, and even systems. Hopefully, you will be able to apply them and admire their wisdom.

Python is the most loved programming language by both beginners and experts, according to multiple surveys. And not only software developers but other people from different disciplines use Python for a variety of different tasks, such as data analysis and visualization, artificial intelligence and machine learning, automation, etc.

There has never been a better time to learn Python, and with our resources created by professional Python developers with decades of experience, you can learn the skills and techniques that really matter in the real world. You can use Python to build web apps, mobile apps, and desktop applications, as well as software testing and even some actual hacking.

Whether your goal is to learn python for Web Development, Data Science, or Machine Learning, we'll go over the most important libraries and resources you need to learn after getting the basics down.

Table of Contents

• Python Basics
• Intermediate Python
• Advanced Python Topics
• Python for Web Development
• Python for Automation
• Python for DevOps
• Python for Data Science
• Python for Machine Learning
• Tips for Learning Programming with Python

Python Basics

First, we need to install python so we can play around with it:

Setting Up Python and Virtualenvs with Pyenv

Then, you can follow this roadmap to go from Python beginner to knowing "just enough to be dangerous" with Python. While this list doesn't cover every single feature, it does cover many of Python’s most noteworthy features and will enable you to read and write Python programs.

1. A Gentle Introduction to Python Programming
2. Variables and Operators
3. Strings and String Methods
4. Basic Data Types in Python
5. Pythonic Loops
6. Exception handling
7. Functions
8. Importing libraries

Intermediate Python

This section will give you the tools to make complex applications and introduce you to concepts about Object Oriented Programming (OOP) in Python:

1. String Formatting
2. List Operations
3. Dictionaries
4. Classes and Objects
5. File System Operations
6. Common Data Structures in Python

Advanced Python Topics

This section builds upon the previous ones and teaches you how to use frameworks and libraries available in the Python language:

1. Asynchronous Programming with asyncio
2. Multithreading
3. Multiprocessing

Once you have gone through all the previous steps, you should have everything you need to make use of Python in real-world applications.

Python is used in almost every industry, and there are many possible career paths you can choose from. Let's look at some of them, shall we?

Python for Web Development

The role of a web developer is to build websites and applications that are accessible using a web browser. Python is typically used on the backend (server) side.

Start by learning more about the web and how your users will interact with your apps on their browsers. This course will help you become more of a well-rounded full-stack web developer:

Web Design Course for Everybody

A popular web framework that many companies use to build web applications is Flask. This tutorial will help you become familiar with building apps with Flask:

Python REST APIs with Flask

You should also become familiar with databases. This tutorial covers how to connect a database to your Flask server:

Connecting a Database to your Flask Project with SQLAlchemy

You can also learn how to deploy Flask apps so you can share them with the world in this article:

How to deploy your Flask apps

Python for Automation

Automation is the field of performing tasks that are repetitive or time-consuming, such as data entry, web scraping, and automation of various system processes, in a way that they can be reused and repeated effortlessly.

We do a lot of automation for running integration tests in software projects, in which we write scripts to interact with an existing program and report errors.

Getting started with testing in Python

Automation can also be used to control other software, such as a web browser, Excel, or even hardware, such as robotic systems. Some libraries worth checking out for this are:

• Selenium
• Beautiful Soup

Python for DevOps

DevOps engineers are responsible for implementing and maintaining infrastructure to support software systems, and they work to ensure that software systems are deployed quickly, efficiently, and with minimal downtime.

This book provides a comprehensive introduction to the field of DevOps:

Python for DevOps: Learn Ruthlessly Effective Automation

Some of the other technologies used by DevOps engineers to automate software deployments and maintain a smooth operation are:

• Docker
• Kubernetes

Python for Data Science

A data scientist's primary concern is to extract insights and knowledge from data. This typically involves a number of different tasks, including collecting and cleaning data, developing predictions and forecasts using statistics, and visualizing results.

A data scientist must be familiar with databases and should be able to query large amounts of data in order to identify patterns and trends. Usually, the main tools used by data scientists are:

1. Free Interactive SQL Course
2. Pandas
3. Data Visualization

If you want to get deeper into data science, consider the following books:

Data Science from Scratch

Python for Data Analysis: Data Wrangling with Pandas, NumPy, IPython

Python for Machine Learning

A machine learning engineer creates and develops computer programs that can "learn" from data and make predictions or decisions without being explicitly programmed to do so to solve real-world problems like predicting the weather or detecting facial expressions.

This typically involves working with large sets of data, which can be explored with the pandas library to prepare the data for use in models:

Getting started with Pandas in Python

A very popular library that helps with training machine learning models for many common use cases is Scikit-learn.

If you are interested in models that use language, check out this article about NLP and Python:

Natural Language Processing (NLP) with Python

These books are an excellent way to get into machine learning with Python, with many examples and some actual real-world projects:

Introduction to Machine Learning with Python

Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow

Tips for Learning Programming with Python

Learning how to write computer programs is hard, but learning Python, in particular, is significantly easier than many other programming languages. These tips will help you stay focused and motivated to achieve your goals:

• Have patience. Learning a new language always requires more effort and time, so understand that it will take time to let everything sink in.
• Don't feel intimidated by complex terms, errors, and issues. Don't give up. This happens to everyone in programming. Research the topic using other resources and come back to it with a clear mind.
• Frustration and pain are a part of the process. Embrace them instead of avoiding them, and you will conquer them.
• Be consistent. If you are not consistent in learning, it will take a lot more time and effort.
• Don't try to learn everything at once. It's better to practice a specific topic every day before moving on to the next.
• Build small projects to try out the things you learn and build up your confidence. You should add them to your github profile to help you land jobs.

***

Did you find this helpful?
I’d love to hear about it. Please let me know in the comments.

Do you have any questions?
Leave your question in a comment below, and we'll answer it with our best advice.

In this guide, we'll introduce asynchronous programming in Python and review fundamental concepts like how to define, create and run asynchronous functions, coroutines, and some common use cases and best practices.

You may have seen asyncio in a new project you started working on. Or perhaps in a code example of how to perform a common task:

• Perhaps you need to use an API and the code examples use asyncio.
• Or you need to integrate a new library that uses asyncio.
• Or you found some code snippets that use asyncio.

Once you understand the concepts in this guide, you will be able to develop programs that can leverage the asyncio library in Python to process many tasks concurrently and make better use of your machine resources, such as additional CPU cores.

Table of Contents

1. Introduction to Asynchronous Programming
2. Getting Started with Asyncio in Python
3. Working with Asyncio Event Loops
4. What Are Coroutines
5. Using Asyncio for Network Programming
6. Advanced Asyncio Concepts
7. When to Use Asyncio
8. When Not to Use Asyncio
9. Best Practices for Asynchronous Programming in Python

Introduction to Asynchronous Programming

Asynchronous refers to the concept of not occurring at the same time as something else. In the context of programming, asynchronous refers to the execution of tasks that do not block the flow of execution of other tasks. This means that multiple tasks can be performed concurrently, rather than sequentially.

This is achieved by using asynchronous function calls, which return control to the calling function before the task is completed, allowing other tasks to be executed in the meantime.

For example, we can make an asynchronous function call. This will issue the request to make the function call and will not wait around for the call to complete. We can choose to check on the status or result of the function call later.

One of the main benefits of asynchronous programming is improved performance, as it allows for non-blocking IO operations and allows multiple tasks to be performed simultaneously.

Getting Started with Asyncio in Python

The asyncio module contains utilities to implement asynchronous programming in Python.

It allows you to write concurrent code using asynchronous programming techniques rather than traditional thread-based concurrent programming.

You may be wondering how asyncio differs from traditional thread-based concurrent programming in Python...

Asyncio is particularly useful in Python because it allows you to write concurrent code in a single-threaded environment, which can be more efficient and easier to work with than using multiple threads.

Understanding the async, await syntax

We use def async to define asynchronous functions in Python. These are special types of functions that don't execute right away when called. Instead, they are scheduled, and we must then use await to execute and wait for the function to return.

This last point is key and is the reason why async functions are so powerful.

We can also execute async functions in other parts of our program with asyncio.run()

async def an_async_function():
    await asyncio.sleep(1)
    print("Hello, World!")

asyncio.run(my_async_function())

What is the Event Loop

The event loop is the central mechanism in asyncio that schedules and runs asynchronous tasks. It works by continuously polling for events and running the appropriate tasks when they are ready.

You can think of the event loop as a kind of "coordinator" that manages the execution of asynchronous tasks.

Creating and customizing event loops

By default, asyncio will create an event loop for you when you use asyncio.run() or call asyncio.AbstractEventLoop.run_until_complete(). However, you can also create and customize your own event loop if needed.

loop = asyncio.get_event_loop()
loop.run_until_complete(an_async_function())
loop.close()

The event loop starts with the loop.run_until_complete() method, which blocks until all tasks have completed. Finally, the event loop is closed with the loop.close() method.

Here are some other ways you can run an event loop in Python using the asyncio module:

1. loop.run_forever(): This method runs the event loop indefinitely. It blocks until the event loop is stopped with the loop.stop method or until an exception is raised.
2. loop.run_until_complete(future): This method runs the event loop until the given future is completed. It blocks until the future is done or until an exception is raised.
3. loop.run_in_executor(executor, func, *args): This method runs the given func in a thread or process executor and returns a Future object. The event loop will run until the Future is done or until an exception is raised.
4. loop.run_as_soon_as_possible(callback): This method schedules the given callback to be run as soon as possible. It does not block.
5. loop.call_later(delay, callback, *args): This method schedules the given callback to be run after the specified delay in seconds. It does not block.

What Are Coroutines

A coroutine is a task that can be suspended and resumed. In the context of asynchronous programming, when we talk about tasks that can be performed concurrently, we are referring to coroutines.

Therefore, coroutines are the unit of concurrency used in asyncio programs.

Many coroutines can be created and executed at the same time. They have control over when they will suspend and resume, allowing them to cooperate as to when concurrent tasks are executed.

This is called cooperative multitasking and is different from the multitasking typically used with threads called preemptive multitasking.

A coroutine can be defined using the async def expression. It can take arguments and return a value, just like a function. Calling a coroutine function will create a coroutine object, but it does not execute the coroutine function right away.

async def my_coroutine(delay):
    await asyncio.sleep(delay)
    print(f'Finished waiting for {delay} seconds')

# creates but does not execute the coroutine
coroutine_1 = my_coroutine(1)
# await starts the coroutine and blocks until finished
result = await my_coroutine(2)

When a coroutine is created but never executed, we will see a warning like this when the Python program finishes:

sys:1: RuntimeWarning: coroutine 'my_coroutine' was never awaited

We can avoid this warning and support top-level await expressions by starting the Python interpreter with a default event loop. This is also the default in IPython.

python -m asyncio script.py

Having top level await is neat, but it can cause problems too. There can only be one event loop running in a thread. If any of your code tries to launch an event loop, perhaps by calling asyncio.run(), you'll get an error, so be careful.

What's so great about coroutines

Threads and processes achieve multitasking managed by the operating system (OS) that chooses which threads and processes should run, when, and for how long. The OS switches between threads and processes rapidly, suspending those that are not running and resuming those granted time to run. This is called preemptive multitasking.

Coroutines in Python provide an alternative type of multitasking called cooperating multitasking. This allows coroutines to cooperate by design, choosing how and when to suspend their execution.

Another key aspect of coroutines is that they are more lightweight than threads. This means they are faster to start and use less memory. They may use less than 1 KB of memory to execute. Essentially a coroutine is a special type of function, whereas a thread is represented by a Python object associated with an operating system thread with which the object must interact.

Coroutines are a new alternate, interesting, and powerful approach to concurrency, different from thread-based and process-based concurrency. As such, we may have thousands of threads in a Python program, but we could easily have tens or hundreds of thousands of coroutines just in one thread.

You can also schedule coroutines for execution in separate threads, with ThreadPoolExecutor, or separate processes using ProcessPoolExecutor.

How to Run Blocking Tasks with Asyncio

We often need to execute a blocking function call within an asyncio application because, in practice, most workloads include a mix of IO-bound operations and also CPU-bound operations.

This could be for many reasons, such as:

• To execute a CPU-bound task like calculating something.
• To execute a blocking IO-bound task like reading or writing from a file.
• To call into a third-party library that does not support asyncio yet.

Making a blocking call directly in an asyncio program will cause the event loop to stop while the blocking call is executing. It will not allow other coroutines to run in the background.

This can be prevented by running the blocking call outside of the event loop, which we can do with asyncio.to_thread().

# execute a function in a separate thread
await asyncio.to_thread(my_blocking_task)

The asyncio.to_thread() function takes a function to execute and any arguments. It returns a coroutine that can be awaited or scheduled as an independent task. The function is then executed in a separate thread.

The asyncio.to_thread() function creates a ThreadPoolExecutor behind the scenes to execute blocking calls.

As such, the asyncio.to_thread() function is only appropriate for IO-bound tasks, and we should not use this method of asyncio for CPU bound tasks.

Using Asyncio for Network Programming

Network calls are a good use case for asyncio since they're IO-bound, and we can just fire and forget the write or read, and our program can go on to perform other tasks. The caller does not need to wait for the operation to complete before returning, so we should not waste time just checking on them.

The read and write operations are performed somehow (e.g., by the underlying operating system or perhaps a task queue like Kafka), and the status of the action and/or data is retrieved by the caller later, once available, or when the caller is ready.

Non-blocking IO is a way of performing IO where reads and writes are requested, although performed asynchronously. The caller does not need to wait for the operation to complete before returning. Non-blocking IO is implemented in practice with asynchronous programming.

Let's go through a few asyncio examples that implement non-blocking IO in client and server networking:

HTTP server

This is an example of an asyncio-based HTTP server that serves static files from a given directory. It uses the aiohttp library to handle HTTP requests and responses.

import asyncio
from aiohttp import web

async def handle(request):
    with open('index.html', 'rb') as f:
        return web.Response(body=f.read(), content_type='text/html')

async def main():
    app = web.Application()
    app.add_routes([web.get('/', handle)])
    return app

if __name__ == '__main__':
    web.run_app(main())

WebSocket server

This is an example of an asyncio-based WebSocket server that broadcasts messages to all connected clients. It uses the websockets library to handle WebSocket connections.

import asyncio
import websockets

async def broadcast(websockets, message):
    for ws in websockets:
        await ws.send(message)

async def handler(websocket, path):
    async for message in websocket:
        await broadcast(websockets, message)

async def main(host, port):
    server = await websockets.serve(handler, host, port)
    await server.wait_closed()

if __name__ == '__main__':
    asyncio.run(main('127.0.0.1', 8888))

WebSocket client

This is an example of an asyncio-based WebSocket client that connects to a server, sends a message, and prints the response. It uses the websockets library to handle the WebSocket connection.

import asyncio
import websockets

async def main():
    async with websockets.connect('ws://localhost:8888') as websocket:
        await websocket.send("Hello, world!")
        response = await websocket.recv()
        print(f'Received: {response}')

asyncio.run(main())

Advanced Asyncio Concepts

So far, we've only seen examples of how to run a single asynchronous function to perform non-blocking IO operations.

When we have many operations to run concurrently, we need to coordinate and manage those operations. Fortunately, the asyncio module has some functions that make it easier for us.

How to use asyncio.gather() to wait for multiple async tasks

import asyncio

async def task_one():
    print("Starting task one")
    await asyncio.sleep(1)
    print("Finishing task one")
    return 1

async def task_two():
    print("Starting task two")
    await asyncio.sleep(2)
    print("Finishing task two")
    return 2

async def main():
    # This will wait for all the coroutines
    results = await asyncio.gather(task_one(), task_two())
    print(results)

asyncio.run(main())

This code will produce the following output:

Starting task one
Starting task two
Finishing task one
Finishing task two
[1, 2]

As you can see, the tasks are run concurrently rather than sequentially. This can be useful for improving the performance of an application by allowing it to make better use of available CPU resources.

The asyncio.gather() method accepts many arguments and returns a list of the results returned by all the coroutines.

It is also common to create many coroutines beforehand and then gather them later. We can collect many coroutines together into a list either manually or using a list comprehension. We can then unpack this list as arguments to gather(). Note the use of the star operator (*) here:

# creates many coroutines
coros = [my_async_task(i) for i in range(10)]
...
# then, run the tasks
results = await asyncio.gather(*coros)

How to use asyncio.wait() to manage multiple asynchronous tasks

The asyncio.wait() function blocks and returns a tuple containing two sets of tasks: those that have completed and those that are still pending. In this example, we iterate over the completed tasks and print their results.

Using the asyncio.wait() function, you can easily manage multiple asynchronous tasks and perform actions when they have completed. This can be particularly useful for situations where you need to wait for multiple tasks to complete before moving on to the next step of your program and for setting timeouts for some tasks.

# Run these tasks and wait 
tasks = [task_one(), task_two()]
    completed, pending = await asyncio.wait(tasks)

    results = [t.result() for t in completed]
    print(results)

Alternatively, we could use the asyncio.as_completed() function to run concurrent tasks and iterate their results as they become available. This is possible because asyncio.as_completed() returns an iterable of the coroutines that can be awaited.

Chaining multiple asynchronous tasks

There are situations where you may need to perform a series of asynchronous tasks in a specific order. Since coroutines are awaitable, another coroutine can await it. You may use this class to chain async tasks since the asyncio library does not (yet) provide a similar function.

class Chain:
    "A class to execute async tasks sequentially"
    def __init__(self, tasks):
        self.tasks = tasks

    async def __call__(self, *args, **kwargs):
        for task in self.tasks:
            args = await task(*args, **kwargs)
            args = (result,)
        return result

To use this class, you can pass a list of asynchronous tasks to the constructor and then call the Chain instance to execute the tasks in the specified order. The result of each task is passed as an argument to the next task in the chain.

chain = Chain([task_one, task_two])
# block until tasks are performed synchronously
result = await chain() # returns the last task result

When to Use Asyncio

Coroutines are an alternative to threading-based concurrency that is provided by the Python language and further supported by the asyncio module. They are suited to non-blocking IO with subprocesses and sockets. However, blocking IO and CPU-bound tasks can be used in a simulated non-blocking manner using threads and processes under the covers.

Any program written using threads or processes can be rewritten or instead written using coroutines if we so desire.

A coroutine is even more lightweight than a process. Processes, like threads, are created and managed by the underlying operating system and are represented by a multiprocessing.Process object.

This means that coroutines are significantly faster than a process to create and start and take up much less memory. A coroutine is just a special function, whereas a Process is an instance of the interpreter that has at least one thread.

As such, we may have thousands of threads in a Python program, but we could easily have tens or hundreds of thousands of coroutines all in one thread.

When Not to Use Asyncio

There are many misconceptions about Python concurrency and especially around asyncio. Using asyncio does not magically solve all the issues with Python.

For example:

• Asyncio works around the Global Interpreter Lock (GIL).
• Asyncio is faster than using threads.
• Asyncio avoids the need for locks and other synchronization methods.
• Asyncio is easier to use than threads.

These are all False.

Only a single coroutine can run at a time by design, and they cooperate to execute. This is just like threads under the GIL. However, the GIL still applies to asyncio code, and the performance of asyncio programs may be affected by the GIL in the same way as multithreaded programs. So asyncio is not faster and is also not multithreaded and not parallel.

Any program you can write with asyncio, you can also write with threads, and it could actually be as fast or even faster. It could also probably be simpler and easier to read and interpret by other developers.

If you need to bypass the GIL and take full advantage of multiple CPU cores in your Python program, you may want to consider using a different concurrency framework like multiprocessing and other libraries such as concurrent.futures. These frameworks allow you to create multiple processes, each with its own Python interpreter and GIL, which can run in parallel on separate CPU cores.

Any concurrency pitfalls you might expect with threads, you can also encounter with coroutines. You must ensure coroutines are thread safe and safe from deadlocks and race conditions, just like with threads or processes.

Another reason not to use asyncio is that you may not like asynchronous programming. Even though asynchronous programming has been popular for some time now in various programming communities, it is different from procedural, object-oriented, and functional programming, and some developers just don’t like it.

If you don’t like it, don’t use it. It’s a fair reason.

You can achieve concurrency in many ways other than using asynchronous programming, using threads or processes as needed.

You can learn all about threads and processes in our guides:

Running multiple processes in Python - a multiprocessing guide

Learn how to write Python programs that can run simultaneously on all processors available and also as background processes.

Hacker CultureAdrian Cruz

Best Practices for Asynchronous Programming in Python

These are some tips that will have you writing efficient and maintainable asynchronous code and save you lots of headaches:

Use asyncio's high-level APIs whenever possible

The asyncio library provides several high-level APIs that make it easy to write concurrent code. For example, you can use the gather function to run multiple tasks concurrently or the as_completed function to iterate over a group of tasks as they complete. These APIs can help you write efficient and maintainable code, as they abstract away many of the low-level details of concurrent programming.

Avoid blocking the event loop

The event loop is the heart of asyncio, and it is responsible for scheduling and executing asynchronous tasks. If you block the event loop, you can cause performance issues and disrupt the execution of other tasks. To avoid blocking the event loop, use non-blocking IO operations whenever possible, and avoid using blocking functions such as time.sleep or threading.Lock.

Use async context managers and async iterators

Async context managers and async iterators can help you write asynchronous code that is easy to understand and maintain. For example, you can use async context managers to manage resources that need to be acquired and released asynchronously, or you can use async iterators to iterate over asynchronous data streams in a natural and intuitive way.

Common pitfalls to avoid when doing asynchronous programming

1. Forgetting to await asynchronous tasks: When you await an asynchronous task, you are telling the event loop to execute that task and pause the current task until the awaited task completes. If you forget to await a task, it will be scheduled to run, but the event loop will not wait for it to complete before moving on to the next task. This can lead to unexpected behavior and race conditions.
2. Not using asynchronous functions when appropriate: On the other hand, it is important to use asynchronous functions when appropriate, as this can significantly improve the performance and scalability of your application. If you have long-running tasks or tasks that perform IO operations, consider using asynchronous functions to ensure that they do not block the event loop.
3. Not using synchronization techniques: Any concurrency pitfalls you might expect with threads, you can also encounter with coroutines. While only one coroutine can run within the event loop at one time, they can be suspended and resumed while using a shared variable. You must ensure coroutines are safe from deadlocks and race conditions, just like with threads or processes.

If you keep these practices in mind and avoid the multiple pitfalls related to concurrency and asynchronous programming, you should be able to develop Python programs that can process many tasks concurrently.

You may even want to experiment with running asynchronous tasks combined with other concurrency techniques, especially when pairing asyncio with multiprocessing and  concurrent.futures.Executor instances.

References

• Event Loop
• Coroutines
• Using Asyncio in Python
• Python Asyncio The Complete Guide

How are you using asyncio in your programs?
I’d love to hear about it. Please let me know in the comments.

Do you have any questions?
Leave your question in a comment below, and I'll answer it with my best advice.

Threads have been available since Python 2, but they are often misunderstood and misused because of the limitations and capabilities of threads in Python. When used correctly, they can make your code run faster and more efficiently with very little effort.

In this guide, you will learn about threads, their use cases, and how to use them to run multiple parts of your Python programs concurrently. We'll also go over some alternatives and scenarios where threads may not be a good option.

By the end of this guide, you will be able to develop programs with improved performance by applying multithreading techniques that make better use of your machine resources, such as additional CPU cores.

Table of Contents

1. What is a Thread
2. Threads and Processes
3. Global Interpreter Lock
4. Starting Threads
5. Waiting for Threads
6. Thread Pool
7. ThreadPoolExecutor
8. Synchronization between Threads
9. When to use Threads
10. Multithreading vs Multiprocessing
11. Why Not Use Asyncio
12. Python Multithreading Best Practices

What is a Thread?

Imagine that you have a bunch of tasks that need to be done, like making a sandwich, doing your homework, and playing a video game. You can only do one thing at a time, so you have to decide what to do first.

Now imagine that you have a friend who can help you with some of these tasks. You can both work on different tasks at the same time, which means you can get everything done faster.

This is similar to how threads work in a computer. The computer has a lot of tasks that it needs to do, and it can only do one thing at a time. But it can create little helpers called threads that can work on different tasks at the same time, which makes the computer faster and more efficient.

A thread is like a separate mini-program that runs within the main program. Each thread has its own set of instructions to follow and its own memory space, which means it can perform a specific task independently of the other threads within the same program.

Threads are helpful because they allow a program to do multiple things at the same time. For example, a web browser might use threads to download a file from the internet while also displaying a progress bar and responding to user input.

The operating system (which is the software that runs on a computer and manages the hardware) creates and manages threads. It gives each thread a priority (which determines how much work it gets to do) and makes sure that all the threads get a fair share of the available resources.

A Python thread is just a reference to the native thread provided by the underlying operating system.

Threads and Processes

Every program is a process and has at least one thread that executes instructions for that process. When we run a Python script, it starts a new Python process that runs our code in the main thread.

A thread is a lightweight subprocess, meaning that it runs within the process of a program and shares the same memory space. This allows multiple threads to run concurrently within a single process and can make a program more efficient by allowing different parts of the program to run at the same time.

The Python process will terminate once all (non-background threads) are terminated.

For more information about processes and when to use them instead of threads, see Running Multiple Processes in Python - A Multiprocessing Guide.

Global Interpreter Lock

In Python, the Global Interpreter Lock (aka GIL) is a mechanism that prevents multiple native threads from executing Python code at once.

This lock is necessary because the Python interpreter is not thread-safe, which means that multiple threads can potentially interfere with each other and cause race conditions.

The GIL makes it easy to write simple, thread-safe Python programs, but it can also limit the performance of multithreaded programs, especially those that are heavily CPU-bound. This is because the GIL prevents multiple threads from executing Python code simultaneously, even if the CPU has multiple cores and could potentially run the threads in parallel.

This means that threads in Python are not truly parallel and instead run in a concurrent manner but only one at a time.

So while threads cannot run in parallel, they can run concurrently, which means that more than one task can be in progress at the same time, even if they are not executing at the same time. Once a pending thread can be resumed, Python will switch to executing that thread while the others are still in progress.

Starting Threads

Threads can be started in a number of ways. The most common way to start a thread is to use the threading module in Python. This module provides a number of classes and functions that can be used to start, control, and manage threads.

Given this function which we want to execute in a separate thread from the main program:

def thread_function():
    print("Thread function executing")
    # Simulate a long task
    sleep(1)

We can create a new Thread and specify the function as the target argument in the constructor:

# Create a new thread and start it
thread = Thread(target=thread_function)
thread.start()

The complete example of executing the function in a separate thread:

from threading import Thread

def thread_function():
    print("Thread function executing")
    # Simulate a long task
    sleep(1)

if __name__ == "__main__":
    # Create a new thread and start it
    thread = Thread(target=thread_function)
    thread.start()

    # The main program continues concurrently with the new thread
    print("Main program continuing to execute")

After the thread is started, the main program continues to execute concurrently with the new thread. The two threads will run concurrently until the main program finishes or until the new thread is stopped or terminated.

Starting threads with arguments

What if we have a function that takes arguments in order to run? Let's update our example to demonstrate how to run a thread with a target function with arguments.

def thread_function(greet, seconds):
    print(greet)
    # Simulate a long task
    sleep(seconds)

We can specify the arguments to the target function in the Thread constructor by setting the args parameter to a tuple of arguments.

# Create a new thread and start it
thread = Thread(target=thread_function, args=("Hello from thread", 3))
thread.start()

Creating daemon threads

The thread constructor also accepts a daemon argument which can be set to True and indicates that we don't need to wait until this thread finishes. These types of threads may be shut down abruptly once the main thread finishes.

# Create a new daemon thread
thread = Thread(target=thread_function, daemon=True)
thread.start()

Daemon threads are typically used for tasks that do not need to be completed before the program exits, such as cleaning up resources or performing periodic maintenance. Also, for tasks that are not related to the main program.

For example, deleting some temp files if you don’t need them anymore may take some time, and we don't need to wait around to know when. Or perhaps some database records to be deleted and also their associated resources. Those can be scheduled in a separate daemon thread.

They are not intended for long-running tasks or tasks that need to be completed before the main program exits.

Waiting for Threads

When we need to wait for another thread to finish executing, we can call the join() method. This is not always needed, as the main thread will not exit until the other threads have finished executing or when only daemon threads are left. But when you need to ensure a thread has completed its task, use join().

# Wait for the thread to finish
thread.join()
print("The thread has completed")

The main program will pause at this point until the thread finishes execution. Once the thread finishes, the main program continues and executes the statement after the join() call returns.

How to set a thread timeout

It is generally a good practice to use timeouts when joining threads in Python because it allows you to specify a maximum amount of time to wait for the thread to finish. This can be especially useful in situations where you are running multiple threads concurrently, and you want to ensure that your program does not get stuck waiting for a thread that may never complete.

The join() method accepts an optional timeout argument to set the maximum amount of seconds to block when waiting for a thread to finish. Once the timeout is up, the thread will not block anymore, and we can call the is_alive() method to check if a timeout happened - if the thread is still alive, the join() call timed out.

Here's an example:

# Wait 10 seconds for the thread to finish
thread.join(timeout=10)
if thread.is_alive():
    print("The thread timed out")
else:
    print("The thread has completed")

Thread Pool

A thread pool is a group of worker threads that are kept in a pool and are available to perform tasks as needed. Thread pools are used to improve the performance and scalability of programs that perform a lot of tasks concurrently.

So in our example before, instead of having a single friend who can help you with your tasks, imagine that you have a group of friends who can help.

Each of your friends is a worker thread, and they are all part of your thread pool. When you have a task that needs to be done, you can ask one of your friends to help you with it. Once they finish the task, they go back to the thread pool and wait for the next task.

Using a thread pool can help to improve the performance of a program because the threads can be reused, which means that the program doesn't have to create a new thread every time it needs to perform a task. This can save time and resources.

Let's see an example of how to use a ThreadPool:

from multiprocessing.pool import ThreadPool

def task_function(x):
    print(f"Calculating square of {x} in thread.")
    return x**2

if __name__ == "__main__"
    with ThreadPool() as pool:
        numbers = [1, 3, 5, 8, 13]
        result = pool.map(task_function, numbers)
        print(result)

In this example, a ThreadPool is used to create and start threads to run the target function for each number in the numbers list.

We can specify an argument to limit how many worker threads to use. If we leave this out, Python creates as many threads as the number of processors in the machine, which is obtained by calling os.cpu_count().

Let's visualize a single thread running a task vs using multiple threads. Note how using more threads brings execution time down:

Thread pools are no more than a wrapper around the threading module, and in general, we should prefer to use the ThreadPoolExecutor, which has a simpler interface and better compatibility with many other libraries, including asyncio.

ThreadPoolExecutor

ThreadPoolExecutor is a class that can be used to create and manage a thread pool. It is part of the concurrent.futures module, which provides a high-level interface for executing tasks concurrently using threads or processes.

A ThreadPoolExecutor manages a fixed number of worker threads and a task queue. When a task is submitted to the executor, it is placed in the task queue, and a worker thread is assigned to perform the task when one becomes available. The worker thread then executes the task and returns the result (if any) to the caller.

A ThreadPoolExecutor differs from a simple thread pool in that it provides a more feature-rich interface for managing and executing tasks concurrently. For example, it allows you to:

• Submit tasks asynchronously using Future objects, which can be used to track the progress of the task and retrieve the result when it is completed.
• Set a maximum number of worker threads and a maximum size for the task queue to control the level of concurrency.
• Handle exceptions and cancel tasks that have not yet started.

Here is a simple example of how to use the ThreadPoolExecutor class to execute a task concurrently in Python:

from concurrent.futures import ThreadPoolExecutor

def task_function(n):
    print(f"Task {n} running")

if __name__ == "__main__":
    # Create a ThreadPoolExecutor with 2 worker threads
    with ThreadPoolExecutor(max_workers=2) as executor:
        # Submit a task to the executor
        future = executor.submit(task_function, 1)

        # The main program continues to execute concurrently with the task
        print("Main program continuing to execute")

In this example, the task_function is a simple function that prints a message to the console. The ThreadPoolExecutor is created with a maximum of 2 worker threads, which means that at most 2 tasks can be executed concurrently.

Since Python 3.5, if we don't specify the number of max_workers, it will default to the number of processors on the machine multiplied by 5.

To submit a task to the executor, you can use the submit method and pass in the function and any arguments that the function requires. This returns a Future object, which represents the result of the task and can be used to track the progress of the task and retrieve the result when it is completed.

We can also use a ThreadPoolExecutor as a simplified version of the ThreadPool we discussed earlier:

from time import sleep
from random import random
from concurrent.futures import ThreadPoolExecutor

def target_function(name):
    # sleep for less than a second
    sleep(random())
    return name
 
# start the thread pool
with ThreadPoolExecutor(10) as executor:
    # execute tasks concurrently and process results in order
    for result in executor.map(target_function, range(10)):
        # retrieve the result
        print(result)

You may be interested in the results of the ThreadPoolExecutor as they become available instead of once all the tasks have been completed. Note that the order in which tasks are completed is not guaranteed.

from concurrent.futures import as_completed
...

# start the thread pool
with ThreadPoolExecutor(10) as executor:
    # submit tasks and collect futures
    futures = [executor.submit(target_function, i) for i in range(10)]
    # process task results as they are available
    for future in as_completed(futures):
        # retrieve the result
        print(future.result())

What Are Futures

A Future object acts as a placeholder for the result of a task that has not yet completed. It allows you to track the progress of the task and retrieve the result when it is completed.

It is also sometimes called a promise or a delay. It provides a context for the result of a task that may or may not be executing and a way of getting a result once it is available.

In Python, the Future object is returned from an Executor, such as the ThreadPoolExecutor, when calling the submit() method to dispatch a task to be executed asynchronously.

A Future also provides access to the result of the task via the result() method. If an exception was raised while executing the task, it will be re-raised when calling the result() method or can be accessed via the exception() method.

• result(): Access the result from running the task.
• exception(): Access any exception raised while running the task.

Both the result() and exception() methods allow a timeout to be specified as an argument, which is the number of seconds to wait for a return value if the task is not yet complete. If the timeout expires, then a TimeoutError will be raised.

Synchronization between threads

When multiple threads are running concurrently, it is important to ensure that they do not interfere with each other. This is done through synchronization, which is the process of ensuring that all threads are in sync and are accessing the same data.

Consider a program that has a shared counter variable that is incremented by multiple threads. If two threads try to increment the counter simultaneously, it is possible that the final value of the counter will be incorrect because the threads may interfere with each other's updates. This is called a race condition.

A race condition is a concurrency failure case when two threads run the same code and access or update the same resource (e.g. data variables, stream, etc.), leaving the resource in an unknown and inconsistent state.

Race conditions often result in unexpected behavior of a program and/or corrupt data.

Try running this example many times, and you will encounter a race condition:

import threading

counter = 0

def increment_counter():
    global counter
    counter += 1

# Create two threads that try to increment the counter simultaneously
t1 = threading.Thread(target=increment_counter)
t2 = threading.Thread(target=increment_counter)

# Start the threads
t1.start()
t2.start()

# Wait for the threads to finish
t1.join()
t2.join()

# Print the final value of the counter
print(counter)

Using threading.local

A key difference between processes and threads is that processes have their own separate memory space, while threads share the memory space of the process that created them. This means that threads can communicate with each other more easily, but it also means that if one thread modifies the shared memory, it can affect the other threads in the process.

One way to make sure threads don't interfere with each other is to separate the data they operate on, so other threads can't access them. We can do that using the threading.local class which provides a dictionary-like object that is specific to a thread, and each thread gets its own separate copy of the values stored in the threading.local object.

import threading

# Create a thread-local data object
local_data = threading.local()

def process_request(request):
    # Store the request data in the thread-local data object
    local_data.request = request

    # Do some processing with the request data
    # ...

# Start a new thread and process a request
t = threading.Thread(target=process_request, args=(request,))
t.start()

In this example, the process_request function stores the request data in the local_data object, which is specific to the thread that is executing the function. This means that the request data will not be shared with any other threads, and each thread can have its own separate copy of the request data.

We can also separate the data between threads using threading and queue:

import threading
import queue

def worker(q):
    while True:
        item = q.get()
        if item is None:
            break
        # process the item
        print(item)
        q.task_done()

q = queue.Queue()

# add items to the queue
for item in range(100):
    q.put(item)

# create worker threads
threads = []
for i in range(4):
    t = threading.Thread(target=worker, args=(q,))
    t.start()
    threads.append(t)

# wait for all tasks to be completed
q.join()

for t in threads:
    t.join()

Locks

A (mutually exclusive) lock can be used to ensure that only one thread at a time executes a critical section of code at a time, while all other threads trying to execute the same code must wait until the currently executing thread is finished with the critical section and releases the lock.

To avoid a race condition in Python using a lock, we can use the threading.Lock class to create a lock object, and then use the acquire and release methods of the lock to control access to the shared resource.

Here is an example of how we can use a lock to avoid a race condition in Python:

from threading import Thread

counter = 0
lock = threading.Lock()

def increment_counter():
    global counter
    with lock:
        # Critical section executed by a single thread at a time
        counter += 1

# Create two threads that try to increment the counter simultaneously
t1 = Thread(target=increment_counter)
t2 = Thread(target=increment_counter)

# Start the threads
t1.start()
t2.start()

# Wait for the threads to finish
t1.join()
t2.join()

# Print the final value of the counter
print(counter)

In this example, the increment_counter function acquires the lock before accessing and modifying the shared counter variable. This ensures that only one thread can execute the code inside the with block at a time, which prevents the race condition from occurring. As a result, the final value of the counter will always be 2, regardless of the order in which the threads execute.

Something to keep in mind is that we must ensure the holder releases any acquired locks. Otherwise, other threads may wait indefinitely for access, and our program may never finish. This is called a deadlock.

When to Use Threads

As we saw earlier, the Python interpreter prevents more than one thread from executing bytecode at the same time.

This is achieved using a lock called the Global Interpreter Lock or GIL, as we learned in the previous section.

To bypass the GIL and take full advantage of multiple CPU cores in your Python program, you may want to consider using a different concurrency framework like  multiprocessing and also concurrent.futures. These frameworks allow you to create multiple processes, each with its own Python interpreter and GIL, which can run in parallel on separate CPU cores.

There are other times when the lock is released by the interpreter, and we can achieve parallel execution of our concurrent code in Python.

Examples of when the lock is released include:

• When a thread is performing blocking IO.
• When a thread is executing C code and explicitly releases the lock.

This makes threads good candidates for IO-bound tasks.

An IO-bound task is a type of task that involves reading from or writing to a device, file, or socket connection. These tasks involve input and output (IO), and the speed of these operations is bound by the device, hard drive, or network connection. This is why these tasks are referred to as IO-bound.

A thread performing an IO operation will block for the duration of the operation. While blocked, this signals to the operating system that a thread can be suspended and another thread can execute, called a context switch.

Additionally, the Python interpreter will release the GIL when performing blocking IO operations, allowing other threads within the Python process to execute.

This is why blocking IO operations are an excellent use case for using threads in Python.

Let's look at practical examples of blocking IO operations in real applications:

Reading or writing a file from the hard drive

# Open the file for writing
def write_file():
    with open('filename.txt', 'w') as f:
        # Write some text to file, could take long...
        f.write('Local data saved.')

Printing a document

If you want to print the output to a specific device, such as a printer, you can use the print() function in combination with a library that provides printing functionality. For example, you can use the pycups library to print to a CUPS (Common Unix Printing System) printer or the win32print library to print to a printer on a Windows operating system.

import cups

# Connect to the local CUPS server
conn = cups.Connection()

# Get a list of available printers
printers = conn.getPrinters()

# Print a document to the default printer
conn.printFile(printers.keys()[0], '/path/to/file.pdf', 'Title', {})

Downloading or uploading a file

import requests

# URL of the file to download
url = 'https://www.example.com/file.zip'

def download_url(url):
    # Send an HTTP GET request to the URL
    response = requests.get(url)
    # Check for a successful response
    if response.status_code == 200:
        # Save the content of the response to a file
        with open('file.zip', 'wb') as f:
            # Writing to file here is blocking
            f.write(response.content)

This will download the file at the specified URL and save it to the current working directory with the name file.zip.

Working with APIs

Here is an example of how you can use the Python requests library to query a real-world API:

import requests

def openweather_api(city="London", country_code="UK"):
    # Set the API endpoint and API key
    api_endpoint = "https://api.openweathermap.org/data/2.5/weather"
    # Set the parameters for the API request
    params = {
        "q": f"{city},{country_code}",
        "appid": "YOUR_API_KEY_HERE",
    }
    # The request to the API endpoint blocks until we get a response
    response = requests.get(api_endpoint, params=params)
    # Check the status code of the response to check if successful
    if response.status_code == 200:
        # If the request was successful, get the data from the response
        data = response.json()
        # You can now process the data as needed
        print(data)
    else:
        print("Request failed with status code:", response.status_code)
        print("Error message:", response.text)

Connecting to a database

import sqlite3

with sqlite3.connect("database.db") as conn:
    # Create a cursor
    cursor = conn.cursor()
    # The execute blocks while connecting to the database
    cursor.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT, age INTEGER)")
    data=('Alice', 25)
    cursor.execute("INSERT INTO users (name, age) VALUES (?, ?)", data)

Taking a photo or recording a video.

To take a photo or record a video using Python, you will need to use a library or module that provides access to the camera or video capture functionality of your system. One option is the cv2 (OpenCV) library:

import cv2

# Open the camera
camera = cv2.VideoCapture(0)

def take_photo(filename='photo.jpg')
    ret, frame = camera.read()
    if ret:
        # Save the photo to a file
        cv2.imwrite(filename, frame)

def take_video(filename='video.mp4', width=640, height=480)
    fourcc = cv2.VideoWriter_fourcc(*'MP4V')
    out = cv2.VideoWriter(filename, fourcc, 30.0, (width,height))
    
    while True:
        # Capture a frame from the camera
        ret, frame = camera.read()
        if not ret:
            break
        # Write the frame to the video file
        out.write(frame)
        # Show the frame
        cv2.imshow("Webcam", frame)

    # Release the camera and video writer
    camera.release()
    out.release()
    cv2.destroyAllWindows()

Multithreading vs Multiprocessing

Threads and Processes are quite different, and choosing one over the other is intentional.

A Python program is a process that has a main thread. You can create many additional threads in a Python process. You can also fork or spawn many Python processes, each of which will have one thread and may spawn additional threads.

More broadly, threads are lightweight and can share memory (data and variables) within a process, whereas processes are heavyweight and require more overhead and impose more limits on sharing memory (data and variables).

Typically, processes are used for CPU-bound tasks, and threads are used for IO-bound tasks, and this is a good heuristic, but this is not always the case. In the real world, we may need to write programs that contain a mix of both IO-bound tasks and CPU-bound tasks.

In this case, we should experiment with using threading for the IO-bound tasks of our programs and processes or multiprocessing too for the CPU-bound parts. Perhaps try it and see.

You can learn all about multiprocessing in this guide:

Running multiple processes in Python - a multiprocessing guide

Learn how to write Python programs that can run simultaneously on all processors available and also as background processes.

Hacker CultureAdrian Cruz

Why Not Use Asyncio

Asyncio is a newer alternative to using threads introduced in Python 3.4 and is probably worth investigating as a replacement for multithreading.

Asyncio is designed to support large numbers of IO operations, perhaps thousands to tens of thousands, all within a single Thread.

If you are writing a new program from scratch, you may be better off using asyncio instead of threads.

On the other hand, if you are working with an existing codebase containing legacy code or targeting an older version of Python, your best bet is to use threads.

Simply put, threads should be considered only when working with legacy code if you need to speed up some IO-bound tasks in an established code base.

You can learn all about asyncio in python with our guide:

Python asyncio - A Guide to Asynchronous Programming

In this guide, we’ll introduce asynchronous programming in Python and review fundamental concepts like how to define, create and run asynchronous functions, coroutines, and some common use cases and best practices.

Hacker CultureAdrian Cruz

Python Multithreading Best Practices

Now that we know threads work and how to use them let’s review some best practices to consider when bringing threads into our Python programs.

Best practices to keep in mind when using threads in Python:

1. Use Context Managers
2. Use Timeouts When Waiting
3. Use Locks to Prevent Race Conditions
4. Consider Using Asyncio

References

• threading module
• ThreadPoolExecutor
• Fluent Python Book
• Python Threading: The Complete Guide

Did you find this guide helpful?
If so, I'd love to know. Please share what you learned in the comments below.

How are you using threads in your programs?
I’d love to hear about it, too. Please let me know in the comments.

Do you have any questions?
Leave your question in a comment below, and I'll answer it with my best advice.

Open source is more important than ever, and more companies are realizing the benefits and increasing their adoption. As long as it's something popular that can speed up the development process, why not use it? It doesn't cost anything!

Well, it doesn't cost money... But I've found there are more subtle costs that could end up having a huge impact on the business, and we should consider them before introducing any dependencies to existing products or processes.

There's the cost of distributing the libraries you're using (and their dependencies) to your customers in your builds. That means your customers need to download a larger bundle, which makes their experience a bit slower.

There's the maintenance cost of outdated libraries, which often include outdated dependencies. Those could have known vulnerabilities with known exploits (and fixes), which can damage the brand, business, and customers.

When a library is no longer maintained, it's almost impossible to get support or request new features. The issues pile up on an endless list of open bugs. And those bugs are almost never fixed when you need them to be. You can count on Murphy's Law to kick in here.

Then there's the cost of the workarounds. Once you start uncovering bugs and you realize nobody is coming to save you, there are two possible outcomes: you either stop using that library, or you try to make it work on your own, taking on weeks of additional work.

On the other hand, there are specific characteristics that separate the quality open-source projects from their abandoned counterparts:

• Bundle sizes. Good projects are modular, so you can import only what you need and keep your bundle sizes low. JavaScript libraries can be checked using bundlephobia.com – also, look for libraries that are tree-shakeable.
• Source code. If you aren't able to inspect the code, don't use it because it could explode. But also, is the code clear? Are there comments? Could you explain how it works to others?
• Up to date. Look at the commit history. Are there recent commits and bug fixes landed? Are they keeping their dependencies up to date?
• Test coverage. If there are no tests, how do we know the code does what it's supposed to do? Browse through the tests and make sure that they cover enough cases.
• Open issues. Check the project's open issues to make sure the maintainers are active and listening. Check for a roadmap too.
• No vulnerabilities. Run npm audit after adding a new dependency to check that it doesn't introduce vulnerabilities. Bonus points if the library you plan to use has zero dependencies.
• Documentation. Can you find available docs that show how the library is meant to be used? Official or otherwise, having something you can refer to and share with other team members is super valuable.

Recently, I've had to estimate these costs when deciding on which libraries to use for a new project and was considering using the popular lodash library. Until I started looking at their long list of open bugs, realizing this project has been dead for the past 6 years, and then I read their source code, ridden with custom macros and over-engineered abstractions (see clone). And then it hit me...

This must be why there are so many open bugs. It takes so much effort to understand just one basic function that most people won't even try. Hell, even the original maintainers probably can't understand it at this point because of all the abstractions.

So I started working on a similar library to replace my lodash imports, and a handful of other libraries. But the main goal is that it should be easy to learn, easy to test, and easy to maintain. There are plenty of docs available to get you started, and I hope you join the conversation on github to help shape its future.

GitHub - adrian2x/atomic-fns: Like Lodash, but for ESNext and with types.

Like Lodash, but for ESNext and with types. Contribute to adrian2x/atomic-fns development by creating an account on GitHub.

GitHubadrian2x

Programming offers a chance to develop creativity, reasoning, and problem-solving skills. As a programmer, you get to bring ideas to life through code and using logic to create programs that can be executed by a computer.

If you know how to write computer programs, you can do all sorts of useful things. You might not be able to write programs to control cars, traffic lights, or your fridge right away, but you could make web pages, your own games, or even a program to help you with your homework.

When faced with obstacles, programming requires you to use problem-solving to find solutions. While it can be challenging (and at times frustrating), programming is also a fun and rewarding activity! The skills gained from programming can be beneficial in multiple professions, even if your chosen career is not directly related to computers.

How to Learn to Code

Like anything you try for the first time, it’s always best to start with the basics. Gymnasts don't usually do backflips on their first try. If you jump ahead too quickly, not only will you easily forget the basics, but you’ll also find the content of the later articles more complicated than it actually is.

Remember that the better you understand the basics, the easier it will be to understand more complicated ideas later on. When you find something frustrating or too challenging, here are some things that I find helpful:

1. Break a problem down into smaller pieces. Try to understand what a small piece of code is doing or think about only a small part of a difficult idea (focus on a small piece of code rather than trying to understand the whole thing at once).
2. If that still doesn’t help, it might be best to leave it alone for a while. Sleep on it and come back to it another day. This is a good way to solve many problems, and it can be particularly helpful for computer programmers.

What is a Computer Program

A program is a list of steps for a computer to follow. Programs are made up of lines of code. Each line tells the computer something specific about how to follow these instructions.

Software is a collection of computer programs.

Programs are like thoughts. If you didn’t have thoughts, you would probably just sit on the couch all day, staring at the ceiling. Your thought “get up off the couch” is a command or instruction that tells your body to stand up. In the same way, computer programs use commands to tell computers what to do.

The Python Programming Language

Like humans, computers use multiple languages to communicate—these are called programming languages. A programming language is simply a way to talk to a computer by using instructions that both humans and computers can understand.

The Python programming language has many features that make it extremely useful for beginners. Most importantly, you can use Python to write simple, efficient programs quite quickly. Python doesn’t use as many complicated symbols as other programming languages, which makes it easier to read and a lot friendlier for beginners.

The Python programming language was actually named after the Monty Python’s Flying Circus TV show, not after the snake.

Installing Python

To install Python, navigate to python.org/downloads and click on Download Python. At the time of this writing, the latest version is 3.11.1. Make sure you download a version higher than 3.10, which is faster than previous versions.

Once downloaded, double-click the file to run the installer.

Make sure you select Add Python to PATH, and accept all the defaults as you click through the installer.

When the setup is complete, you will find your Python folder inside the Applications folder on your computer.

Now that you have Python installed, we can write our first program!

For Windows

Search for the Command application in the Start Menu or the Windows search box.

Enter python in all lowercase. You should see a message from Python like this:

$ python
Python 3.11.0 (main, Jan  9 2023, 15:58:34) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Here, you can play around with the Python interpreter. Note the three greater-than signs (>>>) are called the prompt. This means Python is waiting for our input.

For Mac

Python is already installed on most Macs, but it's probably an outdated version and not the one you want.

To see which version of Python is installed, open the Terminal application from the Applications folder and enter python3. You should then see the Python version and prompt.

Note: A terminal (or shell) is a command-line interface on a computer where you can type commands to the operating system and receive text output in return.

Your First Python Program

Now that you have Python installed, we can start writing some Python code. Enter the following line into the prompt:

print("Hello World")

You should see the "Hello World" line printed out!

To close the terminal session, press Ctrl-Z and Enter, or type exit() and press Enter.

But wait a second... you may be thinking, "How is this useful?". That's a totally fair question. Let's look at a more realistic example.

Copy and paste the following code into your Python shell, and press Enter. Or try clicking the Run button in this sandbox:

You should now see today's date and time printed out!

Note: In Python, the hash symbol (#) indicates a comment. Anything following a hash char in your code is ignored by Python; they are just there for human readers. You can use comments to write notes in plain English within your programs.

Comments can also be used to experiment with your code. If you want to temporarily "turn off" some code so that it doesn't run but don't want to delete it entirely, start the line with a #. This will convert the line into a comment, which Python will ignore when running the program.

Let's take a look at the code starting with this line:

from datetime import datetime

This imports a module called datetime from somewhere, which we will use to display the current date and time. For now, you probably have no idea where it's coming from, and that's completely okay.

The next line is a bit different:

today = datetime.now()

It kind of sounds like this line is saying: today is equal to a datetime now, maybe? Or more like today is datetime now.

That's pretty much exactly what this line does:

1. It creates a new variable name called "today".
2. It assigns the result of datetime.now() to the variable today.

print(today)

Finally, this prints out the value of the variable today.

Now, try to change the name today in the last line to something else, like now, and run the code again.

What to do if you see an error

You may see an error this time, and the output is just the Python interpreter telling us where the problem is.

Traceback (most recent call last):
  File "/var/shared-data/code/main.py", line 6, in <module>
    print(now)
          ^^^
NameError: name 'now' is not defined. Did you mean: 'pow'?

A NameError happens when we either forget to set a variable's name before using it or we made a spelling error when entering the variable name.

The output also reports that this error happened in line 6 of the file main.py. This can help us spot the error quickly.

If we change both of these lines to use the same name "now", the code works again. Try copying these lines into your shell now:

now = datetime.now()
print(now)

Many programming mistakes are simple typos of one character in one line of code. If you spend a long time trying to find one of these mistakes, you're not alone. Many skilled and experienced programmers spend hours looking for these types of small mistakes. Try to laugh about it and move on, knowing that this will happen a lot as you learn to code.

Variables and Operations

Variables are a way of storing things in a program. They're essentially a way to name a value, and they point to the given value in memory. Their value can later be retrieved and used in expressions.

To create a variable, we simply type the name on the left side of an equal sign (=). The right side must be an expression. This can be a primitive value or a more complicated expression.

For example, we can assign the result of a calculation to a variable:

weeks = 52
days = 5 * 52
vacation = 5 * 2 + 4
workdays = days - vacation
workweeks = workdays / 7
print(workdays)

To find the value of a variable, we can use print followed by the name of the variable in parentheses. Try copying the code above to your shell, and then press Enter.

We can also tell Python to change the variable so that it contains a new value and also use more than one name or variable for the same value:

days = 7
week = days
print(week)

Here we're changing the value of the variable days to 7 and then, we're saying that we want the variable week to refer to the current value of the variable days.

However, week isn't a very useful name for a variable because it doesn't tell us much about what the variable is used for. Let's call our variable days_in_week instead.

days_in_week = 7
print(days_in_week)

This makes it clear that we're talking about the number of days in a week. Variable names can be made up of letters, numbers, and underscores (_), but they can't start with a number.

You can use any lowercase letters for variable names, but the name cannot contain a space, so we use underscores to separate words.

You should not use uppercase letters in variable names because they have a special meaning that we'll discuss another time. Python is a case-sensitive language, which means that today is not the same as Today.

The order of operations

You can do all the basic math operations in Python using the corresponding operators, as you saw in that example.

We use parentheses in programming languages to control the order of operations. An operation is anything that uses an operator. Multiplication and division have a higher order than addition and subtraction, so they’re performed first. In other words, if you enter an equation in Python, multiplication or division is performed before addition or subtraction.

For example, in this expression, the numbers 2 and 10 are multiplied first, and then the number 5 is added to the product:

5 + 2 * 10

The result is 25.

We can change the order of operations (and the result) by adding the parentheses around the numbers:

(5 + 2) * 10

The result of this expression is now 70 because the parentheses tell Python to add 5 to 2, and then multiply the result by 10.

Parentheses can be nested, which means that there can be parentheses inside parentheses, like this:

((5 + 2) * 10) / 7

In this case, Python evaluates the innermost parentheses first, then the outer ones, and then the final division. So it adds 5 to 2, then multiplies the result by 10, and then divides that by 7.

Saving Your Python Programs

As we saw, we can write Python code directly into the shell window... but Python programs wouldn’t be very useful if you needed to rewrite them every time you wanted to use them.

Sure, it might be fine for short programs, but a large program could have millions of lines of code!

Luckily, we can save our programs for future use.

To create and save a new program, enter idle in the Windows search box, and select IDLE (Python 3.11.1 64-bit). In a Mac, open your Applications folder and open the Python 3.11 folder to find IDLE.

IDLE (Integrated Development & Learning Environment) is a Python file editor which lets you write programs in Python.

When you open IDLE, you will see the Python shell. Click File › New Window. An empty window will appear.

Paste the code from the previous example into the window, then click File ▸ Save. Enter hello.py as the file name, and save it to your desktop. Then, you can click Run › Run Module to run it.

An alternative to IDLE for Python is PyCharm, which is loved by many Python programmers. You can download the free PyCharm Community Edition from their website.

PyCharm has a lot of extra features that make writing Python code really easy. You can use it to create new programs and to run saved programs directly from within PyCharm.

Run your Python program from the Terminal

We can also run our programs from the Terminal by taking these steps:

For Windows

1. Open the Command Prompt from the Start Menu or Windows search.
2. Enter cd Desktop in the command window to change the directory to the folder containing our program.
3. Type python hello.py to run the program.

For Mac and Linux

1. Open the Terminal app from Applications.
2. Enter  cd ~/Desktop in your terminal to change the directory to the folder containing the program.
3. Type python3 hello.py to run it.

With some luck, you should see the result of your Python program printed out!

Recap

You just learned how to set up Python and how to start playing with it! Also:

• How to save your programs
• How to create variables to name values
• How to use parentheses to control the order of operations.

Now would be a great time for a celebratory dance! 🎉

We can make Python run faster on multi-core CPUs, but one does not simply make use of multiple cores. We must design our programs carefully to leverage all available cores in a given machine effectively.

In this guide, you will learn the difference between regular (sequential) and parallel processing. You'll also learn about different techniques for running multiple processes simultaneously, effectively side-stepping the limitations of the Python interpreter.

By the end of the guide, you'll be able to write programs that run in parallel on all processors available and also as background processes.

Table of Contents

1. Introduction
2. CPU-Bound vs. IO-Bound Tasks
3. Race Conditions and Pitfalls
4. Processes
5. Process Pool
6. ProcessPoolExecutor
7. Coordination Between Processes
8. Sharing State Between Processes
9. When to use Processes
10. Why Not Use Threads
11. Why Not Use Asyncio
12. Recap: Python Multiprocessing Best Practices

Introduction

Since the release of multi-core CPUs all the way back in 2001, we have seen processors grow faster than ever in their performance and capabilities. Modern smartphones and laptop computers today include as many as 8-core chips, and we can find high-end CPUs with up to 64 cores at affordable prices.

Most programs run in a single processor thread by default, so multi-core processors can run many of those programs simultaneously, using their different cores and threads. The OS coordinates with the CPU to run the programs on various cores as needed.

We can leverage multiple cores in Python and run parts of our programs simultaneously. But to be successful, we need to use the right strategy to run our tasks efficiently, depending on the task. This is known as multiprocessing and is the key to unlocking the full performance of our CPU.

Let’s look at examples of particular tasks that may benefit from multiprocessing.

CPU-bound vs. IO-bound tasks

Some multiprocessing applications are complex systems that must perform multiple tasks, often in the background while something more important takes place.

Databases use multiple programs to write new data while remaining able to answer read queries. Since database writes need to be saved, we need to wait for the write to disk to finish so we know the update was completed. This IO-bound task can only run as fast as a disk write, regardless of the kind of CPU.

Similarly, reading from a large filesystem also depends on disk read speeds. Just like reading from a network server depends on the speed of our network connection.

On the other hand, running computations in memory is a CPU-bound task, as the processor executes those instructions. For example: rendering graphics, compressing data, scientific computations, and video streaming all use processing to perform their job.

Race conditions and pitfalls

In a regular (aka sequential) program, the instructions are executed in order from start to finish. Every operation must finish before we can proceed with the next. The order of operations is straightforward, linear, and easy to reason about.

In a multiprocessing scenario, we can quickly run into trouble. Consider a program that saves its result in a file. If we executed this program in multiple cores concurrently, we can’t tell which version of the program will be the last to write to this file.

This is called a race condition. Generally, if we have some shared resource that multiple processes need access to, we will need a way to coordinate between them, so they don’t conflict. After all, the correctness of our program should not depend on accidents of timing.

One way of preventing race conditions is using locks (or mutexes), which grant exclusive access to the resource to a single process for a limited time, preventing others from accessing the resource until it’s released. Something to keep in mind is that we must ensure the holder releases any acquired locks. Otherwise, other processes may wait indefinitely for access, and our program may never finish. This is called a deadlock.

Processes

What is a process

A process is an instance of a program being executed by the CPU. A program is just a series of instructions typically stored on a disk file, and a process is the execution of that program that happens once the OS loads that program in memory and hands it off to the CPU.

Any given process can be executed concurrently by one or many threads if the CPU supports it.

In a multi-core processor, each core executes a single task at a time. However, the cores can switch between tasks, often while a process is waiting for input/output operation or when the OS decides it should switch tasks based on the task scheduler criteria, for example, after some time interval or depending on the number of processes waiting.

We can run multiple processes associated with the same program; for example, opening up several instances of the same program often results in more than one process being executed. However, we can only run as many processes simultaneously as the number of (logical) cores in our processor. The rest will queue up and will have to wait their turn.

Python can detect the number of logical cores in our system like this:

import os
# get the number of logical cpu cores
cores = os.cpu_count()

Running subprocesses

So how can we run multiple processes associated with the same program? One example is running multiple instances of the same program, but we can also launch instances of different programs at will, using different methods.

One of these methods is called "spawning."

Actually, spawning new processes is very straightforward and can be done by constructing a new Process instance and then calling start().

# create a process
process = multiprocessing.Process(target=task, args=(arg1, arg2))
# run the new process
process.start()

The start() function will return right away, and the OS will execute the target function in a separate process as soon as possible.

Naming a process

We can give our processes a name for easier debugging and logging. By default, they will get a name assigned by the Python interpreter, but we can make their names more user-friendly!

Let's see this in practice:

from multiprocessing import current_process, Process

def task():
    process = current_process()
    print("subprocess is", process.name)

# entry point
if __name__ == "__main__":
    process = current_process()
    print("parent is", process.name)

    # initialize Process with name
    other = multiprocessing.Process(target=task, name="database")
    other.start()

    other = Process(target=task)
    # assign a new name to the process
    other.name = "other"
    other.start()

Waiting for processes

When we start a new process in Python, the start() function does not block the parent process. That means it creates a new process and returns immediately, regardless of how long the subprocess may take to run. We can explicitly wait for another process by calling the join() method.

This will block our main program and wait for the child process to terminate. Sometimes we don't want to wait for the process forever, though, and we may need to terminate it manually.

Here's an example:

def task():
    # block for a second
    time.sleep(1)
    print('in subprocess')

# entry point
if __name__ == '__main__':
    process = Process(target=task)
    process.start()
    # wait for the process to finish
    print('Waiting for the process...')
    process.join()
    ...
    # or terminate the process
    process.terminate()

Fork vs. Spawn

When starting a new process, the Python interpreter will hand off the process to the OS for execution. We should take into account how this handoff happens.

Spawning a new process means we will create a new Python interpreter instance from scratch to execute our new Python process. Then, Python will need to create copies of our process functions and references from the parent program to execute the new process concurrently. There is a performance hit during this initialization step.

On the other hand, forking is faster because it does not copy from the parent process; instead, the necessary resources are inherited and copied on write. This makes it trickier to coordinate resources between the parent and child process.

Forking is the default start method on Linux implementations, while macOS and Windows use spawn by default. However, because forking can lead to varying behavior, recent Python docs mention that the fork start method should be considered unsafe as it can lead to subprocesses crashing.

We can tell Python always to use spawn with this snippet:

multiprocessing.set_start_method('spawn')

Security concerns when starting processes

When starting a subprocess on a typical Unix system, fork() is called, and the host process is copied. Modern Unix systems have a copy-on-write implementation of that syscall, meaning that the operation does not copy all the memory of the host process over. Forking is (almost) immediately followed by calling execve() in the child process — which transforms the calling process into a new process.

While Python hides that away from us, we may need to call some OS functions directly and manage the creation of processes through the OS. For example, by using system() calls or invoking shell commands. The problem here is that system() calls expect arguments as strings and will use the shell to evaluate these arguments. This can be exploited by using malicious arguments to execute arbitrary commands or even get full shell access and hack the host system.

A secure way to interact with the OS is by using the subprocess module and setting the keyword shell=False.

subprocess.run(["ls", "-l", "/dev/null"], shell=False, capture_output=True)

Running processes in the background

We may need to start a process to execute sporadic, periodic, or long-running tasks in the background. These are called "daemon" processes, like the demons or spirits from ancient Greek conjured to perform tasks for us in the background.

A process may be configured as a daemon or not, and most processes in concurrent programming, including the main parent process, are non-daemon processes (not background processes) by default.

Some use cases for daemon projects can be:

• Background logging to a file or database
• Retrieving or querying data in the background
• Storing data in a disk or database

The main parent process will terminate once all non-daemon processes finish, even if daemon processes are still running. Therefore, the code it executes must be robust to arbitrary termination, such as the flushing and closing of external resources like streams and files that may not be closed correctly.

Daemon processes do not need to be waited for or manually terminated, as they will be closed when all non-daemon processes terminate. Even so, it is probably good practice to explicitly join all the processes we start.

from multiprocessing import Process, current_process

def task():
    print("daemon started")
    p = current_process()
    print(f"daemon={p.daemon}")

if __name__ == "__main__":
    daemon = Process(target=task, daemon=True)
    # once a process is started, this cannot be changed
    daemon.daemon = True
    daemon.start()
    daemon.join()

Process Pool

Now, let's switch gears to learn how to leverage all the cores in our system, and we can do this in very few lines of code using multiprocessing Pools.

Starting a processing pool

This is a great way to improve the parallelism of our program by just asking Python to distribute a task to all cores in the system. Let’s see an example:

from os import getpid
from multiprocessing import Pool, get_context

def squared(x):
    print(f"started process {getpid()}")
    return x**2

if __name__ == '__main__':
    with get_context("spawn").Pool() as pool:
        result = pool.map(squared, [1, 3, 5, 8, 13])
        print(result)

As we can see, the square function runs in separate processes. Pay special attention to our with statement, where we ask the OS to use the spawn method. This can save some of the headaches discussed before in Fork vs Spawn and should be standard practice.

Spawn from pools

Also, note that while set_start_method('spawn') changes things globally, it’s better to use  get_context("spawn") in our pools, avoiding any side effects.

ProcessPoolExecutor

The ProcessPoolExecutor class is a convenient wrapper for managing multiprocessing pools. It also provides better resource utilization by allowing processes to share resources and tasks.

When using a ProcessPoolExecutor we have more control over how many processes run at once, when new processes are created, and how they are created.

# entry point of the program
if __name__ == '__main__':
    # create a process pool with 4 workers
    executor = ProcessPoolExecutor(max_workers=4)

Note that if max_workers is not set, the pool will default to the number of cpu cores on the machine.

The ProcessPoolExecutor class defines three methods used to control our process pool: submit(), map(), and shutdown().

• submit(): Schedule a function to be executed and returns a Future object.
• map(): Apply a function to an iterable of elements and returns an iterator.
• shutdown(): Shut down the executor.

Running tasks with map()

The map() function will return an iterator immediately. This iterable can be used to access the results from the target task function as they are available in the order that the tasks were submitted.

# perform all tasks in parallel
# returns an iterator and does not block
for result in executor.map(my_task, my_items):
    print(result)

Submitting tasks with submit()

This method takes a function to call and all arguments to the function, then returns a Future object immediately.

The Future object is a promise to return the results from the task (if any) and provides a way to determine whether or not a specific task has been completed.

# submit a task with arguments and get a future object
future = executor.submit(my_task, arg1, arg2) # does not block
# get the result from a future
result = future.result() # blocks

The ProcessPoolExecutor is started when the class is created and must be shut down explicitly by calling shutdown(), which will release any resources.

# shutdown the process pool
executor.shutdown() # blocks
executor.shutdown(wait=False) # doesn't block
# cancel all queued tasks
executor.shutdown(cancel_futures=True) # blocks

Note that if we don't close the process pool, it will be closed automatically when we exit the main thread. If we forget to close the pool and there are still tasks, the main process will only exit once all the queued tasks have been executed.

We can also shut down the ProcessPoolExecutor automatically, using a context manager for the pool. This is actually the preferred way:

# create a process pool
with ProcessPoolExecutor(max_workers=10) as executor:
    # submit tasks and get results
    results = executor.map(my_task, my_items)
    ...
# pool is shutdown at this point

Coordination Between Processes

A mutual exclusion lock or mutex lock is used to protect critical sections of code from concurrent execution and to prevent a race condition.

A mutex lock can be used to ensure that only a single process executes a critical section of code at a time. All other processes trying to execute the same code must wait until the currently executing process is finished with the critical section and releases the lock.

We have a mutual exclusion (mutex) lock in the multiprocessing.Lock class.

Each process must attempt to acquire the lock at the beginning of the critical section. If the lock has not been obtained, then a process will acquire it, and other processes must wait until the process that acquired the lock releases it.

# create a lock
lock = multiprocessing.Lock()
# acquire the lock
lock.acquire()
...
# release the lock
lock.release()

Only one process can have the lock at any time. If a process does not release an acquired lock, it cannot be acquired again.

The process attempting to acquire the lock will block until the lock is acquired, such as if another process currently holds the lock and then releases it.

We can also use the lock with a context manager, allowing the lock to be released automatically once the critical section is completed.

# create a lock
lock = multiprocessing.Lock()

# acquire the lock
with lock:
    # critical section only the holder of the lock can execute

A limitation of a (non-reentrant) mutex lock is that if a process has acquired the lock, it cannot acquire it again. In fact, this situation will result in a deadlock as it will wait forever for the lock to be released so that it can be acquired, but it holds the lock and will not release it.

A reentrant lock will allow a process to acquire the same lock again if it has already been acquired.

There are many reasons why a process may need to acquire the same lock more than once. Imagine critical sections spread across different functions, each protected by the same lock. A process may call into one critical section from another critical section.

To create a reentrant lock, we use the multiprocessing.RLock class instead of a regular Lock.

Sharing State Between Processes

When dealing with multiple processes, we typically need to share data and state between processes. We should wrap the shared data using the following classes:

• multiprocessing.Value: manage a shared value.
• multiprocessing.Array: manage an array of shared values.

For example, a shared ctype value can be defined in a parent process, then shared with multiple child processes. All child and parent processes can then safely read and modify the data within the shared value.

This can be useful in many cases, such as:

• A counter shared among multiple processes.
• Returning data from a child process to a parent process.
• Sharing results of computation among processes.

Python also provides a process-safe queue in the multiprocessing.Queue class.

This shared job queue implementation allows queued items to be processed in parallel by multiple workers. This type of queue can store and transfer any pickle-able* object across process boundaries.

We can add items to a queue for later processing using the put() method and retrieve the items in subprocesses by calling its get() method.

We could also use a simple queue to send data back and forth between two processes, called a Pipe.

For example:

# create a pipe
rec, send = multiprocessing.Pipe()

By default, the first connection (rec) can only be used to receive data and the second connection (send) can only be used to send data.

We can make the connection objects bidirectional by setting the duplex argument to the constructor to True.

# create a duplex pipe
conn1, conn2 = multiprocessing.Pipe(duplex=True)

In this case, we can use both connections to send and receive data.

Objects can be shared between processes using the Pipe.

We have a connection.send() to send objects from one process to another.

# send an object
conn2.send('Hello world')

We can then use the connection.recv() method to receive objects in one process sent by another.

# will block until an object is received
object = conn1.recv()

We can check the status of the pipe using the connection.poll() method. This will return a boolean as to whether there is data to be received and read from the pipe.

# returns True if there is data to read from the pipe.
if conn1.poll():
    ...

We can set a timeout with the "timeout" argument. If specified, the call will block until data is available. The function will return if no data is available before the timeout number of seconds has elapsed.

# waits for 5 seconds
if conn1.poll(timeout=5):
    ...

When to use Processes

Python only executes a single thread at a time because the Global Interpreter Lock (GIL) requires each thread to acquire a lock on the interpreter before executing, preventing all other threads from executing simultaneously.

Although we may have tens, hundreds, or even thousands of concurrent threads in our Python application, only one thread may execute in parallel.

Process-based concurrency is not limited in the same way as thread-based concurrency.

Both threads and processes can execute concurrently (out of order), but only python processes can execute in parallel (simultaneously), not Python threads (with some caveats).

This means that if we want our Python code to run on all CPU cores and make the best use of our system hardware, we should use process-based concurrency.

If process-based concurrency offers true parallelism in Python, why not always use processes?

First, only one thread can run at a time within a Python process under most situations, except:

1. When we are using a Python interpreter that does not use a GIL.
2. When we are performing IO-bound tasks that release the GIL.
3. When we are performing CPU-bound tasks that release the GIL.

For example, when we are reading or writing from a file or socket, the GIL is released, allowing multiple threads to run in parallel.

Common examples include:

• Hard disk drive: Reading, writing, appending, renaming, and deleting files, etc.
• Peripherals: mouse, keyboard, screen, printer, serial, camera, etc.
• Internet: Downloading and uploading files, getting a webpage, querying RSS, etc.
• Database: Select, update, delete, etc. SQL queries.
• Email: Send mail, receive mail, query inbox, etc.

Additionally, many CPU-bound tasks that are known to not rely on the state of the Python interpreter will release the GIL, such as C-code in third-party libraries called from Python.

For example:

• When performing compression operations, e.g. in zlib.
• When calculating cryptographic hashes, e.g. in hashlib.
• When performing encoding/decoding images and video, e.g, in OpenCV.
• When performing matrix operations, e.g. in NumPy and SciPy.
• And many more.

Second, processes and process-based concurrency also have limitations compared to threads.

For example:

1. We may have thousands of threads, but perhaps only tens of processes.
2. Threads are small and fast, but processes are large and slow to create and start.
3. Threads can share data quickly and directly, whereas processes must pickle and transmit data to each other.

Why Not Use Threads

Threads and processes are quite different, and choosing one over the other must be quite intentional.

A Python program is a process that has a main thread. You can create many additional threads in a Python process. You can also fork or spawn many Python processes, each of which will have one main thread and may spawn additional threads.

Threads are lightweight and can share memory (data and variables) within a process, whereas processes are heavyweight and require more overhead and impose more limits on sharing memory (data and variables).

Because of the GIL, only a single thread can execute at a time, making them a good candidate for workloads where we need to switch between tasks that spend a good amount of time waiting, like IO-bound tasks.

CPU-bound tasks won't benefit from multiple threads and will actually slow down because of the overhead of switching between multiple tasks without allowing them a fair share of CPU resources.

You can learn more about when to use threads in our guide:

Threads in Python - The Multithreading Guide

In this guide, you will learn about threads, their use cases, and how to use them to run multiple parts of your Python programs concurrently. When used correctly, they can make your code run faster and more efficiently with very little effort.

Hacker CultureAdrian Cruz

Why Not Use Asyncio

Asyncio is a newer alternative to using a threading.Thread, but it's probably not a good replacement for the multiprocessing.Process class.

Asyncio is designed to support large numbers of IO operations, perhaps thousands to tens of thousands, all within a single Thread.

When using the multiprocessing.Process class, we're typically executing CPU-bound tasks, which is not something that benefits from the asyncio paradigm, and so it's not a good alternative for CPU-intensive workloads.

You can learn more about when to use asyncio in python in our guide:

Python asyncio - A Guide to Asynchronous Programming

In this guide, we’ll introduce asynchronous programming in Python and review fundamental concepts like how to define, create and run asynchronous functions, coroutines, and some common use cases and best practices.

Hacker CultureAdrian Cruz

Recap: Python Multiprocessing Best Practices

Now that we know processes work and how to use them, let’s review some best practices to consider when bringing multiprocessing into our Python programs.

Best practices to keep in mind when using processes in Python:

1. Always Check for Main Module Before Starting Processes
2. Use Context Managers
3. Use Timeouts When Waiting
4. Use Locks to Prevent Race Conditions
5. Use Values, Pipes, and Queues for Shared State

These best practices will help you avoid many concurrency pitfalls like race conditions and deadlocks.

By now, you should know how multiprocessing works in Python in great detail and how to best utilize processes in your programs.

References

• multiprocessing module
• Python Multiprocessing: The Complete Guide
• Why your multiprocessing Pool is stuck
• Effective Python: 90 Specific Ways to Write Better Python
• Python in a Nutshell
• Python Cookbook

Did you find this guide helpful?
If so, I'd love to know. Please share what you learned in the comments below.

How are you using Processes in your programs?
I’d love to hear about it, too. Please let me know in the comments.

Do you have any questions?
Leave your question in a comment below, and I will answer it with my best advice.

Ubuntu is one of the most popular distros out there. It's pretty fast by default, but it can also slow down because of bloat and poor management. Today we are discussing some optimizations we can make to get back to fast ubuntu. Let's go!

The first step is to remove any bloatware that came preinstalled that could be updated or just unnecessary.

sudo apt autoremove --purge

Boot optimizations

Another thing holding you back is that your system starts unnecessary apps and services during startup. Ubuntu has the Startup Applications program that lets you remove any apps that may be slow to start. To show the complete list of startup apps and services, run this:

sudo sed -i 's/NoDisplay=true/NoDisplay=false/g' /etc/xdg/autostart/*.desktop

Now launch Startup Applications and disable any application by unchecking its box. Make sure to read the description so you disable only things you're sure you don't require!

Swap only when you need to

Chances are you have plenty of RAM but you still feel like your machine is not fast enough. It could be that your OS is still using disk swap, instead of the full memory available. Ubuntu will actually use disk space when the RAM is 40% full by default. That can be very low for machines with tons of RAM, so let's increase that.

sudo sysctl vm.swappiness=10

This will tell the kernel to only start using swap space when the RAM is 90% full. To make this change permanent, edit the /etc/sysctl.conf file in a text editor. Change the value of vm.swappiness or add it at the end of the file, like this:

#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#
...
#kernel.sysrq=438
vm.swappiness=10

apt-fast

The default package manager in ubuntu is apt – Advanced Package Tool. But you can also use any other package manager you like. I will be sticking with apt because I like being able to copy/paste commands that I read on forums, blogs, etc. and I want it to just work ^TM. However, one issue I have with apt is that it downloads packages sequentially, one after the other. So much for advanced, huh? We can do better by fetching packages in parallel, using multiple connections. That's exactly what apt-fast is for.

sudo add-apt-repository ppa:apt-fast/stable
sudo apt update
sudo apt install apt-fast
echo "alias apt='apt-fast'" >> ~/.bash_aliases
echo "alias apt-get='apt-fast'" >> ~/.bash_aliases

Liquorix patch

Liquorix is a custom kernel for Debian distros that uses a modern (MuQSS) process and disk scheduler for better performance and throughput. It also disables a lot of verbose logging, by turning off debug flags. Let's install that too:

sudo add-apt-repository ppa:damentz/liquorix
sudo apt-get update
sudo apt install linux-image-liquorix-amd64 linux-headers-liquorix-amd64

Preload apps

Preload is a program that runs in the background and analyzes the apps that you use to determine the ones you use frequently and fetches their binaries and dependencies in memory beforehand. This makes it so when you launch your apps, they start instantly from memory. The best thing about it is that you only have to install it once and it will make your apps start faster after that. You will be amazed!

sudo apt install preload

TLP

If your computer it's overheating, it will become slower to prevent damaging the core components of the CPU. This is called thermal throttling and it can impact the performance of your system. There are some programs that can help diagnose this problem, and even control how your CPU runs.

sudo apt-get install tlp tlp-rdw
sudo tlp start
# TLP started in AC mode (auto).
sudo apt install indicator-cpufreq

After your next restart, you will see an indicator next to the date/time (up on the status bar) from which you can select what speed your CPU should run at.

Alternative distros

Of course, it would be nice to start out with a distro that enabled all these performance optimizations from the start. That means less tweaking and more fun. One such distro is Linux Lite, which is a modified version of ubuntu (using an Xfce desktop) with some nice tools for performance optimization, like Lite Tweaks and preload. This is a seriously snappy distro that can run on any machine with at least 1Gb of RAM.

And that's all I have. Don't forget to restart your system after tweaking the performance settings, and enjoy a faster Ubuntu!

If you have other tips on how to improve Ubuntu performance, I'd like to hear them! Please let us know in the comments below.

Python is a batteries-included language. This means it ships with everything you need to get things working without installing and compiling external packages. That includes many data types and collections to help you manage the data your program needs. Today we're taking a deep dive into the built-in types that are available in python and how to use them.

Booleans

Boolean values are simple True or False values to represent logical truth values. These values can also be combined in boolean algebraic expressions to represent logical operations like ==,  and, or, and not . Here's an example of how we use booleans.

>>> is_true = True
>>> is_false = False
>>> is_true == is_true
True

>>> is_true and is_false
False

>>> is_true and not is_false
True

>>> 1 == True and 0 == False
True

Numeric

Numeric types are used to represent numbers or amounts. Python has a few types of numbers, the main ones being int and float.

Here's an example:

>>> one = 1  		 # an int value
>>> one_float = 1.0  # a float value
>>> total = 1 + 2    # an int value = 3
>>> big = 1234567890123456789012345678901234567890  # also an int

Note in the last example that python has arbitrary integer precision, which means it can hold really large numbers!

Decimal

Even though we can represent floating-point numbers using float types, floats have some limitations in how they're represented internally and can lead to precision issues. Check this out:

>>> .1 + .1 + .1
0.30000000000000004 # WTF
>>> .1 + .1 + .1 == 0.3
False

Hmmm, so even though we're just adding 0.1 three times, somehow we end up with a value that's larger than 0.3. It is only larger by a very small fraction, but still. If we were adding up transactions in a bank account, that could result in giving a user more money than their original amount. So to avoid precision issues, we should use decimals to guarantee correctness.

>>> from decimal import Decimal

>>> dec = Decimal('0.1')
>>> dec + dec + dec
Decimal('0.3')

>>> dec + dec + dec == Decimal('0.3')
True

Decimal can be initialized from integers, strings, floats, or tuples. But if you initialize them from float values, you will run into the same issues as using floats, so just use strings!

Strings

Strings are for representing text data. Like names, addresses, and so on. You can initialize a string by surrounding its value with quotes, either double, single, or even triple quotes, like so:

>>> a_string = "Hello World!"
>>> escaped = "I contain \"double quotes\""

>>> long_string = """ This is a long string \
... with multiple lines """

>>> singles = 'Can also use single quotes'

The de-facto standard is to use double quotes for string values and triple quotes for comments that span multiple lines.

Strings are a sequence that is indexed starting at zero. You can access any element in the string by index, so in our previous example, a_string[0] is the letter H, while a_string[1] is the letter e , and so on. Strings also have a lot of useful methods.

Tuples

Sometimes we have two pieces of data that are related to each other and should be stored together. For example, a first name and last name. Or a point represented by x, y or even more than two pieces, like a 3d point such as x, y, z. In python, we can represent that using a Tuple, which is a container type. To create tuples, simply wrap your values with parenthesis.

>>> point = (0, 3)
>>> point3d = (0, 3, 12)
>>> person = ('Jon', 'Snow')
>>> mixed = ('Jon', 23, 1992)  # tuple with mixed types

Those are all tuples, and they come in very handy. Something to note is that you cannot add more elements to a tuple after creating it. But you can access the elements in a tuple by index, just like strings. So in the example above, point[0]  would evaluate to 0, while point[1] gives 3. Another cool trick is that you can get multiple elements out of a tuple by assigning multiple variables. This is also known as unpacking a tuple:

>>> x, y, z = point3d  # assigning multiple values
>>> x
0

>>> y
1

>>> z
12

Lists

Lists are similar to tuples in that they can hold any number of values, but it is mutable after created. So we can add more to it later. To initialize a list, we add square brackets around its values, like this:

>>> letters = ["a", "b", "c"]
>>> letters
['a', 'b', 'c']

>>> letters.append("x")  # add a single value
>>> letters
['a', 'b', 'c', 'x']

>>> letters.extend(["y", "z"])  # add multiple values
>>> letters
['a', 'b', 'c', 'x', 'y', 'z']

>>> letters[0]
'a'
>>> letters[-1]
'z'

Note how we can also get elements from the list by index. In the last example, we are accessing letters[-1], which means the first value from the end. We can also remove elements from the list or merge values from two lists using list methods.

Sets

Another container type similar to the list is the set. The difference is that a list can contain duplicated values, while a set only contains unique values. Adding a value to a set that is already in the set does nothing.

Another crucial difference is that lists maintain the order of elements as they are added, while sets do not guarantee any particular order.

Sets are pretty useful when we only care about unique values.

For example:

>>> vowels = {"a", "e", "i", "o", "u"}  # initializes a set
>>> vowels
{'e', 'u', 'a', 'o', 'i'}  # note the order changed

>>> vowels.add('a')
{'e', 'u', 'a', 'o', 'i'}

>>> vowels.remove('u')
>>> vowels
{'e', 'a', 'o', 'i'}

We can also perform set operations to join, merge, or even exclude elements from the set. See some more examples here.

Dictionaries

As the name implies, dictionaries or dict in python land, are useful for mappings or data which is identified by some key and has some value associated with that key. An example of this is a real-life dictionary, in which you have the word (key) and the definition as the value. A phone book (remember those?) is another kind of dictionary where the keys are the phone numbers and the values are the information associated with the owner of the phone number, such as name, address, etc.

Let's see some dict examples:

>>> MLB_team = {
...     'Colorado' : 'Rockies',
...     'Boston'   : 'Red Sox',
...     'Minnesota': 'Twins',
...     'Milwaukee': 'Brewers',
...     'Seattle'  : 'Mariners'
... }

# alternatively, use the dict() constructor
>>> MLB_team = dict(
...     Colorado='Rockies',
...     Boston='Red Sox',
...     Minnesota='Twins',
...     Milwaukee='Brewers',
...     Seattle='Mariners'
... )

>>> type(MLB_team)
<class 'dict'>

>>> MLB_team
{'Colorado': 'Rockies', 'Boston': 'Red Sox', 'Minnesota': 'Twins',
'Milwaukee': 'Brewers', 'Seattle': 'Mariners'}

Unlike lists and tuples, dict items are accessed by their key instead of by index.

>>> MLB_team['Minnesota']
'Twins'

>>> MLB_team['Colorado']
'Rockies'

>>> MLB_team['Toronto']
Traceback (most recent call last):
  File "<pyshell#19>", line 1, in <module>
    MLB_team['Toronto']
KeyError: 'Toronto'

If you try to access a key that is not in the dictionary, python throws a KeyError. You can also add and update keys by setting a new value for the key:

>>> MLB_team['Kansas City'] = 'Royals'
>>> MLB_team
{'Colorado': 'Rockies', 'Boston': 'Red Sox', 'Minnesota': 'Twins',
'Milwaukee': 'Brewers', 'Seattle': 'Mariners', 'Kansas City': 'Royals'}

Any given key can only appear in a dictionary once. If you try to set a duplicated key, it will just update the value associated with it. For more info on dictionaries, check out the list of supported operations.

Iterators

We have discussed a few different types of containers, like strings, lists, sets, and dictionaries. In python, we can iterate over the elements in a container using iterators.

The iterator is a special type that represents the position in a sequence and has only one operation, which is next(). If you've worked with database cursors, iterators are a similar concept.

Here's an example:

>>> my_list = [1, 2, 3]
>>> iterator = iter(my_list)

>>> next(iterator)
1
>>> next(iterator)
2
>>> next(iterator)
3

Note that now the iterator has returned the last element in the list. Watch what happens if we call next() again:

>>> next(iterator)
StopIteration

Python raises a StopIteration exception to signal the end of the sequence. You should handle this using a try/except block.

Immutables

Sometimes you want your data to be read-only. Meaning you want to prevent changing their values once it's initialized. This is where immutable types come in handy.

In Python, strings and tuples are immutable. Once you initialize them, you can't change them, you can only assign a new value to them.

Lists, sets, and dictionaries are mutable after the fact since you can add, remove and update their values. However, python also has a frozenset which behaves like an immutable set and supports the same operations as regular sets.

There is no built-in frozendict but we might get one soon. If you do need an immutable dictionary, you can install the frozendict package with pip.  

Functions

Functions are a special data type that only runs when we ask it to. Functions are created using a function definition, and they only have one operation, which is to call it. They can receive data to operate on as arguments. A function can also return some data back to the caller.

>>> def add(num, other):
...     return num + other
...

>>> add(1, 1)
2

>>> add(0.1, 0.2)
0.30000000000000004

>>> type(add)
<class 'function'>

Here we defined a function called add which receives two parameters and returns the sum of the parameters. We then called the function with different values and got back a result, which is the sum. Finally, we use another built-in function to inspect the type of our function, which is (duh) a function. See more built-in functions here.

Classes

Classes are how we make objects in python. An object is just an abstraction for a data type composed of multiple pieces of data. For example, consider a dog. We might know many things about a dog, like its name, kind, age, and weight. We can represent all these pieces of data about a dog using strings or tuples, but a better approach is to use an object to hold all the information about our dog:

class Dog:
    def __init__(self, name, kind, age):
        self.name = name
        self.kind = kind
        self.age = age

Here we have defined a class that contains information about our dog. Now we can represent multiple types of dogs by creating instances.

>>> leo = Dog("Leo", "german shepherd", 1)
>>> neeko = Dog("Neeko", "pitbull", 2)

This creates two new Dog instances — one for a one-year-old German shepherd dog named Leo and one for a two-year-old pitbull named Neeko. We can also access and update dog attributes by name:

>>> leo.age = 1.5
>>> leo.age
1.5

Classes are special too, because we can add functionality by defining methods that we can then call in the instances. Methods are just a special type of function which is associated with a specific class.

class Dog:
    def __init__(self, name, kind, age):
        self.name = name
        self.kind = kind
        self.age = age
        
    def bark(self):
    	return "Woof, woof!"
        
    def sit(self):
    	return "Okay I'm sitting."

Now we can call these methods on our instances:

>>> leo.bark()
'Woof, woof!'
>>> leo.sit()
"Okay I'm sitting."

None

Finally, the last type in our list is None. This is just a special python type to represent empty values. It supports no other operations, and functions that don't have a return statement will implicitly return None when they're done.

Productivity is been talked about endlessly. It seems every day there’s a new app or a new book or a new something out there that will change your life and revolutionize your productivity to take it to the next level.

But will it, really? Or will it give you yet another thing to do before you realize it’s not for you? How much will you have to pay for that course or that coach or that subscription? How much will that money be worth when you realize the only system that works is one that works for you?

And really, the only person who can make it work is you, by creating a system designed for your specific needs. So let’s get to work on creating that system.

We can get started with a simple technique called making a list.

All you need is a good list to make a plan. You can quote me on that.

A Todo List

You need a To-do list to keep track of things you must do. Like commitments or assignments or anything that needs to be done. You could take this further and categorize your list by due dates (this week, next week, next month, etc.) Everything that must be done goes on this list.

A Watch List

The watch list is everything you need to remember or wait for someone else to get back to you, or follow up on.

A Later List

Everything else goes into another list of things that you want to do when you have the time or wish you could do sometime later.

~

If something doesn’t fit into any of these lists, then it must not be that important since you won’t even want to do it later, so just ignore it.

Every day you will look at your To-do list and pick out 3 to 5 things that you want to do that day. Then you will try as much as possible to get those things done. If you do them, it will be a good day, and you will feel accomplished.

If you get into this habit, you will realize how many days went by before when you didn’t get a single important thing done. But every time you do something, you get to cross it off your list and get that rush of endorphins, making you feel all proud and productive.

A cool thing about this list is that if you find yourself procrastinating on a particular to-do on your list, you can pick something else to do in the meantime!

As John Perry, a philosophy professor from Stanford would put it:

The list of tasks one has in mind will be ordered by importance. Tasks that seem most urgent and important are on top. But there are also worthwhile tasks to perform lower down on the list. Doing these tasks becomes a way of not doing the things higher up on the list. With this sort of appropriate task structure, the procrastinator becomes a useful citizen. Indeed, the procrastinator can even acquire, as I have, a reputation for getting a lot done.

Only check your email twice a day

Doing email only twice a day will make you more productive the rest of the day. That’s because you’ll be distracted less often and will have more time to focus on the tasks at hand. Anyone who absolutely cannot wait until the end of the day will try to call you, text you, or reach you via smoke signals. Emails kill your flow and take away your ability to focus on something long enough to be productive.

You can take this further by categorizing your email with labels and filters, so all messages go into an Action, Pending, or Review folder. Everything else you should archive as soon as the conversation is done.

Turn off your notifications

Same as with email, being constantly notified of your friend's latest adventures/meals is distracting and interrupts your flow. It’s best just to turn off all the notifications and check your phone when you have time.

Don’t answer the phone

Unless you’re waiting for an important call, just keep your phone on silent and let all calls go to voice mail. If it’s important, they will leave a message. Then when you have time, go through your voicemail and return the important calls.

Don’t ever answer calls from random phone numbers. There are a lot of robo-callers, spam marketers, and even scammers out there. Don’t fall for that.

I only ever answer calls from known numbers that I have in my contacts. That’s it.

Start the day right

No matter what you do or what time you wake up, just make sure you start the day right. That might be by having a nice breakfast or a light workout, or even some meditation. Whatever suits you, it’s important to have a chance to peacefully collect your thoughts and prepare mentally and emotionally for the day ahead.

It will also allow you to review your To-do list and gather anything you need for your next task!

And that’s it! Do you have a different productivity system already in place? Let me know in the comments below!

~

PS. Other people recommend making a To-Don't list to identify busy work that isn’t helping you meet your goals and help you focus on the work that is.

In a previous episode, we set up Python for development like a Boss.

In this one, I’ll show you how to set up your local machine for nodejs development so you can get started with node today. Let’s go!

Requirements

• Ubuntu 20.x / WSL
• Internet connection

nvm (node version manager)

Similar to Pyenv for python, nvm is a program that allows us to manage multiple versions of node and makes switching between versions in different projects as seamless as it can be.

Let's install nvm following their instructions on Github. Make sure you check Github to install the latest version if you're reading this from the future! Open up your terminal and run this command to install nvm.

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash

This will download nvm, install it locally, and add some lines to your .bashrc (or .zshrc) and will tell you something like "restart your terminal to apply". So let's restart the terminal.

To verify that nvm is installed correctly, run this

command -v nvm

You should see that it prints nvm if it's working. If not, follow the troubleshooting guide.

Now, we can install nodejs using nvm. Let's get the latest version by running:

nvm install node # "node" is an alias for the latest version

You should see some output ending like this

...
Now using node v16.1.0 (npm v7.11.2)
Creating default alias: default -> node (-> v16.1.0)

Installing other versions

The whole point of using nvm is that we can manage multiple versions of node. This is important because you might have some apps that require a specific version, while you can also test new features when a new version is released without messing with the previous versions required for other projects. In the previous step, we downloaded the latest version of node but it's recommended to stick with the Long Term Support versions for stability and long-term updates. This is what you want for apps you plan to maintain for a long time.

nvm install --lts

This installed v14.17.0 for me today. Since this is the latest LTS version, I will also go ahead and tell my system to default to using this for every new shell.

 nvm alias default 14
 # default -> 14 (-> v14.17.0)

If I now run nvm ls, it will show me all my installed versions along with other available versions.

Now, my system is set up to use the LTS version that I just installed, but for newer projects where I'd like to try more recent features added to nodejs, I want to use the latest version available. We can achieve this by creating a .nvmrc file in the project root directory (or any parent directory) to tell nvm which version it should use for this particular project.

mkdir ~/src/nodeproj
cd ~/src/nodeproj
echo "node" > .nvmrc # to default to the latest version

Now if I just run nvm use, it'll look for that .nvmrc and switch to whatever it says in there that it should use.

If you'd like, you can also set up nvm to automatically switch versions based on the local .nvmrc file (without having to enter nvm use manually). To do that, follow the official guide here.

Bonus: yarn package manager

We are now done with the node part. That was easy, right? But node wouldn't be what it is without its robust ecosystem of packages. The node maintainers support the official npm package manager which ships with every version of node. However, more recently Facebook introduced yarn to optimize package resolution, offline support, and reproducible builds. And it's very fast! On average, about 4x faster than npm.

To use yarn, you just need to run one command to install it via npm (oh the irony...)

npm install -g yarn

You can use yarn to install packages just like you would use npm. There are only minor differences in the syntax of the commands like yarn add as opposed to npm install and a few others. To help you get used to it, here's a handy yarn cheatsheet for ya!

Yarn cheatsheet

The one-page guide to Yarn: usage, examples, links, snippets, and more.

Devhints.io cheatsheets

And that's all you need to start playing with nodejs today. Go write some code!

Have other nodejs tips for getting started? Please let us know in the comments below!