How to Parallelize a For Loop in Python
Parallelizing a for loop in Python can significantly boost performance for computationally intensive tasks. Here are three popular ways to achieve it:
Using the concurrent.futures Module
The ThreadPoolExecutor or ProcessPoolExecutor from concurrent.futures allows you to easily parallelize tasks:from concurrent.futures import ThreadPoolExecutor
def task(n):
return n * n
nums = [1, 2, 3, 4, 5]
with ThreadPoolExecutor() as executor:
results = list(executor.map(task, nums))
print(results) # Output: [1, 4, 9, 16, 25]
Using the multiprocessing Module
The multiprocessing module is ideal for CPU-bound tasks:from multiprocessing import Pool
def task(n):
return n * n
nums = [1, 2, 3, 4, 5]
with Pool() as pool:
results = pool.map(task, nums)
print(results) # Output: [1, 4, 9, 16, 25]
Using joblib
The joblib library is perfect for simple parallel processing:from joblib import Parallel, delayed
def task(n):
return n * n
nums = [1, 2, 3, 4, 5]
results = Parallel(n_jobs=-1)(delayed(task)(n) for n in nums)
print(results) # Output: [1, 4, 9, 16, 25]
Key Points:
Use ThreadPoolExecutor for I/O-bound tasks.
Use multiprocessing or joblib for CPU-bound tasks.
Ensure your function is stateless and thread-safe for best results.
Each of these approaches lets you efficiently utilize multiple cores, reducing execution time for large datasets or complex calculations.