numpy - Quick evaluation of many functions at same point in Python -


problem: need fast way in python3 evaluate many (in thousands) functions @ same argument. in sense, kind of need opposite of numpy's broadcasting allows evaluate one function @ multiple points.

my solution: @ moment store functions in list , iterate on list classic loop evaluate functions individually. slow.

examples, ideas , links packages welcome.

edit: people have asked functions like: 1. computational in nature. no i/o. 2. involve usual algebraic operations +, -, *, / , ** , indicator function. no trigonometric functions or other special functions.

if functions io bound (meaning spend of time waiting io operation complete), using multiple threads may fair solution.

if functions cpu bound (meaning spend of time doing actual computational work), multiple threads not you, unless using python implementation not have global interpreter lock.

what can here, use multiple python processes. easiest solution being multiprocessing module. here example:

#!/usr/bin/env python3 multiprocessing import pool functools import reduce  def a(x):         return reduce(lambda memo, i: memo + i, x) def b(x):         return reduce(lambda memo, i: memo - i, x) def c(x):         return reduce(lambda memo, i: memo + i**2, x)  my_funcs = [a, b, c]  #create process pool of 4 worker processes pool = pool(4)  async_results = [] f in my_funcs:         #seconds parameter apply_async should tuple of parameters pass function         async_results.append(pool.apply_async(f, (range(1, 1000000),))) results = list(map(lambda async_result: async_result.get(), async_results)) print(results) 

this method allows utilize cpu power in parallel: pick pool size matches number of cpus in environment. limitation of approach functions must pickleable.


Comments

Popular posts from this blog

magento2 - Magento 2 admin grid add filter to collection -

Android volley - avoid multiple requests of the same kind to the server? -

Combining PHP Registration and Login into one class with multiple functions in one PHP file -