numpy - Quick evaluation of many functions at same point in Python -
problem: need fast way in python3
evaluate many (in thousands) functions @ same argument. in sense, kind of need opposite of numpy's
broadcasting allows evaluate one function @ multiple points.
my solution: @ moment store functions in list , iterate on list classic loop evaluate functions individually. slow.
examples, ideas , links packages welcome.
edit: people have asked functions like: 1. computational in nature. no i/o. 2. involve usual algebraic operations +, -, *, / , ** , indicator function. no trigonometric functions or other special functions.
if functions io bound (meaning spend of time waiting io operation complete), using multiple threads may fair solution.
if functions cpu bound (meaning spend of time doing actual computational work), multiple threads not you, unless using python implementation not have global interpreter lock.
what can here, use multiple python processes. easiest solution being multiprocessing
module. here example:
#!/usr/bin/env python3 multiprocessing import pool functools import reduce def a(x): return reduce(lambda memo, i: memo + i, x) def b(x): return reduce(lambda memo, i: memo - i, x) def c(x): return reduce(lambda memo, i: memo + i**2, x) my_funcs = [a, b, c] #create process pool of 4 worker processes pool = pool(4) async_results = [] f in my_funcs: #seconds parameter apply_async should tuple of parameters pass function async_results.append(pool.apply_async(f, (range(1, 1000000),))) results = list(map(lambda async_result: async_result.get(), async_results)) print(results)
this method allows utilize cpu power in parallel: pick pool size matches number of cpus in environment. limitation of approach functions must pickleable.
Comments
Post a Comment