代码之家  ›  专栏  ›  技术社区  ›  MTR

Python并行运行函数

  •  -1
  • MTR  · 技术社区  · 2 年前

    有没有办法通过并行运行来加快速度?最长的处理时间是scipy。ndimage。地图坐标。

    import multiprocessing
    import numpy as np
    import scipy.ndimage
    
    pool = multiprocessing.Pool()
    n=6
    x0=350
    y0=350
    r=150
    num=10000
    #z = np.gradient(sensor_dat, axis=1)
    z = np.random.randn(700,700)
    
    def func1(i):
        x1, y1 = x0 + r * np.cos(2 * np.pi * i / n), y0 + r * np.sin(2 * np.pi * i / n)
        x, y = np.linspace(x0, x1, num), np.linspace(y0, y1, num)
        zi = scipy.ndimage.map_coordinates(z, np.vstack((y, x)))
        return zi
    
    [func4(i) for i in range(36)]
    #pool.map(func1,range(36))
    
    

    我从 Is there a simple process-based parallel map for python? 使用 pool.map(func1,range(36)) 但我犯了个错误 Can't pickle <function func1 at 0x0000019408E6F438>: attribute lookup func1 on __main__ failed

    我发现 How to accelerate scipy.map_coordinates for multiple interpolations? 但不要认为这是相关的。ndimage。map_坐标是大部分处理时间,但我认为它不会加快处理速度。

    2 回复  |  直到 2 年前
        1
  •  0
  •   Dudly01    2 年前

    是的,你可以。只需按照说明书中的说明操作即可 multiprocessing documentation 并衡量使用多个工人是否会更快。

    以下是我测试的代码:

    from multiprocessing import Pool
    import numpy as np
    from scipy import ndimage
    from time import time
    
    n=6
    x0=350
    y0=350
    r=150
    num=10000
    #z = np.gradient(sensor_dat, axis=1)
    z = np.random.randn(700,700)
    
    def f(i):
        x1, y1 = x0 + r * np.cos(2 * np.pi * i / n), y0 + r * np.sin(2 * np.pi * i / n)
        x, y = np.linspace(x0, x1, num), np.linspace(y0, y1, num)
        zi = ndimage.map_coordinates(z, np.vstack((y, x)))
        return zi
    
    if __name__ == '__main__':
        begin = time()
        [f(i) for i in range(36)]
        end = time()
        print('Single worker took {:.3f} secs'.format(end - begin))
    
        begin = time()
        with Pool() as p:
            p.map(f, list(range(36)))
        end = time()
        print('Parallel workers took {:.3f} secs'.format(end - begin))
    

    这将在我的机器上产生以下输出:

    Single worker took 0.793 secs
    Parallel workers took 0.217 secs