Parallel#

class Parallel(func, iterarg=None, iterkwargs=None, args=None, kwargs=None, ncpus=None, maxcpu=None, maxmem=None, interval=None, parallelizer=None, serial=False, progress=False, callback=None, globaldict=None, label=None, die=True, **func_kwargs)[source]#

Bases: object

Parallelization manager

For arguments and usage documentation, see sc.parallelize(). Briefly, this class validates input arguments, sets the number of CPUs, creates a process (or thread) pool, starts the jobs running, retrieves the results from each job, and processes them into outputs.

Useful methods:: reset(): reset the Parallel object to its initial pre-run state run_async(): the method that actually executes the parallelization (NB, used with every method, not only async ones) monitor(): monitor the progress of an asynchronous run finalize(): get the results from each job and process it run(): shortcut to calling run_async() followed by finalize()
Useful attributes and properties:: running (bool): whether or not the jobs are running ready (bool): whether or not the jobs are ready status (str): a string description of the current state (not run, running, or done) jobs (list): a list of jobs to run or being run (empty prior to run) results (list): list of all results (the output from the jobs; empty prior to run) success (list): whether each job completed successfully (true/false) exceptions (list): if not, store the exceptions that were raised times (dict): timing information on when the jobs were started, when they finished, and how long each job took

Example:

import sciris as sc

def slowfunc(i):
    sc.randsleep(seed=i)
    return i**2

# Standard usage
P = sc.Parallel(slowfunc, iterarg=range(10), parallelizer='multiprocess-async')
P.run_async()
P.monitor()
P.finalize()
print(P.times)

New in version 3.0.0.

New in version 3.1.0: “globaldict” argument

Attributes

`ready`
`running`
`status`

Methods

init()[source]#: Perform all remaining initialization steps; this can safely be called after object creation

reset()[source]#: Reset to the pre-run state

disp()[source]#: Display the full representation of the object

set_defaults()[source]#: Define defaults for parallelization

validate_args()[source]#: Validate iterarg and iterkwargs

set_ncpus()[source]#: Configure number of CPUs

set_method()[source]#: Choose which method to use for parallelization

make_pool()[source]#: Make the pool and map function

make_argslist()[source]#: Construct argument list

run_async()[source]#: Choose how to run in parallel, and do it

monitor(interval=0.1, **kwargs)[source]#: Monitor progress – only usable with async

finalize(get_results=True, close_pool=True, process_results=True)[source]#: Get results from the jobs and close the pool

process_results()[source]#: Parse the returned results dict into separate lists

run()[source]#: Actually run the parallelization