utils.mpi¶
Module for parallelization using mpi4py.
Allows for easy parallelization in master/slaves mode with one master submitting function or method calls to slaves. Uses mpi4py if available, otherwise processes calls sequentially in one process.
Examples:¶
Save the following lines in demo_mpi.py
and run:
> mpirun -n 10 python demo_mpi.py
Use master/slaves parallelization with the Network class:
from pyunicorn import Network, mpi def master(): net = Network.BarabasiAlbert(n_nodes=1000, n_links_each=10) print(net.newman_betweenness()) mpi.info() mpi.run()
Do a Monte Carlo simulation as master/slaves:
from pyunicorn import Network, mpi def do_one(): net = Network.BarabasiAlbert(n_nodes=100, n_links_each=10) return net.global_clustering() def master(): n = 1000 for i in range(0, n): mpi.submit_call("do_one", ()) s = 0 for i in range(0, n): s += mpi.get_next_result() print(s/n) mpi.info() mpi.run()
Do a parameter scan without communication with a master, and just save the results in files:
import numpy from pyunicorn import Network, mpi offset = 10 n_max = 1000 s = 0 n = mpi.rank + offset while n <= n_max + offset: s += Network.BarabasiAlbert(n_nodes=n).global_clustering() n += mpi.size numpy.save("s"+str(mpi.rank), s)
-
pyunicorn.utils.mpi.
abort
()[source]¶ Abort execution on all MPI nodes immediately.
Can be called by master and slaves.
-
pyunicorn.utils.mpi.
get_next_result
()[source]¶ Return result of next earlier submitted call whose result has not yet been got.
Can only be called by the master.
If the call is not yet finished, waits for it to finish.
Return type: object Returns: return value of call, or None of there are no more calls in the queue.
-
pyunicorn.utils.mpi.
get_result
(id)[source]¶ Return result of earlier submitted call.
Can only be called by the master.
If the call is not yet finished, waits for it to finish. Results should be collected in the same order as calls were submitted. For each slave, the results of calls assigned to that slave must be collected in the same order as those calls were submitted. Can only be called once per call.
Parameters: id (object) – id of an earlier submitted call, as provided to or returned by submit_call(). Return type: object Returns: return value of call.
-
pyunicorn.utils.mpi.
run
(verbose=False)[source]¶ Run in master/slaves mode until master() finishes.
Must be called on all MPI nodes after function master() was defined.
On the master, run() calls master() and returns when master() returns.
On each slave, run() calls slave() if that is defined, or calls serve() otherwise, and returns when slave() returns, or when master() returns on the master, or when master calls terminate().
Parameters: verbose (bool) – whether processing information should be printed.
-
pyunicorn.utils.mpi.
submit_call
(name_to_call, args=(), kwargs={}, module='__main__', time_est=1, id=None, slave=None)[source]¶ Submit a call for parallel execution.
If called by the master and slaves are available, the call is submitted to a slave for asynchronous execution.
If called by a slave or if no slaves are available, the call is instead executed synchronously on this MPI node.
Examples:
Provide ids and time estimate explicitly:
for n in range(0,10): mpi.submit_call("doit", (n,A[n]), id=n, time_est=n**2) for n in range(0,10): result[n] = mpi.get_result(n)
Use generated ids stored in a list:
for n in range(0,10): ids.append(mpi.submit_call("doit", (n,A[n]))) for n in range(0,10): results.append(mpi.get_result(ids.pop()))
Ignore ids altogether:
for n in range(0,10): mpi.submit_call("doit", (n,A[n])) for n in range(0,10): results.append(mpi.get_next_result())
Call a module function and use keyword arguments:
mpi.submit_call("solve", (), {"a":a, "b":b}, module="numpy.linalg")
Call a static class method from a package:
mpi.submit_call("Network._get_histogram", (values, n_bins), module="pyunicorn")
Note that it is module=”pyunicorn” and not module=”pyunicorn.network” here.Parameters: - name_to_call (str) – name of callable object (usually a function or static method of a class) as contained in the namespace specified by module.
- args (tuple) – the positional arguments to provide to the callable object. Tuples of length 1 must be written (arg,). Default: ()
- kwargs (dict) – the keyword arguments to provide to the callable object. Default: {}
- module (str) – optional name of the imported module or submodule in whose namespace the callable object is contained. For objects defined on the script level, this is “__main__”, for objects defined in an imported package, this is the package name. Must be a key of the dictionary sys.modules (check there after import if in doubt). Default: “__main__”
- time_est (float) – estimated relative completion time for this call; used to find a suitable slave. Default: 1
- id (object or None) – unique id for this call. Must be a possible dictionary key. If None, a random id is assigned and returned. Can be re-used after get_result() for this is. Default: None
- slave (int > 0 and < mpi.size, or None) – optional no. of slave to assign the call to. If None, the call is assigned to the slave with the smallest current total time estimate. Default: None
Return object: id of call, to be used in get_result().