DistArray API Reference¶

`distarray` Package¶

`client` Module¶

The Distarray data structure.`DistArray` objects are proxies for collections of LocalArray objects. They are meant to roughly emulate NumPy ndarrays.

class distarray.client.DistArray(mdmap, dtype)¶

Bases: object

context¶

dist¶

dtype¶

fill(value)¶

classmethod from_localarrays(key, context)¶: The caller has already created the LocalArray objects. key is their name on the engines. This classmethod creates a DistArray that refers to these LocalArrays.

get_dist_matrix()¶

get_localarrays()¶

Pull the LocalArray objects from the engines.

Returns:	one localarray per process
Return type:	list of localarrays

get_localshapes()¶

get_ndarrays()¶

Pull the local ndarrays from the engines.

Returns:	one ndarray per process
Return type:	list of ndarrays

global_size¶

grid_shape¶

itemsize¶

mean(axis=None, dtype=<type 'float'>, out=None)¶

nbytes¶

ndim¶

shape¶

std(axis=None, dtype=None, out=None)¶

sum(axis=None, dtype=None, out=None)¶

toarray()¶: Returns the distributed array as an ndarray.

tondarray()¶: Returns the distributed array as an ndarray.

var(axis=None, dtype=None, out=None)¶

`client_map` Module¶

ClientMDMap class and auxiliary ClientMap classes.

The ClientMDMap is a multi-dimensional map class that manages the one-dimensional maps for each DistArray dimension. The ClientMDMap class represents the distribution information for a distributed array, independent of the distributed array’s data. ClientMDMaps allow DistArrays to reduce overall communication when indexing and slicing by determining which processes own (or may possibly own) the indices in question. Two DistArray objects can share the same ClientMDMap if they have the exact same distribution.

The one-dimensional ClientMap classes keep track of which process owns which index in that dimension. This class has several subclasses for specific distribution types, including BlockMap, CyclicMap, NoDistMap, and UnstructuredMap.

class distarray.client_map.ClientBlockCyclicMap(size, grid_size, block_size=1)¶

Bases: distarray.client_map.ClientMapBase

dist = 'c'¶

classmethod from_dim_data(dim_data_seq)¶

classmethod from_global_dim_dict(glb_dim_dict)¶

get_dimdicts()¶

owners(idx)¶

class distarray.client_map.ClientBlockMap(size, grid_size)¶

Bases: distarray.client_map.ClientMapBase

dist = 'b'¶

classmethod from_dim_data(dim_data_seq)¶

classmethod from_global_dim_dict(glb_dim_dict)¶

get_dimdicts()¶

owners(idx)¶

class distarray.client_map.ClientMDMap(context, shape, dist, grid_shape=None)¶

Bases: object

Governs the mapping between global indices and process ranks for multi-dimensional objects.

classmethod from_dim_data(context, dim_datas)¶: Creates a ClientMDMap from a sequence of dim_data dictionary tuples from each LocalArray.

classmethod from_global_dim_data(context, glb_dim_data)¶

get_local_dim_datas()¶

owning_ranks(idxs)¶

Returns a list of ranks that may possibly own the location in the idxs tuple.

For many distribution types, the owning rank is precisely known; for others, it is only probably known. When the rank is precisely known, owning_ranks() returns a list of exactly one rank. Otherwise, returns a list of more than one rank.

If the idxs tuple is out of bounds, raises IndexError.

owning_targets(idxs)¶

Like owning_ranks() but returns a list of targets rather than ranks.

Convenience method meant for IPython parallel usage.

class distarray.client_map.ClientMapBase¶

Bases: object

Base class for one-dimensional client-side maps.

Maps keep track of the relevant distribution information for a single dimension of a distributed array. Maps allow distributed arrays to keep track of which process to talk to when indexing and slicing.

Classes that inherit from ClientMapBase must implement the owners() abstractmethod.

owners(idx)¶

Returns a list of process IDs in this dimension that might possibly own idx.

Raises IndexError if idx is out of bounds.

class distarray.client_map.ClientNoDistMap(size, grid_size)¶

Bases: distarray.client_map.ClientMapBase

dist = 'n'¶

classmethod from_dim_data(dim_data_seq)¶

classmethod from_global_dim_dict(glb_dim_dict)¶

get_dimdicts()¶

owners(idx)¶

class distarray.client_map.ClientUnstructuredMap(size, grid_size, indices=None)¶

Bases: distarray.client_map.ClientMapBase

dist = 'u'¶

classmethod from_dim_data(dim_data_seq)¶

classmethod from_global_dim_dict(glb_dim_dict)¶

get_dimdicts()¶

owners(idx)¶

distarray.client_map.client_map_factory(size, dist, grid_size)¶: Returns an instance of the appropriate subclass of ClientMapBase.

distarray.client_map.map_from_dim_datas(dim_datas)¶

Generates a ClientMap instance from a santized sequence of dim_data dictionaries.

Parameters:	dim_datas (sequence of dictionaries) – Each dictionary is a “dimension dictionary” from the distributed array protocol, one per process in this dimension of the process grid. The dimension dictionaries shall all have the same keys and values for global attributes: dist_type, size, proc_grid_size, and perhaps others.
Returns:
Return type:	An instance of a subclass of ClientMapBase.

distarray.client_map.map_from_global_dim_dict(global_dim_dict)¶

`context` Module¶

Context objects contain the information required for distarrays to communicate with localarrays.

class distarray.context.Context(client=None, targets=None)¶

Bases: object

Context objects manage the setup and communication of the worker processes for DistArray objects. A DistArray object has a context, and contexts have an MPI intracommunicator that they use to communicate with worker processes.

Typically there is just one context object that uses all processes, although it is possible to have more than one context with a different selection of engines.

cleanup(close=True, all_other_contexts=False)¶

Delete keys that this context created from all the engines.

If all_other_contexts is False (the default), then this deletes from the engines all the keys from only this context. Otherwise, it deletes all keys from all other contexts.

close()¶

delete_key(key)¶: Delete the specific key from all the engines.

dump_keys(all_other_contexts=False)¶

Return a list of the key names present on the engines.

If all_other_contexts is False (the default), then this returns only the keys for this context. Otherwise, it returns the keys for all other contexts.

The list is a list of tuples (key name, list of targets), and is sorted by key name. This is intended to be convenient and readable to print out.

empty(shape, dtype=<type 'float'>, dist={0: 'b'}, grid_shape=None)¶

from_global_dim_data(global_dim_data, dtype=<type 'float'>)¶

Make a DistArray from global dim_data structures.

Parameters:	global_dim_data (tuple of dict) – A global dimension dictionary per dimension. See following Note section. dtype (numpy dtype, optional) – dtype for underlying arrays
Returns:	result – An empty DistArray of the specified size, dimensionality, and distribution.
Return type:	DistArray

Note

The global_dim_data tuple is a simple, straightforward data structure that allows full control over all aspects of a DistArray’s distribution information. It does not contain any of the array’s data, only the metadata needed to specify how the array is to be distributed. Each dimension of the array is represented by corresponding dictionary in the tuple, one per dimension. All dictionaries have a dist_type key that specifies whether the array is block, cyclic, or unstructured. The other keys in the dictionary are dependent on the dist_type key.

Block

dist_type is 'b'.
bounds is a sequence of integers, at least two elements.

The bounds sequence always starts with 0 and ends with the global size of the array. The other elements indicate the local array global index boundaries, such that successive pairs of elements from bounds indicates the start and stop indices of the corresponding local array.
comm_padding integer, greater than or equal to zero.
boundary_padding integer, greater than or equal to zero.

These integer values indicate the communication or boundary padding, respectively, for the local arrays. Currently only a single value for both boundary_padding and comm_padding is allowed for the entire dimension.

Cyclic

dist_type is 'c'
proc_grid_size integer, greater than or equal to one.

The size of the process grid in this dimension. Equivalent to the number of local arrays in this dimension and determines the number of array sections.

size integer, greater than or equal to zero.

The global size of the array in this dimension.

block_size integer, optional. Greater than or equal to one.

If not present, equivalent to being present with value of one.

Unstructured

dist_type is 'u'
indices sequence of one-dimensional numpy integer arrays or buffers.

The len(indices) is the number of local unstructured arrays in this dimension.

To compute the global size of the array in this dimension, compute sum(len(ii) for ii in indices).

Not-distributed

The 'n' distribution type is a convenience to specify that an array is not distributed along this dimension.

dist_type is 'n'
size integer, greater than or equal to zero.

The global size of the array in this dimension.

fromarray(arr, dist={0: 'b'}, grid_shape=None)¶: Convert an ndarray to a distarray.

fromfunction(function, shape, **kwargs)¶

fromndarray(arr, dist={0: 'b'}, grid_shape=None)¶: Convert an ndarray to a distarray.

load_dnpy(name)¶

Load a distributed array from .dnpy files.

The .dnpy file format is a binary format inspired by NumPy’s .npy format. The header of a particular .dnpy file contains information about which portion of a DistArray is saved in it (using the metadata outlined in the Distributed Array Protocol), and the data portion contains the output of NumPy’s save function for the local array data. See the module docstring for distarray.local.format for full details.

Parameters:	name (str or list of str) – If a str, this is used as the prefix for the filename used by each engine. Each engine will load a file named `<name>_<rank>.dnpy`. If a list of str, each engine will use the name at the index corresponding to its rank. An exception is raised if the length of this list is not the same as the context’s communicator’s size.
Returns:	result – A DistArray encapsulating the file loaded on each engine.
Return type:	DistArray
Raises:	`TypeError` – If name is an iterable whose length is different from the context’s communicator’s size.

`decorators` Module¶

Decorators for defining functions that use DistArrays.

class distarray.decorators.DecoratorBase(fn)¶

Bases: object

Base class for decorators, handles name wrapping and allows the decorator to take an optional kwarg.

determine_context(args, kwargs)¶: Determine a context from a functions arguments.

key_and_push_args(args, kwargs, context=None, da_handler=None)¶

Push a tuple of args and dict of kwargs to the engines. Return a tuple with keys corresponding to args values on the engines. And a dictionary with the same keys and values which are the keys to the input dictionary’s values.

This allows us to use the following interface to execute code on the engines:

>>> def foo(*args, **kwargs):
>>>     args, kwargs = _key_and_push_args(args, kwargs)
>>>     exec_str = "remote_foo(*%s, **%s)"
>>>     exec_str %= (args, kwargs)
>>>     context.execute(exec_str)

process_return_value(context, result_key)¶

Figure out what to return on the Client.

Parameters:	key (string) – Key corresponding to wrapped function’s return value.
Returns:	A DistArray (if locally all values are DistArray), a None (if locally all values are None), or else, pull the result back to the client and return it. If all but one of the pulled values is None, return that non-None value only.
Return type:	Varied

push_fn(context, fn_key, fn)¶: Push function to the engines.

class distarray.decorators.local(fn)¶

Bases: distarray.decorators.DecoratorBase

Decorator to run a function locally on the engines.

class distarray.decorators.vectorize(fn)¶

Bases: distarray.decorators.DecoratorBase

Analogous to numpy.vectorize. Input DistArray’s must all be the same shape, and this will be the shape of the output distarray.

get_local_array(da, arg_keys)¶

`error` Module¶

Define error classes.

exception distarray.error.ContextError¶: Bases: distarray.error.DistArrayError

exception distarray.error.DistArrayError¶: Bases: exceptions.Exception

exception distarray.error.InvalidCommSizeError¶: Bases: distarray.error.MPIDistArrayError

exception distarray.error.InvalidRankError¶: Bases: distarray.error.MPIDistArrayError

exception distarray.error.MPICommError¶: Bases: distarray.error.MPIDistArrayError

exception distarray.error.MPIDistArrayError¶: Bases: distarray.error.DistArrayError

`functions` Module¶

Distributed unfuncs for distributed arrays.

distarray.functions.absolute(a, *args, **kwargs)¶

distarray.functions.arccos(a, *args, **kwargs)¶

distarray.functions.arccosh(a, *args, **kwargs)¶

distarray.functions.arcsin(a, *args, **kwargs)¶

distarray.functions.arcsinh(a, *args, **kwargs)¶

distarray.functions.arctan(a, *args, **kwargs)¶

distarray.functions.arctanh(a, *args, **kwargs)¶

distarray.functions.conjugate(a, *args, **kwargs)¶

distarray.functions.cos(a, *args, **kwargs)¶

distarray.functions.cosh(a, *args, **kwargs)¶

distarray.functions.exp(a, *args, **kwargs)¶

distarray.functions.expm1(a, *args, **kwargs)¶

distarray.functions.invert(a, *args, **kwargs)¶

distarray.functions.log(a, *args, **kwargs)¶

distarray.functions.log10(a, *args, **kwargs)¶

distarray.functions.log1p(a, *args, **kwargs)¶

distarray.functions.negative(a, *args, **kwargs)¶

distarray.functions.reciprocal(a, *args, **kwargs)¶

distarray.functions.rint(a, *args, **kwargs)¶

distarray.functions.sign(a, *args, **kwargs)¶

distarray.functions.sin(a, *args, **kwargs)¶

distarray.functions.sinh(a, *args, **kwargs)¶

distarray.functions.sqrt(a, *args, **kwargs)¶

distarray.functions.square(a, *args, **kwargs)¶

distarray.functions.tan(a, *args, **kwargs)¶

distarray.functions.tanh(a, *args, **kwargs)¶

distarray.functions.add(a, b, *args, **kwargs)¶

distarray.functions.arctan2(a, b, *args, **kwargs)¶

distarray.functions.bitwise_and(a, b, *args, **kwargs)¶

distarray.functions.bitwise_or(a, b, *args, **kwargs)¶

distarray.functions.bitwise_xor(a, b, *args, **kwargs)¶

distarray.functions.divide(a, b, *args, **kwargs)¶

distarray.functions.floor_divide(a, b, *args, **kwargs)¶

distarray.functions.fmod(a, b, *args, **kwargs)¶

distarray.functions.hypot(a, b, *args, **kwargs)¶

distarray.functions.left_shift(a, b, *args, **kwargs)¶

distarray.functions.mod(a, b, *args, **kwargs)¶

distarray.functions.multiply(a, b, *args, **kwargs)¶

distarray.functions.power(a, b, *args, **kwargs)¶

distarray.functions.remainder(a, b, *args, **kwargs)¶

distarray.functions.right_shift(a, b, *args, **kwargs)¶

distarray.functions.subtract(a, b, *args, **kwargs)¶

distarray.functions.true_divide(a, b, *args, **kwargs)¶

distarray.functions.less(a, b, *args, **kwargs)¶

distarray.functions.less_equal(a, b, *args, **kwargs)¶

distarray.functions.equal(a, b, *args, **kwargs)¶

distarray.functions.not_equal(a, b, *args, **kwargs)¶

distarray.functions.greater(a, b, *args, **kwargs)¶

distarray.functions.greater_equal(a, b, *args, **kwargs)¶

`ipython_utils` Module¶

The single IPython entry point.

`metadata_utils` Module¶

exception distarray.metadata_utils.GridShapeError¶: Bases: exceptions.Exception

exception distarray.metadata_utils.InvalidGridShapeError¶: Bases: exceptions.Exception

distarray.metadata_utils.make_grid_shape(shape, dist, comm_size)¶

Generate a grid_shape from shape tuple and dist tuple.

Does not assume that dim_data has proc_grid_size set for each dimension.

Attempts to allocate processes optimally for distributed dimensions.

Parameters:	shape (tuple of int) – The global shape of the array. distdims (sequence of int) – The indices of the distributed dimensions. comm_size (int) – Total number of processes to distribute.
Returns:	dist_grid_shape
Return type:	tuple of int
Raises:	`GridShapeError` – if not possible to distribute comm_size processes over number of dimensions.

distarray.metadata_utils.normalize_dist(dist, ndim)¶

Return a tuple containing dist-type for each dimension.

Parameters:	dist (str, list, tuple, or dict) –
Returns:	Contains string distribution type for each dim.
Return type:	tuple of str

Examples

>>> normalize_dist({0: 'b', 3: 'c'}, 4)
('b', 'n', 'n', 'c')

distarray.metadata_utils.normalize_grid_shape(grid_shape, ndims)¶: Adds 1’s to grid_shape so it has ndims dimensions.

distarray.metadata_utils.validate_grid_shape(grid_shape, dist, comm_size)¶: Validates grid_shape tuple against the dist tuple and comm_size.

`mpiutils` Module¶

Entry point for MPI.

distarray.mpiutils.create_comm_of_size(size=4)¶: Create a subcommunicator of COMM_PRIVATE of given size.

distarray.mpiutils.create_comm_with_list(nodes)¶: Create a subcommunicator of COMM_PRIVATE with a list of ranks.

distarray.mpiutils.mpi_type_for_ndarray(a)¶

`random` Module¶

Emulate numpy.random

class distarray.random.Random(context)¶

Bases: object

normal(loc=0.0, scale=1.0, size=None, dist={0: 'b'}, grid_shape=None)¶

Draw random samples from a normal (Gaussian) distribution.

The probability density function of the normal distribution, first derived by De Moivre and 200 years later by both Gauss and Laplace independently [2], is often called the bell curve because of its characteristic shape (see the example below).

The normal distributions occurs often in nature. For example, it describes the commonly occurring distribution of samples influenced by a large number of tiny, random disturbances, each with its own unique distribution [2].

Parameters:

loc (float) – Mean (“centre”) of the distribution.
scale (float) – Standard deviation (spread or “width”) of the distribution.
size (tuple of ints) – Output shape. If the given shape is, e.g., (m, n, k), then m * n * k samples are drawn.
dist (dist dictionary) – Dictionary describing how to distribute the array along each axis.
grid_shape (tuple) – Tuple describing the processor grid topology.

Notes

The probability density for the Gaussian distribution is

\[p(x) = \frac{1}{\sqrt{ 2 \pi \sigma^2 }} e^{ - \frac{ (x - \mu)^2 } {2 \sigma^2} },\]

where \(\mu\) is the mean and \(\sigma\) the standard deviation. The square of the standard deviation, \(\sigma^2\), is called the variance.

The function has its peak at the mean, and its “spread” increases with the standard deviation (the function reaches 0.607 times its maximum at \(x + \sigma\) and \(x - \sigma\) [2]). This implies that numpy.random.normal is more likely to return samples lying close to the mean, rather than those far away.

References

[1]	Wikipedia, “Normal distribution”, http://en.wikipedia.org/wiki/Normal_distribution

[2]	(1, 2, 3) P. R. Peebles Jr., “Central Limit Theorem” in “Probability, Random Variables and Random Signal Principles”, 4th ed., 2001, pp. 51, 51, 125.

rand(size=(d0, d1, ..., dn))¶

Random values in a given shape.

Create a distarray of the given shape and propagate it with random samples from a uniform distribution over [0, 1).

Parameters:	size (tuple of ints) – The dimensions of the returned array, should all be positive. If no argument is given a single Python float is returned. dist (dist dictionary) – Dictionary describing how to distribute the array along each axis. grid_shape (tuple) – Tuple describing the processor grid topology.
Returns:	out – Random values.
Return type:	distarray, shape `(d0, d1, ..., dn)`

randint(low, high=None, size=None)¶

Return random integers from low (inclusive) to high (exclusive).

Return random integers from the “discrete uniform” distribution in the “half-open” interval [low, high). If high is None (the default), then results are from [0, low).

Parameters:	low (int) – Lowest (signed) integer to be drawn from the distribution (unless `high=None`, in which case this parameter is the highest such integer). high (int, optional) – if provided, one above the largest (signed) integer to be drawn from the distribution (see above for behavior if `high=None`). size (int or tuple of ints, optional) – Output shape. Default is None, in which case a single int is returned. dist (dist dictionary) – Dictionary describing how to distribute the array along each axis. grid_shape (tuple) – Tuple describing the processor grid topology.
Returns:	out – size-shaped distarray of random integers from the appropriate distribution, or a single such random int if size not provided.
Return type:	distarray of ints

randn(size=(d0, d1, ..., dn))¶

Return samples from the “standard normal” distribution.

Parameters:	size (tuple of ints) – The dimensions of the returned array, should be all positive. If no argument is given a single Python float is returned. dist (dist dictionary) – Dictionary describing how to distribute the array along each axis. grid_shape (tuple) – Tuple describing the processor grid topology.
Returns:	out – A `(d0, d1, ..., dn)`-shaped distarray of floating-point samples from the standard normal distribution.
Return type:	distarray

seed(seed=None)¶

Seed the random number generators on each engine.

Parameters:

seed (None, int, or array of integers) –

Base random number seed to use on each engine. If None, then a non-deterministic seed is obtained from the operating system. Otherwise, the seed is used as passed, and the sequence of random numbers will be deterministic.

Each individual engine has its state adjusted so that it is different from each other engine. Thus, each engine will compute a different sequence of random numbers.

`testing` Module¶

Functions used for tests.

class distarray.testing.CommNullPasser¶

Bases: type

Metaclass.

Applies the comm_null_passes decorator to every method on a generated class.

class distarray.testing.MpiTestCase(methodName='runTest')¶

Bases: unittest.case.TestCase

Base test class for MPI test cases.

Overload the comm_size class attribute to change the default (default is 4).

comm_size = 4¶

classmethod setUpClass()¶

classmethod tearDownClass()¶

distarray.testing.comm_null_passes(fn)¶

Decorator. If self.comm is COMM_NULL, pass.

This allows our tests to pass on processes that have nothing to do.

distarray.testing.import_or_skip(name)¶

Try importing name, raise SkipTest on failure.

Parameters:	name (str) – Module name to try to import.
Returns:	module – Module object imported by importlib.
Return type:	module object
Raises:	`unittest.SkipTest` – If the attempted import raises an ImportError.

Examples

>>> h5py = import_or_skip('h5py')
>>> h5py.get_config()
<h5py.h5.H5PYConfig at 0x103dd5a78>

distarray.testing.temp_filepath(extension='')¶

Return a randomly generated filename.

This filename is appended to the directory path returned by tempfile.gettempdir() and has extension appended to it.

`utils` Module¶

Utilities.

distarray.utils.all_equal(iterable)¶

Return True if all elements in iterable are equal.

Also returns True if iterable is empty.

distarray.utils.divisors_minmax(n, dmin, dmax)¶: Find the divisors of n in the interval (dmin,dmax].

distarray.utils.flatten(seq, to_expand=<function list_or_tuple at 0x1d46b18>)¶: Flatten a nested sequence.

distarray.utils.has_exactly_one(iterable)¶: Does iterable have exactly one non-None element?

distarray.utils.list_or_tuple(seq)¶

distarray.utils.mirror_sort(seq, ref_seq)¶

Sort seq into the order that ref_seq is in.

>>> mirror_sort(range(5),[1,5,2,4,3])
[0, 4, 1, 3, 2]

distarray.utils.mult_partitions(n, s)¶

Compute the multiplicative partitions of n of size s

>>> mult_partitions(52,3)
[(2, 2, 13)]
>>> mult_partitions(52,2)
[(2, 26), (4, 13)]

distarray.utils.mult_partitions_recurs(n, s, pd=0)¶

distarray.utils.multi_for(iterables)¶

distarray.utils.sanitize_indices(indices)¶

Check and possibly sanitize indices.

Parameters:	indices (int, slice, or sequence of ints and slices) – If an int or slice is passed in, it is converted to a 1-tuple.
Returns:	(‘point’, indices) if all indices are ints, or (‘view’, indices) if some indices are slices.
Return type:	2-tuple
Raises:	`TypeError` – If indices is not all ints or slices.

distarray.utils.slice_intersection(s1, s2)¶

Compute a slice that represents the intersection of two slices.

Currently only implemented for steps of size 1.

Return type:	slice object

DistArray API Reference¶

`distarray` Package¶

`client` Module¶

`client_map` Module¶

`context` Module¶

`decorators` Module¶

`error` Module¶

`functions` Module¶

`ipython_utils` Module¶

`metadata_utils` Module¶

`mpiutils` Module¶

`random` Module¶

`testing` Module¶

`utils` Module¶

`world` Module¶

Subpackages¶

DistArray API Reference¶

distarray Package¶

client Module¶

client_map Module¶

context Module¶

decorators Module¶

error Module¶

functions Module¶

ipython_utils Module¶

metadata_utils Module¶

mpiutils Module¶

random Module¶

testing Module¶

utils Module¶

world Module¶

Subpackages¶

`distarray` Package¶

`client` Module¶

`client_map` Module¶

`context` Module¶

`decorators` Module¶

`error` Module¶

`functions` Module¶

`ipython_utils` Module¶

`metadata_utils` Module¶

`mpiutils` Module¶

`random` Module¶

`testing` Module¶

`utils` Module¶

`world` Module¶