DistArray API Reference

distarray Package

error Module

Define error classes.

exception distarray.error.ContextError

Bases: distarray.error.DistArrayError

exception distarray.error.DistArrayError

Bases: exceptions.Exception

exception distarray.error.DistributionError

Bases: distarray.error.DistArrayError

exception distarray.error.InvalidCommSizeError

Bases: distarray.error.MPIDistArrayError

exception distarray.error.InvalidRankError

Bases: distarray.error.MPIDistArrayError

exception distarray.error.MPICommError

Bases: distarray.error.MPIDistArrayError

exception distarray.error.MPIDistArrayError

Bases: distarray.error.DistArrayError

metadata_utils Module

exception distarray.metadata_utils.GridShapeError

Bases: exceptions.Exception

exception distarray.metadata_utils.InvalidGridShapeError

Bases: exceptions.Exception

distarray.metadata_utils.block_cyclic_size(dim_data)
distarray.metadata_utils.block_size(dim_data)
distarray.metadata_utils.c_or_bc_chooser(dim_data)
distarray.metadata_utils.check_grid_shape_postconditions(grid_shape, shape, dist, comm_size)
distarray.metadata_utils.check_grid_shape_preconditions(shape, dist, comm_size)

Verify various distarray parameters are correct before making a grid_shape.

distarray.metadata_utils.cyclic_size(dim_data)
distarray.metadata_utils.distribute_block_indices(dd)

Fill in start and stop in dim dict dd.

distarray.metadata_utils.distribute_cyclic_indices(dd)

Fill in start in dim dict dd.

distarray.metadata_utils.distribute_indices(dd)

Fill in index related keys in dim dict dd.

distarray.metadata_utils.make_grid_shape(shape, dist, comm_size)

Generate a grid_shape from shape tuple and dist tuple.

Does not assume that dim_data has proc_grid_size set for each dimension.

Attempts to allocate processes optimally for distributed dimensions.

Parameters:
  • shape (tuple of int) – The global shape of the array.
  • dist (tuple of str) – dist_type character per dimension.
  • comm_size (int) – Total number of processes to distribute.
Returns:

dist_grid_shape
Return type:

tuple of int
Raises:

GridShapeError – if not possible to distribute comm_size processes over number of dimensions.
distarray.metadata_utils.non_dist_size(dim_data)
distarray.metadata_utils.normalize_dim_dict(dd)

Fill out some degenerate dim_dicts.

distarray.metadata_utils.normalize_dist(dist, ndim)

Return a tuple containing dist-type for each dimension.

Parameters:dist (str, list, tuple, or dict) –
Returns:Contains string distribution type for each dim.
Return type:tuple of str

Examples

>>> normalize_dist({0: 'b', 3: 'c'}, 4)
('b', 'n', 'n', 'c')
distarray.metadata_utils.normalize_grid_shape(grid_shape, shape, dist, comm_size)

Adds 1s to grid_shape so it has ndims dimensions. Validates grid_shape tuple against the dist tuple and comm_size.

distarray.metadata_utils.normalize_reduction_axes(axes, ndim)
distarray.metadata_utils.positivify(index, size)

Check that an index is within bounds and return a positive version.

Parameters:index (Integral or slice) –
Raises:IndexError – for out-of-bounds indices
distarray.metadata_utils.sanitize_indices(indices, ndim=None, shape=None)

Classify and sanitize indices.

  • Wrap naked Integral, slice, or Ellipsis indices into tuples
  • Classify result as ‘value’ or ‘view’
  • Expand Ellipsis objects to slices
  • If the length of the tuple-ized indices is < ndim (and it’s provided), add slice(None)’s to indices until indices is ndim long
  • If shape is provided, call positivify on the indices
Raises:
  • TypeError – If indices is other than Integral, slice or a Sequence of these
  • IndexError – If len(indices) > ndim
Return type:

2-tuple of (str, n-tuple of slices and Integral values)
distarray.metadata_utils.shapes_from_dim_data_per_rank(ddpr)

Given a dim_data_per_rank object, return the shapes of the localarrays. This requires no communication.

distarray.metadata_utils.size_chooser(dist_type)

Get a function from a dist_type.

distarray.metadata_utils.size_from_dim_data(dim_data)

Get a size from a dim_data.

distarray.metadata_utils.tuple_intersection(t0, t1)

Compute intersection of a (start, stop, step) and a (start, stop) tuple.

Assumes all values are positive.

Parameters:
  • t0 (2-tuple or 3-tuple) – Tuple of (start, stop, [step]) representing an index range
  • t1 (2-tuple) – Tuple of (start, stop) representing an index range
Returns:

A tightly bounded interval.
Return type:

3-tuple or None
distarray.metadata_utils.unstructured_size(dim_data)

testing Module

Functions used for tests.

class distarray.testing.CommNullPasser

Bases: type

Metaclass.

Applies the comm_null_passes decorator to every method on a generated class.

class distarray.testing.ContextTestCase(methodName='runTest')

Bases: unittest.case.TestCase

Base test class for test cases that use a Context.

Overload the ntargets class attribute to change the default number of engines required. A cls.context object will be created with targets=range(cls.ntargets). Tests will be skipped if there are too few targets.

ntargets

int or ‘any’, default=4

If an int, indicates how many engines are required for this test to run. If the string ‘any’, indicates that any number of engines may be used with this test.

ntargets = 4
classmethod setUpClass()
classmethod tearDownClass()
class distarray.testing.MpiTestCase(methodName='runTest')

Bases: unittest.case.TestCase

Base test class for MPI test cases.

Overload the comm_size class attribute to change the default number of processes required.

comm_size

int, default=4

Indicates how many MPI processes are required for this test to run. If fewer than comm_size are available, the test will be skipped.

comm_size = 4
classmethod setUpClass()
classmethod tearDownClass()
distarray.testing.assert_localarrays_allclose(l0, l1, check_dtype=False, rtol=1e-07, atol=0)

Call np.testing.assert_allclose on l0 and l1.

Also, check that LocalArray properties are equal.

distarray.testing.assert_localarrays_equal(l0, l1, check_dtype=False)

Call np.testing.assert_equal on l0 and l1.

Also, check that LocalArray properties are equal.

distarray.testing.check_targets(required, available)

If available < required, raise a SkipTest with a nice error message.

distarray.testing.comm_null_passes(fn)

Decorator. If self.comm is COMM_NULL, pass.

This allows our tests to pass on processes that have nothing to do.

distarray.testing.import_or_skip(name)

Try importing name, raise SkipTest on failure.

Parameters:name (str) – Module name to try to import.
Returns:module – Module object imported by importlib.
Return type:module object
Raises:unittest.SkipTest – If the attempted import raises an ImportError.

Examples

>>> h5py = import_or_skip('h5py')
>>> h5py.get_config()
<h5py.h5.H5PYConfig at 0x103dd5a78>
distarray.testing.raise_typeerror(fn)

Decorator for protocol validator functions.

These functions return (success, err_msg), but sometimes we would rather have an exception.

distarray.testing.temp_filepath(extension='')

Return a randomly generated filename.

This filename is appended to the directory path returned by tempfile.gettempdir() and has extension appended to it.

utils Module

Utilities.

distarray.utils.all_equal(iterable)

Return True if all elements in iterable are equal.

Also returns True if iterable is empty.

class distarray.utils.count_round_trips(client)

Bases: object

Context manager for counting the number of roundtrips between a IPython client and controller.

Usage:
>>> with count_round_trips(client) as r:
...     send_42_messages()

>>> r.count
42
update_count()
distarray.utils.distarray_random_getstate()
distarray.utils.distarray_random_setstate(state)
distarray.utils.divisors_minmax(n, dmin, dmax)

Find the divisors of n in the interval (dmin,dmax].

distarray.utils.flatten(seq, to_expand=<function list_or_tuple at 0x7f04edaabcf8>)

Flatten a nested sequence.

distarray.utils.get_from_dotted_name(dotted_name)
distarray.utils.has_exactly_one(iterable)

Does iterable have exactly one non-None element?

distarray.utils.list_or_tuple(seq)
distarray.utils.mirror_sort(seq, ref_seq)

Sort seq into the order that ref_seq is in.

>>> mirror_sort(range(5),[1,5,2,4,3])
[0, 4, 1, 3, 2]
distarray.utils.mult_partitions(n, s)

Compute the multiplicative partitions of n of size s

>>> mult_partitions(52,3)
[(2, 2, 13)]
>>> mult_partitions(52,2)
[(2, 26), (4, 13)]
distarray.utils.mult_partitions_recurs(n, s, pd=0)
distarray.utils.multi_for(iterables)
distarray.utils.remove_elements(to_remove, seq)
distarray.utils.slice_intersection(s1, s2)

Compute a slice that represents the intersection of two slices.

Currently only implemented for steps of size 1.

Return type:slice object
distarray.utils.uid()