DistArray API Reference¶
distarray Package¶
error Module¶
Define error classes.
- exception distarray.error.ContextError¶
- exception distarray.error.DistArrayError¶
Bases: exceptions.Exception
- exception distarray.error.InvalidCommSizeError¶
- exception distarray.error.InvalidRankError¶
- exception distarray.error.MPICommError¶
- exception distarray.error.MPIDistArrayError¶
metadata_utils Module¶
- exception distarray.metadata_utils.GridShapeError¶
Bases: exceptions.Exception
- exception distarray.metadata_utils.InvalidGridShapeError¶
Bases: exceptions.Exception
- distarray.metadata_utils.distribute_block_indices(dd)¶
Fill in start and stop in dim dict dd.
- distarray.metadata_utils.distribute_cyclic_indices(dd)¶
Fill in start in dim dict dd.
- distarray.metadata_utils.distribute_indices(dd)¶
Fill in index related keys in dim dict dd.
- distarray.metadata_utils.make_grid_shape(shape, dist, comm_size)¶
Generate a grid_shape from shape tuple and dist tuple.
Does not assume that dim_data has proc_grid_size set for each dimension.
Attempts to allocate processes optimally for distributed dimensions.
Parameters: - shape (tuple of int) – The global shape of the array.
- dist (tuple of str) – dist_type character per dimension.
- comm_size (int) – Total number of processes to distribute.
Returns: dist_grid_shape
Return type: tuple of int
Raises: GridShapeError – if not possible to distribute comm_size processes over number of dimensions.
- distarray.metadata_utils.normalize_dim_dict(dd)¶
Fill out some degenerate dim_dicts.
- distarray.metadata_utils.normalize_dist(dist, ndim)¶
Return a tuple containing dist-type for each dimension.
Parameters: dist (str, list, tuple, or dict) – Returns: Contains string distribution type for each dim. Return type: tuple of str Examples
>>> normalize_dist({0: 'b', 3: 'c'}, 4) ('b', 'n', 'n', 'c')
- distarray.metadata_utils.normalize_grid_shape(grid_shape, ndims, dist, comm_size)¶
Adds 1s to grid_shape so it has ndims dimensions. Validates grid_shape tuple against the dist tuple and comm_size.
- distarray.metadata_utils.normalize_reduction_axes(axes, ndim)¶
- distarray.metadata_utils.positivify(index, size)¶
testing Module¶
Functions used for tests.
- class distarray.testing.CommNullPasser¶
Bases: type
Metaclass.
Applies the comm_null_passes decorator to every method on a generated class.
- class distarray.testing.ContextTestCase(methodName='runTest')¶
Bases: unittest.case.TestCase
Base test class for test cases that use a Context.
Overload the ntargets class attribute to change the default number of engines required. A cls.context object will be created with targets=range(cls.ntargets). Tests will be skipped if there are too few targets.
- ntargets int or 'any', default=4¶
If an int, indicates how many engines are required for this test to run. If the string ‘any’, indicates that any number of engines may be used with this test.
- ntargets = 4
- classmethod setUpClass()¶
- classmethod tearDownClass()¶
- class distarray.testing.MpiTestCase(methodName='runTest')¶
Bases: unittest.case.TestCase
Base test class for MPI test cases.
Overload the comm_size class attribute to change the default number of processes required.
- comm_size int, default=4¶
Indicates how many MPI processes are required for this test to run. If fewer than comm_size are available, the test will be skipped.
- comm_size = 4
- classmethod setUpClass()¶
- classmethod tearDownClass()¶
- distarray.testing.assert_localarrays_allclose(l0, l1, check_dtype=False, rtol=1e-07, atol=0)¶
Call np.testing.assert_allclose on l0 and l1.
Also, check that LocalArray properties are equal.
- distarray.testing.assert_localarrays_equal(l0, l1, check_dtype=False)¶
Call np.testing.assert_equal on l0 and l1.
Also, check that LocalArray properties are equal.
- distarray.testing.check_targets(required, available)¶
If available < required, raise a SkipTest with a nice error message.
- distarray.testing.comm_null_passes(fn)¶
Decorator. If self.comm is COMM_NULL, pass.
This allows our tests to pass on processes that have nothing to do.
- distarray.testing.import_or_skip(name)¶
Try importing name, raise SkipTest on failure.
Parameters: name (str) – Module name to try to import. Returns: module – Module object imported by importlib. Return type: module object Raises: unittest.SkipTest – If the attempted import raises an ImportError. Examples
>>> h5py = import_or_skip('h5py') >>> h5py.get_config() <h5py.h5.H5PYConfig at 0x103dd5a78>
- distarray.testing.raise_typeerror(fn)¶
Decorator for protocol validator functions.
These functions return (success, err_msg), but sometimes we would rather have an exception.
- distarray.testing.temp_filepath(extension='')¶
Return a randomly generated filename.
This filename is appended to the directory path returned by tempfile.gettempdir() and has extension appended to it.
utils Module¶
Utilities.
- distarray.utils.all_equal(iterable)¶
Return True if all elements in iterable are equal.
Also returns True if iterable is empty.
- class distarray.utils.count_round_trips(client)¶
Bases: object
Context manager for counting the number of roundtrips between a IPython client and controller.
- Usage:
>>> with count_round_trips(client) as r: ... send_42_messages()
>>> r.count 42
- update_count()¶
- distarray.utils.distarray_random_getstate()¶
- distarray.utils.distarray_random_setstate(state)¶
- distarray.utils.divisors_minmax(n, dmin, dmax)¶
Find the divisors of n in the interval (dmin,dmax].
- distarray.utils.flatten(seq, to_expand=<function list_or_tuple at 0x7f61abbf4b90>)¶
Flatten a nested sequence.
- distarray.utils.get_from_dotted_name(dotted_name)¶
- distarray.utils.has_exactly_one(iterable)¶
Does iterable have exactly one non-None element?
- distarray.utils.list_or_tuple(seq)¶
- distarray.utils.mirror_sort(seq, ref_seq)¶
Sort seq into the order that ref_seq is in.
>>> mirror_sort(range(5),[1,5,2,4,3]) [0, 4, 1, 3, 2]
- distarray.utils.mult_partitions(n, s)¶
Compute the multiplicative partitions of n of size s
>>> mult_partitions(52,3) [(2, 2, 13)] >>> mult_partitions(52,2) [(2, 26), (4, 13)]
- distarray.utils.mult_partitions_recurs(n, s, pd=0)¶
- distarray.utils.multi_for(iterables)¶
- distarray.utils.remove_elements(to_remove, seq)¶
- distarray.utils.sanitize_indices(indices)¶
Check and possibly sanitize indices.
Parameters: indices (int, slice, or sequence of ints and slices) – If an int or slice is passed in, it is converted to a 1-tuple. Returns: (‘point’, indices) if all indices are ints, or (‘view’, indices) if some indices are slices. Return type: 2-tuple Raises: TypeError – If indices is not all ints or slices.
- distarray.utils.slice_intersection(s1, s2)¶
Compute a slice that represents the intersection of two slices.
Currently only implemented for steps of size 1.
Return type: slice object
- distarray.utils.uid()¶