Parallel Programming

More than parallel For loops.

A Functional Style for Process Generation

A general way to parallelize many kinds of computation is to replace a function g occurring in a functional operation by Composition[Queue,g]. This new operation will cause all instances of calls of the function g to be queued for parallel evaluation. The result of this composition is a process id (pid) that will appear inside the structure constructed with the outer computation where g occurred. To put back the results of the computation of g, wrap your whole expression in Wait. Wait will replace any pid inside its expression by the result returned by the corresponding process.

Parallel Map

A parallel version of Map is easy to develop with this paradigm. The sequential Map wraps a function f around all elements in a list.

Simply use Composition[Queue,f] instead of f to schedule all mappings for parallel execution. The result is a list of pids.

Finally, simply wait for the processes to finish. Every pid will be replaced with the result of its associated process.

A Parallel Generalized Inner Product

Say, you want a generalized inner product,

where d is the common last dimension of a and first dimension of b.
(Think of p as plus and t as times.)

You can use Composition[Queue,p] in place of p to cause all calculcations of p[...] to be queued for concurrent execution.
The result is a tensor of process IDs.

Now, simply wait for all processes in this expression.

This scheduling is not optimal in terms of communication. One better possibility is to parallelize only along the dimensions of a and send the tensor b once to each processor before scheduling the jobs.

Parallel Tables and Sums

This code generates a 1010 matrix of random numbers, where each row is computed in parallel.

Here is a sum whose elements are computed in parallel. Each element of the sum is a numerical integration.

Here is the corresponding table of parallel exact results. The value of is .

Explicit Scheduling: Gaxpy

Taken verbatim from Golub&van Loan, Matrix Computations, ed., The Johns Hopkins Univ. Press, 1996.

Y=Y+A.X, A an nn matrix, X, Y, n-vectors

Sample (symbolic) matrices and vectors.

Scheduling. Find a number of processors that is a divisor of n.

The workload (number of rows) for each processor.

The client code

Export the code (not the data!).

Start all remote processes.

Document converted by Mathematica of Wolfram Research