6.11 Combining MPI and Threads

In certain cases, it may be advantageous to combine MPI (distributed-memory) and threads (shared-memory) parallelization. FFTW supports this, with certain caveats. For example, if you have a cluster of 4-processor shared-memory nodes, you may want to use threads within the nodes and MPI between the nodes, instead of MPI for all parallelization.

In particular, it is possible to seamlessly combine the MPI FFTW routines with the multi-threaded FFTW routines (see Multi-threaded FFTW). However, some care must be taken in the initialization code, which should look something like this:

     int threads_ok;
     
     int main(int argc, char **argv)
     {
         int provided;
         MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
         threads_ok = provided >= MPI_THREAD_FUNNELED;
     
         if (threads_ok) threads_ok = fftw_init_threads();
         fftw_mpi_init();
     
         ...
         if (threads_ok) fftw_plan_with_nthreads(...);
         ...
     
         MPI_Finalize();
     }

First, note that instead of calling MPI_Init, you should call MPI_Init_threads, which is the initialization routine defined by the MPI-2 standard to indicate to MPI that your program will be multithreaded. We pass MPI_THREAD_FUNNELED, which indicates that we will only call MPI routines from the main thread. (FFTW will launch additional threads internally, but the extra threads will not call MPI code.) (You may also pass MPI_THREAD_SERIALIZED or MPI_THREAD_MULTIPLE, which requests additional multithreading support from the MPI implementation, but this is not required by FFTW.) The provided parameter returns what level of threads support is actually supported by your MPI implementation; this must be at least MPI_THREAD_FUNNELED if you want to call the FFTW threads routines, so we define a global variable threads_ok to record this. You should only call fftw_init_threads or fftw_plan_with_nthreads if threads_ok is true. For more information on thread safety in MPI, see the MPI and Threads section of the MPI-2 standard.

Second, we must call fftw_init_threads before fftw_mpi_init. This is critical for technical reasons having to do with how FFTW initializes its list of algorithms.

Then, if you call fftw_plan_with_nthreads(N), every MPI process will launch (up to) N threads to parallelize its transforms.

For example, in the hypothetical cluster of 4-processor nodes, you might wish to launch only a single MPI process per node, and then call fftw_plan_with_nthreads(4) on each process to use all processors in the nodes.

This may or may not be faster than simply using as many MPI processes as you have processors, however. On the one hand, using threads within a node eliminates the need for explicit message passing within the node. On the other hand, FFTW's transpose routines are not multi-threaded, and this means that the communications that do take place will not benefit from parallelization within the node. Moreover, many MPI implementations already have optimizations to exploit shared memory when it is available, so adding the multithreaded FFTW on top of this may be superfluous.