StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHalo exchange not working properly in MPI
text
Body
copied!<p>I'm writing some code which does calculations on a large 3D grid, and uses a halo exchange procedure so that it can work using MPI. I'm getting the wrong results from my code, and I'm pretty sure it's because of the halo exchange not working properly.</p> <p>Basically I have a large 3D array, a chunk of which is held on each process. Each process has an array which is 2 elements bigger in each dimension than the chunk of data it is holding - so that we can halo exchange into each face of the array without affecting the data stored in the rest of the array. I have the following code to do the halo exchange communication:</p> <pre><code> MPI_Type_vector(g->ny, g->nx, g->nx, MPI_DOUBLE, &face1); MPI_Type_commit(&face1); MPI_Type_vector(2*g->ny, 1, g->nx, MPI_DOUBLE, &face2); MPI_Type_commit(&face2); MPI_Type_vector(g->nz, g->nx, g->nx * g->ny, MPI_DOUBLE, &face3); MPI_Type_commit(&face3); /* Send to WEST receive from EAST */ MPI_Sendrecv(&(g->data)[current][0][0][0], 1, face1, g->west, tag, &(g->data)[current][0][0][0], 1, face1, g->east, tag, MPI_COMM_WORLD, MPI_STATUS_IGNORE); /* Send to EAST receive from WEST */ MPI_Sendrecv(&(g->data)[current][g->nz-1][0][0], 1, face1, g->east, tag, &(g->data)[current][g->nz-1][0][0], 1, face1, g->west, tag, MPI_COMM_WORLD, MPI_STATUS_IGNORE); /* Send to NORTH receive from SOUTH */ MPI_Sendrecv(&(g->data)[current][0][0][0], 1, face2, g->north, tag, &(g->data)[current][0][0][0], 1, face2, g->south, tag, MPI_COMM_WORLD, MPI_STATUS_IGNORE); /* Send to SOUTH receive from NORTH */ MPI_Sendrecv(&(g->data)[current][0][g->ny-1][0], 1, face2, g->south, tag, &(g->data)[current][0][0][0], 1, face2, g->north, tag, MPI_COMM_WORLD, MPI_STATUS_IGNORE); /* Send to UP receive from DOWN */ MPI_Sendrecv(&(g->data)[current][0][0][0], 1, face3, g->up, tag, &(g->data)[current][0][0][0], 1, face3, g->down, tag, MPI_COMM_WORLD, MPI_STATUS_IGNORE); /* Send to DOWN receive from UP */ MPI_Sendrecv(&(g->data)[current][0][0][g->nx-1], 1, face3, g->down, tag, &(g->data)[current][0][0][g->nx-1], 1, face3, g->up, tag, MPI_COMM_WORLD, MPI_STATUS_IGNORE); </code></pre> <p><code>g->nx</code>, <code>g->ny</code> and <code>g->nz</code> are the sizes of the array chunk that this process is holding, and <code>g->west</code>, <code>g->east</code>, <code>g->north</code>, <code>g->south</code>, <code>g->up</code> and <code>g->down</code> are the ranks of the adjacent processes in each direction, found using the following code:</p> <pre><code>/* Who are my neighbours in each direction? */ MPI_Cart_shift( cart_comm, 2, 1, &g->north, &g->south); MPI_Cart_shift( cart_comm, 1, 1, &g->west, &g->east); MPI_Cart_shift( cart_comm, 0, 1, &g->up, &g->down); </code></pre> <p>The array on each process is defined as:</p> <pre><code>array[2][g->nz][g->ny][g->nx] </code></pre> <p>(It has two copies because I need one to update into each time through my update routine, once I've done the halo exchange).</p> <p>Can anyone tell me if I'm doing the communication correctly? Particularly the defining of the vector types. Will the vector types I've defined in the code extract each face of a 3D array? And do the MPI_Sendrecv calls look right?</p> <p>I'm completely lost as to why my code isn't working, but I'm pretty sure it's communications related.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload