Databases
StackOverflow2013
Postsdbo
POQuestions about torque checkpoint MPI jobs with BLCR
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POQuestions about torque checkpoint MPI jobs with BLCR
    primarykey
    data
    text
    <p>We're trying to use torque to checkpoint MPI jobs, but it seems that torque can only handle jobs running on a single node. I checked the code and found that when using qhold to checkpoint a job, qhold sends a PBS_BATCH_HoldJob request to pbs server, then pbs server relays this request to master host, and then master host checkpoints the job processes running on itself with BLCR, but not send the request to its sister nodes, so it seems that MPI jobs can not be checkpointable in torque. </p> <p>Another problem, after checkpoint succeeds(as reported by qhold), torque sends a signal 15 to the process in master host to kill the process, then torque would copy the checkpoint file to pbs_server and remove all the files locally. When using qrls to restart this job,the scheduler would allocate new nodes for this job, and copy the checkpoint file to the new nodes and then restart the job through the checkpoint file, then the problem comes:</p> <ol> <li><p>Assume torque can checkpoint the processes of MPI jobs in every nodes, and usually our job uses a huge chunk of memory, and therefore the checkpoint file is very large, but the pbs server doesn't have a disk large enough to contain the checkpoint files.</p></li> <li><p>In our environment, before the MPI job starts, we pull some large meta data from another cluster directly to the nodes allocated for the MPI job for computing, and after checkpoint/restart, the job processes may resume in some different nodes, and the meta data might be missing.</p></li> </ol> <p>If there's someone who can tell me how you do checkpoint for MPI jobs, and if my question can be answered and it's need to modify torque code, I also like to do that.</p> <p>Thanks.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USzero323
    1. USlevin li
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POQuestions about torque checkpoint MPI jobs with BLCR
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POQuestions about torque checkpoint MPI jobs with BLCR
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    1. COI can't answer your Torque question, but I'd like to mention, that you also need to have you MPI library compiled with BLCR support, otherwise no distributed checkpointing would occur.
      singulars
      1. POQuestions about torque checkpoint MPI jobs with BLCR
      1. USHristo Iliev
    2. COThanks, I've just found that only using resource manager with BLCR to checkpoint MPI jobs is not the correct method, maybe I should try some tools like ompi-checkpoint which provides distributed checkpointing.
      singulars
      1. POQuestions about torque checkpoint MPI jobs with BLCR
      1. USlevin li
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload