 |
|
|
| Main Page |
7.0 A More Complex DAG
Each job in a DAGMan DAG must have only one queue command in it, so a DAG with multiple jobs has one submit file per job. Theoretically you can reuse submit files if you are careful and use the $(Cluster) macro, but that is rarely desirable. We will now make a DAG with four nodes in it: a setup node, two nodes that do analysis, and a cleanup node. For now, of course, all of these nodes will do the same thing, but hopefully the principle will be clear.
First, make sure that your submit file has only one queue command in it, as when we first wrote it:
Universe = vanilla Executable = simple Arguments = 4 10 Log = simple.log Output = simple.out Error = simple.error Queue
Now copy these files:
% cp submit job.setup.submit % cp submit job.work1.submit % cp submit job.work2.submit % cp submit job.finalize.submit
Edit the various submit files. Change the output and error entries (but not the log entry) to point to results.NODE.output and results.NODE.error files where NODE is actually the middle word in the submit file (job.NODE.submit). So job.finalize.error would include:
Output = results.finalize.output Error = results.finalize.error
Here is one possible set of settings for the output entries:
% grep -i '^output' job.*.submit job.finalize.submit:Output = results.finalize.output job.setup.submit:Output = results.setup.output job.work1.submit:Output = results.work1.output job.work2.submit:Output = results.work2.output
This is important so that the various nodes don't overwrite each other's output.
Leave the log entries alone. Old versions of DAGMan requires that all nodes output their logs in the same location. Condor will ensure that the different jobs will not overwrite each other's entries in the log. Newer versions of DAGMan lift this requirement, and allow each job to use its own log file -- but you may want to use one common log file anyway because it's convenient to have all of your job status information in a single place.
Log = simple.log
Also change the arguments entries so that the second argument is something unique to each node. This way each of our jobs will calculate something different and we can tell apart their outputs.
For job work1, change the second argument to 11 so that it looks something like:Arguments = 4 11
Now construct your dag in the file called simple.dag:
Job Setup job.setup.submit Job Work1 job.work1.submit Job Work2 job.work2.submit Job Final job.finalize.submit PARENT Setup CHILD Work1 Work2 PARENT Work1 Work2 CHILD Final
Submit your new DAG and monitor it.
% condor_submit_dag simple.dag Checking all your submit files for log file names. This might take a while... Done. ----------------------------------------------------------------------- File for submitting this DAG to Condor : simple.dag.condor.sub Log of DAGMan debugging messages : simple.dag.dagman.out Log of Condor library debug messages : simple.dag.lib.out Log of the life of condor_dagman itself : simple.dag.dagman.log Condor Log file for all jobs of this DAG : /home/users/roy/condor-test/simple.log Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 29. ----------------------------------------------------------------------- % ./watch_condor_q DAGMan runs and submit the first job: -- Submitter: ws-01.gs.unina.it : <192.167.1.21:32783> : ws-01.gs.unina.it ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 29.0 roy 6/2 23:42 0+00:00:03 R 0 9.8 condor_dagman -f - 30.0 |-Setup 6/2 23:42 0+00:00:00 I 0 9.8 simple 4 10 2 jobs; 1 idle, 1 running, 0 held That first job starts -- Submitter: ws-01.gs.unina.it : <192.167.1.21:32783> : ws-01.gs.unina.it ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 29.0 roy 6/2 23:42 0+00:00:53 R 0 9.8 condor_dagman -f - 30.0 |-Setup 6/2 23:42 0+00:00:01 R 0 9.8 simple 4 10 2 jobs; 0 idle, 2 running, 0 held The first job finishes, but DAGMan hasn't reacted yet -- Submitter: ws-01.gs.unina.it : <192.167.1.21:32783> : ws-01.gs.unina.it ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 29.0 roy 6/2 23:42 0+00:01:03 R 0 9.8 condor_dagman -f - 1 jobs; 0 idle, 1 running, 0 held The next two jobs start up. -- Submitter: ws-01.gs.unina.it : <192.167.1.21:32783> : ws-01.gs.unina.it ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 29.0 roy 6/2 23:42 0+00:01:13 R 0 9.8 condor_dagman -f - 31.0 |-Work1 6/2 23:43 0+00:00:00 R 0 9.8 simple 4 11 32.0 |-Work2 6/2 23:43 0+00:00:00 R 0 9.8 simple 4 12 3 jobs; 0 idle, 3 running, 0 held Those jobs have finished, and DAGMan will notice soon -- Submitter: ws-01.gs.unina.it : <192.167.1.21:32783> : ws-01.gs.unina.it ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 29.0 roy 6/2 23:42 0+00:01:23 R 0 9.8 condor_dagman -f - 1 jobs; 0 idle, 1 running, 0 held Now our final node is running -- Submitter: ws-01.gs.unina.it : <192.167.1.21:32783> : ws-01.gs.unina.it ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 29.0 roy 6/2 23:42 0+00:01:33 R 0 9.8 condor_dagman -f - 33.0 |-Final 6/2 23:43 0+00:00:01 R 0 9.8 simple 4 13 2 jobs; 0 idle, 2 running, 0 held All finished! -- Submitter: ws-01.gs.unina.it : <192.167.1.21:32783> : ws-01.gs.unina.it ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held
You can see that the Final node wasn't run until after the Work nodes, which were not run until after the Setup node.
Examine your results:
% tail --lines=500 results.*.output ==> results.finalize.output <== Thinking really hard for 4 seconds... We calculated: 26 ==> results.setup.output <== Thinking really hard for 4 seconds... We calculated: 20 ==> results.work1.output <== Thinking really hard for 4 seconds... We calculated: 22 ==> results.work2.output <== Thinking really hard for 4 seconds... We calculated: 24
Examine your log:
% cat simple.log
000 (030.000.000) 06/02 23:42:13 Job submitted from host: <192.167.1.21:32783>
DAG Node: Setup
...
001 (030.000.000) 06/02 23:43:04 Job executing on host: <192.167.1.21:32782>
...
005 (030.000.000) 06/02 23:43:08 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
000 (031.000.000) 06/02 23:43:18 Job submitted from host: <192.167.1.21:32783>
DAG Node: Work1
...
000 (032.000.000) 06/02 23:43:18 Job submitted from host: <192.167.1.21:32783>
DAG Node: Work2
...
001 (031.000.000) 06/02 23:43:25 Job executing on host: <192.167.1.21:32782>
...
001 (032.000.000) 06/02 23:43:27 Job executing on host: <192.167.1.21:32782>
...
005 (031.000.000) 06/02 23:43:29 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
005 (032.000.000) 06/02 23:43:31 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
000 (033.000.000) 06/02 23:43:38 Job submitted from host: <192.167.1.21:32783>
DAG Node: Final
...
001 (033.000.000) 06/02 23:43:44 Job executing on host: <192.167.1.21:32782>
...
005 (033.000.000) 06/02 23:43:48 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
Examine the DAGMan log:
% cat simple.dag.dagman.out
6/2 23:42:12 ******************************************************
6/2 23:42:12 ** condor_scheduniv_exec.29.0 (CONDOR_DAGMAN) STARTING UP
6/2 23:42:12 ** /opt/condor-6.7.19/bin/condor_dagman
6/2 23:42:12 ** $CondorVersion: 6.7.19 May 10 2006 $
6/2 23:42:12 ** $CondorPlatform: I386-LINUX_RH9 $
6/2 23:42:12 ** PID = 16376
6/2 23:42:12 ** Log last touched time unavailable (No such file or directory)
6/2 23:42:12 ******************************************************
6/2 23:42:12 Using config file: /opt/condor-6.7.19/etc/condor_config
6/2 23:42:12 Using local config files: /opt/condor-6.7.19/local.ws-01/condor_config.local
6/2 23:42:12 DaemonCore: Command Socket at <192.167.1.21:33718>
6/2 23:42:12 DAGMAN_SUBMIT_DELAY setting: 0
6/2 23:42:12 DAGMAN_MAX_SUBMIT_ATTEMPTS setting: 6
6/2 23:42:12 DAGMAN_STARTUP_CYCLE_DETECT setting: 0
6/2 23:42:12 DAGMAN_MAX_SUBMITS_PER_INTERVAL setting: 5
6/2 23:42:12 allow_events (DAGMAN_IGNORE_DUPLICATE_JOB_EXECUTION, DAGMAN_ALLOW_EVENTS) setting: 50
6/2 23:42:12 DAGMAN_RETRY_SUBMIT_FIRST setting: 1
6/2 23:42:12 DAGMAN_RETRY_NODE_FIRST setting: 0
6/2 23:42:12 DAGMAN_MAX_JOBS_IDLE setting: 0
6/2 23:42:12 DAGMAN_MAX_JOBS_SUBMITTED setting: 0
6/2 23:42:12 DAGMAN_MUNGE_NODE_NAMES setting: 1
6/2 23:42:12 DAGMAN_DELETE_OLD_LOGS setting: 1
6/2 23:42:12 argv[0] == "condor_scheduniv_exec.29.0"
6/2 23:42:12 argv[1] == "-Debug"
6/2 23:42:12 argv[2] == "3"
6/2 23:42:12 argv[3] == "-Lockfile"
6/2 23:42:12 argv[4] == "simple.dag.lock"
6/2 23:42:12 argv[5] == "-Condorlog"
6/2 23:42:12 argv[6] == "/home/users/roy/condor-test/simple.log"
6/2 23:42:12 argv[7] == "-Dag"
6/2 23:42:12 argv[8] == "simple.dag"
6/2 23:42:12 argv[9] == "-Rescue"
6/2 23:42:12 argv[10] == "simple.dag.rescue"
6/2 23:42:12 DAG Lockfile will be written to simple.dag.lock
6/2 23:42:12 DAG Input file is simple.dag
6/2 23:42:12 Rescue DAG will be written to simple.dag.rescue
6/2 23:42:12 All DAG node user log files:
6/2 23:42:12 /home/users/roy/condor-test/simple.log (Condor)
6/2 23:42:12 Parsing simple.dag ...
6/2 23:42:12 Dag contains 4 total jobs
6/2 23:42:12 Deleting any older versions of log files...
6/2 23:42:12 MultiLogFiles: deleting older version of /home/users/roy/condor-test/simple.log
6/2 23:42:12 Bootstrapping...
6/2 23:42:12 Number of pre-completed nodes: 0
6/2 23:42:12 Registering condor_event_timer...
6/2 23:42:13 Submitting Condor Node Setup job(s)...
6/2 23:42:13 submitting: condor_submit -a dag_node_name' '=' 'Setup
-a +DAGManJobId' '=' '29
-a submit_event_notes' '=' 'DAG' 'Node:' 'Setup
-a +DAGParentNodeNames' '=' '""
job.setup.submit
6/2 23:42:13 assigned Condor ID (30.0)
6/2 23:42:13 Just submitted 1 job this cycle...
6/2 23:42:13 Event: ULOG_SUBMIT for Condor Node Setup (30.0)
6/2 23:42:13 Number of idle job procs: 1
6/2 23:42:13 Of 4 nodes total:
6/2 23:42:13 Done Pre Queued Post Ready Un-Ready Failed
6/2 23:42:13 === === === === === === ===
6/2 23:42:13 0 0 1 0 0 3 0
6/2 23:43:08 Event: ULOG_EXECUTE for Condor Node Setup (30.0)
6/2 23:43:08 Number of idle job procs: 0
6/2 23:43:13 Event: ULOG_JOB_TERMINATED for Condor Node Setup (30.0)
6/2 23:43:13 Node Setup job proc (30.0) completed successfully.
6/2 23:43:13 Node Setup job completed
6/2 23:43:13 Number of idle job procs: 0
6/2 23:43:13 Of 4 nodes total:
6/2 23:43:13 Done Pre Queued Post Ready Un-Ready Failed
6/2 23:43:13 === === === === === === ===
6/2 23:43:13 1 0 0 0 2 1 0
6/2 23:43:18 Submitting Condor Node Work1 job(s)...
6/2 23:43:18 submitting: condor_submit -a dag_node_name' '=' 'Work1
-a +DAGManJobId' '=' '29
-a submit_event_notes' '=' 'DAG' 'Node:' 'Work1
-a +DAGParentNodeNames' '=' '"Setup"
job.work1.submit
6/2 23:43:18 assigned Condor ID (31.0)
6/2 23:43:18 Submitting Condor Node Work2 job(s)...
6/2 23:43:18 submitting: condor_submit -a dag_node_name' '=' 'Work2
-a +DAGManJobId' '=' '29
-a submit_event_notes' '=' 'DAG' 'Node:' 'Work2
-a +DAGParentNodeNames' '=' '"Setup"
job.work2.submit
6/2 23:43:18 assigned Condor ID (32.0)
6/2 23:43:18 Just submitted 2 jobs this cycle...
6/2 23:43:18 Event: ULOG_SUBMIT for Condor Node Work1 (31.0)
6/2 23:43:18 Number of idle job procs: 1
6/2 23:43:18 Event: ULOG_SUBMIT for Condor Node Work2 (32.0)
6/2 23:43:18 Number of idle job procs: 2
6/2 23:43:18 Of 4 nodes total:
6/2 23:43:18 Done Pre Queued Post Ready Un-Ready Failed
6/2 23:43:18 === === === === === === ===
6/2 23:43:18 1 0 2 0 0 1 0
6/2 23:43:28 Event: ULOG_EXECUTE for Condor Node Work1 (31.0)
6/2 23:43:28 Number of idle job procs: 1
6/2 23:43:28 Event: ULOG_EXECUTE for Condor Node Work2 (32.0)
6/2 23:43:28 Number of idle job procs: 0
6/2 23:43:33 Event: ULOG_JOB_TERMINATED for Condor Node Work1 (31.0)
6/2 23:43:33 Node Work1 job proc (31.0) completed successfully.
6/2 23:43:33 Node Work1 job completed
6/2 23:43:33 Number of idle job procs: 0
6/2 23:43:33 Event: ULOG_JOB_TERMINATED for Condor Node Work2 (32.0)
6/2 23:43:33 Node Work2 job proc (32.0) completed successfully.
6/2 23:43:33 Node Work2 job completed
6/2 23:43:33 Number of idle job procs: 0
6/2 23:43:33 Of 4 nodes total:
6/2 23:43:33 Done Pre Queued Post Ready Un-Ready Failed
6/2 23:43:33 === === === === === === ===
6/2 23:43:33 3 0 0 0 1 0 0
6/2 23:43:38 Submitting Condor Node Final job(s)...
6/2 23:43:38 submitting: condor_submit -a dag_node_name' '=' 'Final
-a +DAGManJobId' '=' '29
-a submit_event_notes' '=' 'DAG' 'Node:' 'Final
-a +DAGParentNodeNames' '=' '"Work1,Work2"
job.finalize.submit
6/2 23:43:38 assigned Condor ID (33.0)
6/2 23:43:38 Just submitted 1 job this cycle...
6/2 23:43:38 Event: ULOG_SUBMIT for Condor Node Final (33.0)
6/2 23:43:38 Number of idle job procs: 1
6/2 23:43:38 Of 4 nodes total:
6/2 23:43:38 Done Pre Queued Post Ready Un-Ready Failed
6/2 23:43:38 === === === === === === ===
6/2 23:43:38 3 0 1 0 0 0 0
6/2 23:43:48 Event: ULOG_EXECUTE for Condor Node Final (33.0)
6/2 23:43:48 Number of idle job procs: 0
6/2 23:43:53 Event: ULOG_JOB_TERMINATED for Condor Node Final (33.0)
6/2 23:43:53 Node Final job proc (33.0) completed successfully.
6/2 23:43:53 Node Final job completed
6/2 23:43:53 Number of idle job procs: 0
6/2 23:43:53 Of 4 nodes total:
6/2 23:43:53 Done Pre Queued Post Ready Un-Ready Failed
6/2 23:43:53 === === === === === === ===
6/2 23:43:53 4 0 0 0 0 0 0
6/2 23:43:53 All jobs Completed!
6/2 23:43:53 Note: 0 total job deferrals because of -MaxJobs limit (0)
6/2 23:43:53 Note: 0 total job deferrals because of -MaxIdle limit (0)
6/2 23:43:53 Note: 0 total PRE script deferrals because of -MaxPre limit (0)
6/2 23:43:53 Note: 0 total POST script deferrals because of -MaxPost limit (0)
6/2 23:43:53 **** condor_scheduniv_exec.29.0 (condor_DAGMAN) EXITING WITH STATUS 0
Clean up your results. Be careful about deleting the simple.dag.* files, you do not want to delete the simple.dag file, just simple.dag.* .
% rm simple.dag.* % rm results.*
|
|
