 |
|
|
| Main Page |
| Mandadory TAGS for DAGs |
| Exercise |
Dyrect Acyclic Graphs (DAG) Jobs
A DAG (directed acyclic graph) represents a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs. The jobs are nodes (vertices) in the graph and the edges (arcs) identify the dependencies.
Although not requiring a great number of new attributes, the structure of the JDL description for a DAG differs significantly from the one for a job and it is more complex. Due to this we start by providing an example of DAG description so that the structure of the JDL in this case is clear from the beginning. The new attributes and the slight differences in the semantic for some of the known ones will be highlighted in the text.
It is important to note that upon submission of a DAG, besides the identifier associated with each node, the WMS assigns also to the DAG itself an identifier that has to be used as the handle for monitoring and controlling the whole DAG.
Example - DAG example
It can be described by the following JDL:
[
Type = "dag";
VirtualOrganisation = "gilda";
InputSandbox = {"/tmp/foo", "/home/gliteuser/bar", "/tmp/myconf"};
max_nodes_running = 5;
nodes = [
nodeA = [
description = [
JobType = "Normal";
Executable = "a.exe";
InputSandbox = {"/home/data/myfile.txt", root.InputSandbox};
];
];
mynode = [
description = [
JobType = "Normal";
Executable = "b.exe";
Arguments = "1 2 3";
RetryCount = 3;
Requirements = other.GlueCEInfoTotalCPUs > 2;
OutputSandbox = {"myoutput.txt", "myerror.txt" };
];
];
nodeD = [
description = [
JobType = "Normal";
Executable = "b.exe";
Arguments = "1 2 3";
RetryCount = 3;
InputSandbox = {"/home/pippo", "/home/test/b.exe"};
];
];
nodeC = [
file = "/home/test/c.jdl";
];
nodeB = [
file = "foo.jdl";
];
dependencies = { { nodeA, nodeB }, { nodeA, nodeC }, {nodeA, mynode },
{ { nodeB, nodeC, mynode }, nodeD }
};
];
];
Mandadory TAGS for DAGs:
Type: In this case the value for the Type attribute is “DAG”.
Type="DAG";
VirtualOrganisation: The Virtual Organisation must be the same for the DAG and all its nodes
VirtualOrganization="gilda";
Nodes: The Nodes attribute is the core of the DAG description and it is used for specifying the nodes and their dependencies. It contains all the nodes of the DAG and the attribute named Dependencies. Each node corresponds to a classad attribute (nodeA, nodeB, nodeC, nodeD and mynode in the previous Example), whose name can be freely set by the user, which contains the description of the job representing the node. Such description can be provided through the File or Description attributes:
Dependencies: The Dependencies attribute is a list of lists representing the dependencies between the nodes of the DAG described in the nodes attribute. Each sub-list specifies dependencies between a sub set of nodes.
Nodes = [
nodeName1 = [
description = [
JobType="Normal";
Executable="abc.exe";
Arguments="1 2 3";
....
];
];
nodeFile2 = [
file = "node2.jdl";
];
dependencies = {
{nodeName1, nodeFile2}
}
];
In this exercise we will use again the RegularExplorer Java class that you have seen in the Job Management Commands section. The goal is submit multiple jobs to sample multiple areas of the surface in a single step. This is achieved using a easy DAG without dependecies.
We will make use of a simple bash script (dag_gener.sh) to generate the JDL nodes and the main DAG JDL. This script reads an argument.list file with the list of the different input parameters to sample the surface and takes as input a number n of lines to read. It will generate n JDL nodes.
Download here the files of this exercise.
Download here the Shapes Data.
Step 1: Create an argument.list file. Insert in this file your input parameters (x0, y0, x1, y1, #_of_samples) according to the following format:
arguments=0 0 10 10 2000 out1.dat
arguments=10 10 20 20 2000 out2.dat
arguments=20 20 30 30 2000 out3.dat
Step 2: Create the node jdls and the DAG JDL files running the dag_gener.sh script :
$ dag_gener 3 $ ls arguments.list dag_gener.sh expl1.jdl expl2.jdl expl3.jdl father_job.jdl jobid.txt regExplorerTemplate.jdl ShapesData.tar.gz
Step 3: Take a look to the file father_job.jdl and checks the format:
[ type = "dag"; max_nodes_running = 3; nodes = [ explorer1 = [ file ="expl1.jdl" ; ]; explorer2 = [ file ="expl2.jdl" ; ]; explorer3 = [ file ="expl3.jdl" ; ]; dependencies = {} ]; ]
Step 4: DAG submission can be accomplished through the same commands used for simple jobs. Run the DAG job (father_job.jdl) generated by dag_gener.sh:
$ glite-job-submit -o jobid.txt father_job.jdl Selected Virtual Organisation name (from proxy certificate extension): gilda Connecting to host glite-rb.ct.infn.it, port 7772 Logging to host glite-rb.ct.infn.it, port 9002 ================================ glite-job-submit Success ===================================== The dag has been successfully submitted to the Network Server. Use glite-job-status command to check job current status. Your dag identifier is: - https://glite-rb.ct.infn.it:9000/uDvUGtm8m_OVPCCRk349fA The dag identifier has been saved in the following file: /home/scardaci/dag/jobid.txt ============================================================================================= $ glite-job-status -i jobid.txt ************************************************************* BOOKKEEPING INFORMATION: Status info for the Job : https://glite-rb.ct.infn.it:9000/GujM4PRwXLl1bAxiM37EZw Current Status: Running Status Reason: unavailable Destination: dagman Submitted: Sun Jul 16 21:46:07 2006 CEST ************************************************************* - Nodes information: Status info for the Job : https://glite-rb.ct.infn.it:9000/E5ROK_BY3LzmpAGATayZ2g Node Name: explorer3 Current Status: Scheduled Status Reason: Job successfully submitted to Globus Destination: grid-ce.bio.dist.unige.it:2119/jobmanager-lcgpbs-short Submitted: Sun Jul 16 21:46:07 2006 CEST ************************************************************* Status info for the Job : https://glite-rb.ct.infn.it:9000/5x0lRXgCg4nnw8duk7K5ew Node Name: explorer2 Current Status: Scheduled Status Reason: Job successfully submitted to Globus Destination: grid-ce.bio.dist.unige.it:2119/jobmanager-lcgpbs-long Submitted: Sun Jul 16 21:46:07 2006 CEST ************************************************************* Status info for the Job : https://glite-rb.ct.infn.it:9000/sSCrk2w86hrCZ85PMnNchA Node Name: explorer1 Current Status: Running Status Reason: Job successfully submitted to Globus Destination: grid-ce.bio.dist.unige.it:2119/jobmanager-lcgpbs-infinite Submitted: Sun Jul 16 21:46:07 2006 CEST *************************************************************
Step 5: Monitor the job until its status is DONE and finally retrieve the output :
$ glite-job-status -i jobid.txt ************************************************************* BOOKKEEPING INFORMATION: Status info for the Job : https://glite-rb.ct.infn.it:9000/GujM4PRwXLl1bAxiM37EZw Current Status: Done (Success) Exit code: 0 Status Reason: Job terminated successfully Destination: dagman Submitted: Sun Jul 16 21:46:07 2006 CEST ************************************************************* - Nodes information: Status info for the Job : https://glite-rb.ct.infn.it:9000/E5ROK_BY3LzmpAGATayZ2g Node Name: explorer3 Current Status: Done (Success) Exit code: 0 Status Reason: Job terminated successfully Destination: grid-ce.bio.dist.unige.it:2119/jobmanager-lcgpbs-short Submitted: Sun Jul 16 21:46:07 2006 CEST ************************************************************* Status info for the Job : https://glite-rb.ct.infn.it:9000/5x0lRXgCg4nnw8duk7K5ew Node Name: explorer2 Current Status: Done (Success) Exit code: 0 Status Reason: Job terminated successfully Destination: grid-ce.bio.dist.unige.it:2119/jobmanager-lcgpbs-long Submitted: Sun Jul 16 21:46:07 2006 CEST ************************************************************* Status info for the Job : https://glite-rb.ct.infn.it:9000/sSCrk2w86hrCZ85PMnNchA Node Name: explorer1 Current Status: Done (Success) Exit code: 0 Status Reason: Job terminated successfully Destination: grid-ce.bio.dist.unige.it:2119/jobmanager-lcgpbs-infinite Submitted: Sun Jul 16 21:46:07 2006 CEST ************************************************************* $ glite-job-output -i jobid.txt Retrieving files from host: glite-rb.ct.infn.it ( for https://glite-rb.ct.infn.it:9000/E5ROK_BY3LzmpAGATayZ2g ) Retrieving files from host: glite-rb.ct.infn.it ( for https://glite-rb.ct.infn.it:9000/5x0lRXgCg4nnw8duk7K5ew ) Retrieving files from host: glite-rb.ct.infn.it ( for https://glite-rb.ct.infn.it:9000/sSCrk2w86hrCZ85PMnNchA ) ********************************************************************************* JOB GET OUTPUT OUTCOME Output sandbox for the job: - https://glite-rb.ct.infn.it:9000/GujM4PRwXLl1bAxiM37EZw Children files retrieved/purged: - https://glite-rb.ct.infn.it:9000/E5ROK_BY3LzmpAGATayZ2g - https://glite-rb.ct.infn.it:9000/5x0lRXgCg4nnw8duk7K5ew - https://glite-rb.ct.infn.it:9000/sSCrk2w86hrCZ85PMnNchA Files are stored in the directory: /tmp/glite/glite-ui/scardaci_GujM4PRwXLl1bAxiM37EZw ********************************************************************************* $ cd /tmp/glite/glite-ui/scardaci_GujM4PRwXLl1bAxiM37EZw $ ls scardaci_5x0lRXgCg4nnw8duk7K5ew scardaci_E5ROK_BY3LzmpAGATayZ2g scardaci_sSCrk2w86hrCZ85PMnNchA $ ls scardaci_5x0lRXgCg4nnw8duk7K5ew/ explorer.err explorer.out out2.dat
Step 6: Take a look to the files out<i>.dat returned by the job using gnuplot to visualize the tridimensional surface as in the previous exercise.
|
|
