The OMII Advanced GridSAM Practical

Overview

For this practical we will be focusing on the client API provided by GridSAM, building on the concepts introduced in the Introductory Practical. 

For those interested, the GridSAM client docs contains a wealth of information concerning GridSAM to do with deployment strategies, development, quality assurance and use cases.

Top

Developing GridSAM Java clients in OMII

The GridSAM client itself is built around a Java API, and the documentation provided by GridSAM includes JavaDocs which you can use as a reference for this practical.

This practical will introduce the Java API with the following example:

Basically, the example involves submitting JSDL that executes which is constructed in-program to a GridSAM server, and monitoring the job until completion.

For this tutorial, you will need an FTP server running to service the staging out requests required by the JSDL (much like in the Introductory Practical):

Ensure that <port_number> is different to those also on the same machine as you e.g. 55521, 55522, 55523, etc...

You need to create a directory in <omii_client_home>, ensuring it is different to directories created by other users on your machine, eg:

The tutorial materials will now be download, compiled and run in here.  We'll refer to this directory as user1 from now on, but you should substitute this with the directory name you have used.

The materials necessary for this practical can be found on the website. You can download the materials package either by:

• Browser: right-clicking on materials/GSExample-linux.tar.gz and selecting 'Save Target As' (or equivalent) to <omii_client_home>/user1 (for example).

Or:

• Command line (from within the <omii_client_home>/user1 directory): wget http://www.gs.unina.it/~omii/practicals/AdvancedPractical/materials/GSExample-linux.tar.gz.

Then, use the following commands to unpack that archive:

You should find the following files:

• GSExample.java: the source Java for this tutorial

• GSExample.class: the compiled Java for this tutorial

• GSExampleCompile: a script that compiles the above

• GSExampleRun: a script that runs the above

Before we delve into the API mechanics required to develop code that conducts job submission and monitoring, let us compile and run the code and see what happens.

Top

Compile the Java

It's already compiled, but why not compile it again - at least to verify the compilation script works on your machine before you make big changes to the code.  You'll need to edit the file GSExample.java so that it points to the correct server where GridSAM is installed.  Change the string omiitest12.omii.ac.uk:18080 to point to the demo GridSAM server (using <server> and <port>).

You can then compile the Java:

The script looks like this:

Which basically compiles GSExample.java against the GridSAM libraries.

Top

Run the Java

Prior to running the example client, you also need to define the location of your local FTP server - i.e. your local machine hostname and FTP port, so the GridSAM server is able to stage the output file back to you.  You need to change the GSExampleRun script which looks like this:

Therefore, you need to edit it and change the ftp.server property to point to your local FTP server e.g. yourmachine.full.domain:55521.

You can then run the script using the simple command:

You should see the following output:

Like the GridSAM status command, you can see the various stages the job has traversed, except this client operates for the lifetime of the job and continually outputs the job status.

Top

Typical Java Client Development Process

The process, from a client point of view, to achieve submitting a job and monitoring it until completion is as follows:

1. Create a ClientSideJobManager: this is the entry point into GridSAM system for submission and monitoring

2. Create out JSDL to submit

3. Submit the job (described in JSDL) to GridSAM server using ClientSideJobManager

4. Monitor status of job until it is complete

We'll now look at each of these activities and how they are achieved.

Top

The Java

You should be able to determine the rationale behind the following yourself:

1.  Create a ClientSideJobManager:

ClientSideJobManager jobManager = new ClientSideJobManager(new String[] {
    "-s",
    "http://<yourserver>:<yourport>/gridsam/services/gridsam?wsdl },
    ClientSideJobManager.getStandardOptions());

2.  Create our JSDL to submit:

String xJSDLString = createJSDLDescription("/bin/ls", "-a");

• createJSDLDescription() convenience function accepts program to run and arguments

• Returns JSDL

3.  Submit the JSDL to GridSAM server:

JobInstance job = jobManager.submitJob(xJSDLString);
String jobID = job.getID();

4.  Monitor job until completion:

String state = "";
do {
    // Update Job Manager’s status of the job
    job = jobManager.findJobInstance(jobID);

    // Get status of job
    List jsList = job.getJobStages();
    if (jsList.size() > 0) {
        JobStage js = (JobStage) jsList.get(jsList.size()-1);
        state = js.getState().toString();
    }
    Thread.sleep(2000);
} while (!state.equals("done") && !state.equals("failed"));

Top

Exercises

Develop the Java client:

1. Modify the client so that:

   • It takes 3 input text files and submits a job that concatenates these files together into a single file

   • Stages this concatenated file back to the client's FTP server

   • HINTS: you'll need to:

     • Modify/create a different createJSDLDescription() function that handles inputs.  Replace the <Target> element in <DataStaging> with <Source> to accomplish this

     • Put the 3 input files in the datastaging directory for input staging

     • Create your own program (as a script) that is also staged to the server to perform the concatenation

2. Modify the client so it performs a simple client workflow that:

   • Submits, and monitors until completion, 3 multiple jobs simultaneously (e.g. /bin/ls, /bin/env, /bin/arch)

   • Then submits the output from these jobs to a fourth job that performs the concatenation and output staging above.

Top