Asynchronous Delivery

OGSA-DAI provides a several ways of delivering results asynchronously. This section will show you how to use this functionality.

Delivering data to and from a URL

The output from an activity can be delivered to a file on a remote site. A data service can push data to a URL or pull update or bulk load data from a URL.

Use the deliverFromURL activity to pull a file from a URL. Supported protocols are http, https and ftp. Note that when using the ftp protocol you have to specify the username and password within the URL (hence they are transferred in plain text). For example:
```
http://www.ogsadai.org.uk/tutorial/transformRowSet.xsl
ftp://user:password@example.com/path/filename
```
The DeliverFromURL class corresponding to the activity is constructed from a String:
```
DeliverFromURL deliver = new DeliverFromURL( url );
```
Now you can connect the activity's output to the input of another activity:
```
myactivity.setInput( deliver.getOutput() );
```

Delivery to a URL works in a similar way via the deliverToURL activity and its corresponding DeliverToURL class:

DeliverToURL deliver = new DeliverToURL( url );
deliver.setInput( myactivity.getOutput() );

You can use the ftp protocol.

See examples/tutorials/clienttoolkit/XSLTransformWithDelivery.java for an example.

Delivering data to and from a File

The output from an activity can be delivered to a file on the server file system. Like the DeliverFromURL and DeliverToURL activities, a data service can push data to a file or pull update or bulk load data from a file.

Use the deliverFromFile activity to pull a file from the GDS server file system.
The DeliverFromFile class corresponding to the activity is constructed from a String:
```
DeliverFromFile deliver = new DeliverFromFile( "C:/path/to/myfile.txt" );
```
Now you can connect the activity's output to the input of another activity:
```
myactivity.setInput( deliver.getOutput() );
```

Delivery to a file works in a similar way via the deliverToFile activity and its corresponding DeliverToFile class:

DeliverToFile deliver = new DeliverToFile( "C:/path/to/myfile.txt" );
deliver.setInput( myactivity.getOutput() );

Note that, by default, these activities are disabled in activity configuration documents used to configure OGSA-DAI services, due to the security issue arising from enabling access to anywhere on the GDS server file system.

Delivering Data to a GridFTP Location

The output from an activity can be delivered to a file on a remote Grid FTP server. The deliverToGFTP activity is used for this purpose.

The DeliverToGFTP class corresponding to the activity is constructed from three input arguments: the name of the remote host, the port and the file name:
```
DeliverToGFTP deliver =  new DeliverToGFTP("ogsadai.org.uk",
8080, "myresults.txt" );
```

Alternatively, you can construct an empty DeliverToGFTP object and set the values at some later time:

DeliverToGFTP deliver = new DeliverToGFTP();
deliver.setHost( host_name );
deliver.setPort( port_number );
deliver.setFile( file_name );

A DeliverToGFTP activity object has only one input and no input.

Pulling Data from a data service back to the Client

Using the outputStream you can open a connection to a data service and retrieve data using the DataTransport portType. This is a great way to run a query that returns a large amount of data without either the server or client failing with a java.lang.OutOfMemoryError.

Set up a DataService and prepare a request which delivers the output of an activity, e.g. a query activity, to an outputStream (use the OutputStreamActivity class).

SQLQuery sqlQuery = new SQLQuery( "select * from littleblackbook" );
OutputStreamActivity outputStream = new OutputStreamActivity(sqlQuery.getOutput());

ActivityRequest request = new ActivityRequest();
request.addActivity( sqlQuery );
request.addActivity( outputStream );

Perform the request. The data service will wait for the Client to pull the data:
```
service.perform( request );
```
Use the getResultSet() method on the OutputStream to get the results as a JDBC ResultSet object:
```
ResultSet rs = outputStream.getResultSet();
```
This will create a ResultSet object that will be of type ResultSet.TYPE_FORWARD_ONLY. When the next row is requested from the ResultSet object, the DataTransport connection to the data service will be used to retrieve the data.

See examples/tutorials/clienttoolkit/ProcessingSQLQueryLargeResults.java for an example.

Pushing Data from the Client into a data service

Using the inputStream you can open a connection to a data service and supply data using the DataTransport portType.

Set up a data service and prepare a request which pushes the data arriving at the inputStream (use the InputStreamActivity class) into the input of another activity, e.g. an update activity.

InputStreamActivity inputStream = new InputStreamActivity();
SQLUpdate update = new SQLUpdate( "insert into littleblackbook values (?,?,?,?)" );
update.setParameter(1, inputStream.getOutput());
update.setParameter(2, inputStream.getOutput());
update.setParameter(3, inputStream.getOutput());
update.setParameter(4, inputStream.getOutput());

ActivityRequest request = new ActivityRequest();
request.addActivity( inputStream );
request.addActivity( update );

Perform the request. The data service will wait for the Client to push the data:
```
service.perform( request );
```
Use the getDataTransport() method on the InputStream to get a DataTransport object, which allows the Client to push data into the inputStream:
```
DataTransport putter = inputStream.getDataTransport();
```

Use the DataTransport object to push data into the the inputStream and subsequently into the parameters of the SQL update:

putter.putBlock(new Integer(10010));
putter.putBlock("Albert Einstein");
putter.putBlock("Patent Office, Bern");
putter.putBlock("0123456789");

Finally, use the putClosingBlock() method to tell the InputStream that all blocks have been sent:
```
putter.putClosingBlock();
```

See OGSA-DAI/examples/tutorials/clienttoolkit/ParameterisedUpdateViaGDT.java for an example.

Pulling Data from another data service

It is possible to configure the request to pull data from a source data service to another (sink) data service. To configure this the deliverFromDT activity (and the corresponding Java class DeliverFromDT) is used at the source data service to push the data to an outputStream activity (corresponding Java class OutputStreamActivity) at the sink data service.

In this example you will use to data services:

The sink data service request consists of an deliverFromDT activity.
The source data service request consists of an SQL query whose output goes to a outputStream activity.

The deliverFromDT activity pulls data from the source data service's outputStream activity. In order to contact the source data service and pull the data to the sink data service's deliverToDT activity must know the URL of the sink and the name of the outputStream activity.

It is important that the sink data service's inputStream activity is started before the source data service's deliverToGDT activity otherwise this activity will fail.

Set up the sink data service:

String handle = "http://HOST:PORT/wsrf/services/ogsadai/DataService";
String resourceID = "MySQLResource";
DataService sourceService = WSRFServiceFetcher.getInstance().getWSRFDataService(handle, resourceID);

Prepare the request for the source data service. The source data service should deliver the output of an SQL query to the sink data service.

SQLQuery sqlQuery =  new SQLQuery( "select * from littleblackbook where id<100" );
OutputStreamActivity outputStream = new OutputStreamActivity();
outputStream.setInput(sqlQuery.getOutput());
ActivityRequest sourceRequest = new ActivityRequest();
sourceRequest.add( sqlQuery );
sourceRequest.add( outputStream );

Start up the sink:
```
Response sourceResponse = sourceService.perform( sourceRequest );
```
The sink data service will wait for data to arrive at its input stream activity and then deliver the results in the response.

Set up the source data service.

String handle = "http://HOST:PORT/wsrf/services/ogsadai/DataService";
String resourceID = "MySQLResource2";
DataService sourceService = WSRFServiceFetcher.getInstance().getWSRFDataService(handle, resourceID);

Prepare the request for the sink data service. The sink data service should pull the output from the source output stream. Create a DeliverFromDT object activity.
```
DeliverFromDT deliverFromDT = new DeliverFromDT();
```

Then set the DataTransport URL to the URL of the sink and the stream ID to the name of the sink's inputStream activity:

deliverFromDT.setDataTransportURL( sourceService.getURL() );
deliverFromDT.setDataTransportResourceID( sourceService.getResourceID() );
deliverFromDT.setDataTransportStreamID( outputStream.getName() );

The DataTransport can deliver data in blocks or in full. You can set this as follows using the DataTransportMode class:
```
deliverFromDT.setDataTransportMode( DataTransportMode.BLOCK );
```

Finally create a new ActivityRequest and add the activity to it:

ActivityRequest sinkRequest = new ActivityRequest();
sinkRequest.add( deliverFromDT );

Start up the source request:

Response sinkResponse = sinkService.perform( sinkRequest );

Display the sink response - this will show the results of the query you've run.

See examples/tutorials/clienttoolkit/GDTExample.java for an example.

Pulling Data from another data service

Similar to pulling data to another data service, you can push data to a data service.

Set up a source data service and prepare a request which delivers the output of an activity, e.g. a query activity, to an deliverToDT activity (use the DeliverToDT class.
Set up a sink data service and prepare a request with an inputStream (use the InputStreamActivity class). Set the DataTransport URL and resource ID to the URL and ID of the sink data service, the input stream name to the name of the sink's inputStream activity and set the delivery mode using a DataTransportMode object.
Connect to the sink data service and perform the request. The service waits for another data service to push the data.
Connect to the source data service and perform the request. This will begin to push the data to the sink's inputStream activity.