ISSGC'06 :: gLite Advanced Practical

Before to start to interact with the gLite File Catalog (LFC) you need to have a valid proxy with VOMS extensions. As you learned in the introductionary exercise, you can check the validity of your proxy with:

Listing the entries of a LFC directory

LFC Directory Management

The two commands are self-explaining. Again, man can give you more details about the meaning of the flags.
We already set up a directory into LFC, called /grid/gilda/ischia06.

TO DO: Create your own home LFC directory under the /grid/gilda/ischia06 directory using your name or surname or account name.
You will use that dir for the following exercises.

Summary of the LFC commands

Storage Elements Interactions

gLite provides another set of tools to deal with Storage Elements and File Catalogs. They form the so called lcg_utils tools.
These commands allow users and jobs running on Worker Nodes to copy files between a User Interface or Worker Node and a Storage Element, replicate files among Storage Elements and registers atomically all the completed operation into the File Catalog.

Be sure to have set up correctly the following environment variable:

Upload a file to a Storage Element and register it into the file catalog: lcg-cr (Copy & Register)

In the previous example, we have created a local file called myTest.dat and then uploaded the file into a grid Storage Element whose hostname is opteron.gs.unina.it, and then register the uploaded file into the File Catalog with the LogicalFileName lfn:/grid/gilda/ischia06/tcaland/myTest.dat, inside the LFC directory /grid/gilda/ischia06 that I have created previously. I have also used the -v flag to get more verbosity and the the parameter --vo gilda because I am a member of the gilda Virtual Organization.

In the output of lcg-cr, please notice that the GUID that was assigned to the file (guid:b1391277-52ed-4030-9ca2-56c0356d2c41), its SURL (srm://opteron.gs.unina.it/dpm/gs.unina.it/home/gilda/generated/2006-07-16/file5fa84a9a-376d-44e9-9381-a4a8262731e7) and the TURL (gsiftp://opteron.gs.unina.it/opteron.gs.unina.it:/storage/gilda/2006-07-16/file5fa84a9a-376d-44e9-9381-a4a8262731e7.115.0).

As said during the theoretical part this morning, the GUID is a human non-readable string generated by the FileCatalog and garanteed to be unique, the SRL (Site Resource Locator) gives you information on which SE the file is actually stored, while the TURL (Transport URL Or Temporary URL) gives information on which protocol is used to access/transfer the file and its temporary because its location inside the SE can be changed by the SE deamons according to the internal policy set up by the administrator (for example, the file can be stored actually on the disk1 of a disk array and maybe moved later on disk2, or later onto a tape of the library, etc).

TO DO: Create a new local file or use one already existent in your UI, upload it onto a SE, and register it in the Catalog using lcg-cr.

Definition: A grid file is a file that is stored into a Storage Element AND is registered in a File Catalog (has an assigned Logical File Name).

Make a replica of a grid file into another Storage Element: lcg-rep

For example, let's replicate on a different Storage Element the file that we have previously uploaded:

where

–

vo_name specifies the Virtual Organization the user belongs to

–

src_file specifies the Logical File Name, the Grid Unique IDentifier or the Site URL of the file we want to list the replicas.

For example, let's check if the previously issued replica command was successful:

$ lcg-lr --vo gilda lfn:/grid/gilda/ischia06/tcaland/myTest.dat
srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/gilda \
    /generated/2006-07-16/filed270060a-d8ee-4d59-8506-7cc68f47bdb8
srm://opteron.gs.unina.it/dpm/gs.unina.it/home/gilda \
    /generated/2006-07-16/file5fa84a9a-376d-44e9-9381-a4a8262731e7

Yes, it was: we do have two replicas (one on aliserv6 and one of opteron) of the grid file lfn:/grid/gilda/ischia06/tcaland/myTest.dat.

TO DO: Make two replicas of the file you have previously uploaded and registered.

Downloading a Grid file in a SE to a local destination (UI or WN)

If you want to download a Grid file saved on a Storage Element to your User Interface, or you need to do that from a job running on a Worker Node, you can use the following command:

lcg-cp [ -v | --verbose ] --vo <vo_name> <src_file> <dest_file>

where

–

vo specifies the Virtual Organization the user belongs to

–

src_file specifies the source file name: the protocol can be LFN, GUID, SURL or local file

–

dest_file specifies the destination. The protocol can be file:/// or gsiftp:///

Example:

$ lcg-cp -v --vo gilda lfn:/grid/gilda/ischia06/tcaland/myTest.dat file://$PWD/myTest2.dat
Using grid catalog type: lfc
Using grid catalog : lfc-gilda.ct.infn.it
Source URL: lfn:/grid/gilda/ischia06/tcaland/myTest.dat
File size: 0
VO name: gilda
Source URL for copy: 
gsiftp://gildase.oact.inaf.it/gildase.oact.inaf.it:/data01/gilda \
        /2006-07-16/file4f2e4f41-5ef2-4afe-bf5d-4180836387e2.40.0
Destination URL: file:///home/users/tcaland/myTest2.dat
# streams: 1
# set timeout to  0 (seconds)
            0 bytes      0.00 KB/sec avg      0.00 KB/sec inst
Transfer took 2030 ms

$ ls -la myTest*
-rw-r--r--    1 tcaland  users           0 Jul 16 13:30 myTest2.dat
-rw-r--r--    1 tcaland  users           0 Jul 16 12:16 myTest.dat

•

TO DO: Download in your home the file you have previously uploaded and registered.

How to delete a replicas

lcg-del [ -a ] | [ -s se ] [ -v | --verbose ] --vo <vo_name> <file>

where

–

a is used to delete all replicas of the given file. The entry in the file catalog will be also deleted

–

se specifies the SE from which you want to remove the replica

–

vo specifies the Virtual Organization the user belongs to

–

file specifies the Logical File Name, the Grid Unique IDentifier or the Site URL.

Example:
Let me add one more replica to myTest.dat (to have 3 replicas in total):

$ lcg-rep -v -d gildase.oact.inaf.it --vo gilda lfn:/grid/gilda/ischia06/tcaland/myTest.dat
$ lcg-lr --vo gilda lfn:/grid/gilda/ischia06/tcaland/myTest.dat
srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/gilda \
     /generated/2006-07-16/filed270060a-d8ee-4d59-8506-7cc68f47bdb8
srm://gildase.oact.inaf.it/dpm/oact.inaf.it/home/gilda \
     /generated/2006-07-16/file1f0333e4-89c6-4fcf-ae86-a5f9938206a8
srm://opteron.gs.unina.it/dpm/gs.unina.it/home/gilda \
     /generated/2006-07-16/file5fa84a9a-376d-44e9-9381-a4a8262731e7

•

Let's delete just one replica:

$ lcg-del -v -s aliserv6.ct.infn.it --vo gilda lfn:/grid/gilda/ischia06/tcaland/myTest.dat
VO name: gilda
set timeout to 0 seconds

$ lcg-lr --vo gilda lfn:/grid/gilda/ischia06/tcaland/myTest.dat
srm://gildase.oact.inaf.it/dpm/oact.inaf.it/home/gilda \
    /generated/2006-07-16/file1f0333e4-89c6-4fcf-ae86-a5f9938206a8
srm://opteron.gs.unina.it/dpm/gs.unina.it/home/gilda \
    /generated/2006-07-16/file5fa84a9a-376d-44e9-9381-a4a8262731e7

•

Let's delete the last two replicas at a time:

$ lcg-del -v -a --vo gilda lfn:/grid/gilda/ischia06/tcaland/myTest.dat
VO name: gilda
set timeout to 0 seconds

$ lcg-lr --vo gilda lfn:/grid/gilda/ischia06/tcaland/myTest.dat
lcg_lr: No such file or directory

$ lfc-ls -l /grid/gilda/ischia06/tcaland/
$
(you will not see anymore myTest.dat)

•

TO DO: Delete the replicas of the file you have previously uploaded and registered and make sure that it is also removed from the File Catalog

File Catalog Interactions

Listing the entries of a LFC directory

LFC Directory Management

Summary of the LFC commands

Storage Elements Interactions

Upload a file to a Storage Element and register it into the file catalog: lcg-cr (Copy & Register)

Make a replica of a grid file into another Storage Element: lcg-rep

Top