-- TWikiAdminUser - 2014-07-26

India-CMS GRID User Guide


The GRID introduces the concept of a GRID certificate to authenticate users. A GRID certificate is issued by a Certificate Authority (CA) which checks the identity of the user and guarantees that the holder of this certificate is existing and his certificate is valid.

The certificate is used for authentication instead of the user's account to avoid the replication of the user's account to all GRID sites. When authenticating to a site, the user's certificate is mapped to a local account under which all commands are executed.

The certificate itself is not used during the actual GRID usage. All GRID jobs use a proxy of the certificate with a limited lifetime. This enhances security because the user has to re-establish the validty of his certificate after the lifetime of the proxy has ended.

In the following, the application procedure for a GRID certificate and the generation of a proxy is described. The application has to be done once followed by the installation of the certificate in the home directory of the user's account. The proxy generation has to be repeated everytime no valid proxy exists on the user's submission machine.

Application procedure

  • Getting Grid Certificate

  • To work in Grid Environment the user should have a valid grid certificate signed by recognized grid computing certification authority, as for our country user can apply for certificate to "Indian Grid Certificate Authority (IGCA)"
Please follow the instruction on (Indian Grid Certificae Authority (IGCA) and apply for certificate.
  • Make sure you executed all steps carefully and after recieving the certificate please load it into your internet browser
  • Please register yourself with the virtual organization (VO membership) e.g. CMS VO membership
  • Getting an account at India-CMS Tier-III at TIFR, Mumbai and T2_IN_TIFR

    • After getting certificate and VO membership, Please send an email with your lxplus username and certificate DN, to all members in this list and your guide or supervisor.
    • You will recieve your account information and machine name in the reply to you email, you will also recieve temporary onetime password which you have change when you login into your account.
    • Configure your account with your grid certificate ( As described in these steps GRIDCredentials)

    Installation of the certificate

    After the successful application, the certificate has to be installed in the user's home directory following these instructions:

  • Export or 'backup' the certificate from the browser used for the application. The interface for this varies from browser to browser. The exported file will probably have the extension .p12 or .pfx. Guard this file carefully. Store it off your computer, or remove it once you are finished with this process.
  • Copy the file to the user's home directory.
  • Create a directory in the user's home directory:
    mkdir $HOME/.globus 
  • Extract the certificate creating a public and a private key file replacing YourCert.p12 with the filename chosen during step 1:
    openssl pkcs12 -in YourCert.p12 -clcerts -nokeys -out $HOME/.globus/usercert.pem  
  • openssl pkcs12 -in YourCert.p12 -nocerts -out $HOME/.globus/userkey.pem 
    The user will be asked to define a passphrase during this step. This passphrase has to be entered every time a proxy is created from the certificate. For security reasons, an empty passphrase is not adviseable.
  • Set the access mode on your userkey.pem and usercert.pem files:
    chmod 400 $HOME/.globus/userkey.pem chmod 600 $HOME/.globus/usercert.pem 
  • Further protection of the $HOME/.globus directory is necessary to prevent everyone except the user to enter this directory:
    chmod go-rx $HOME/.globus 

The user's GRID certificate (usercert.pem and userkey.pem) can be copied to every other machine to access the GRID by transporting the $HOME/.globus directory. The security measures described above have to be repeated.

Proxy generation

Note: the execution of commands indicates below requires a installed and setup GRID user interface (UI) on the user's machine. The installation and setup of an UI is described in the following section: User interface.

After installation of the certificate in the user's home directory, the following command creates a user proxy:

voms-proxy-init -voms cms 

using the passphrase defined during installation.

To check how long the user's proxy is valid, use the following command:

voms-proxy-info -all 

A valid proxy should produce a similar output like:

 voms-proxy-info -all subject   : /C=TW/O=AP/OU=GRID/CN=BRIJ KISHOR JASHAL 122117/CN=proxy issuer    : /C=TW/O=AP/OU=GRID/CN=BRIJ KISHOR JASHAL 122117 identity  : /C=TW/O=AP/OU=GRID/CN=BRIJ KISHOR JASHAL 122117 type      : proxy strength  : 1024 bits path      : /tmp/x509up_u51562 timeleft  : 11:59:49 key usage : Digital Signature, Key Encipherment, Data Encipherment, Key Agreement === VO cms extension information === VO        : cms subject   : /C=TW/O=AP/OU=GRID/CN=BRIJ KISHOR JASHAL 122117 issuer    : /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch attribute : /cms/Role=NULL/Capability=NULL timeleft  : 11:59:49 uri       : voms.cern.ch:15002

How to use Tier-II account at T2_IN_TIFR


At T2_IN_TIFR we are using Disk Pool Manager (DPM) as storage solution. As per WLCG recommendations we have deployed multiple access technologies for accessing the storage area. Details for utilising T2 resources.

Following are the access protocols currently enabled.

The user space at T2_IN_TIFR can be used in following ways,

  1. Storing your analysis data
  2. To stage out the output of your jobs running on any WLCG site (including T2_IN_TIFR)

Stageout and publication

CRAB allows you to copy your analysis outputs directly to a Tier2 or Tier3 Storage Element. You can decide to store them in a Storage Element of an "Official CMS site" or in some SE of your choosing. The difference between the two is technical, not political. "Official CMS site" simply means a site fully described in CMS data transfer tool (PhEDEx ). If the target site has a working PhEDEx node connected to the stageout location, then the site is listed in SiteDB and crab can discover all the details needed for the stageout from the site name in the CMS format Tx_CC_SiteName, where x is the tier number (usually 2 or 3) and CC is the country code, e.g. T2_IT_Pisa, T2_US_UCSD, T3_IT_Trieste etc. If the target Storage Element is not known to PhEDEx , you will have to indicate full contact details in

1. Stage out to an "official CMS site" without publication in DBS

you have to configure the crab.cfg with:
                copy_data = 1
                storage_element = T2_IN_TIFR  
                 user_remote_dir = subdirectory where your output will be stored   
    • The CMS site names are reported in the !CMS.SiteDB . The mapping between the StorageElement name and CMS site names is reported here
    • The area in which your output file will be written is:
        site's endpoint +  /store/user/<username>/<user_remote_dir>/<output-file-name>
where the site's endpoint is discovered by CRAB and your username is extracted form SiteDB

For example: will write the output into:
               storage_element = T2_IN_TIFR  
               user_remote_dir = myTestDir 
 will write the output into: 

2. Stage out to an "Official CMS site" with publication in DBS

you have to configure the crab.cfg with:

    [USER]     copy_data = 1     
     storage_element = T2_IN_TIFR      
     publish_data_name = <data name to publish >
     dbs_url_for_publication = <your local dbs_url> 

      site's endpoint +  /store/user/<username>/<primarydataset>/<publish_data_name>/<PSETHASH>/<output_file_name>

where the site's endpoint is discovered by CRAB and your username is extracted form SiteDB .

In you need more information about publication as how to publish your data in a different directory i.e /store/group/groupname/.... please read the https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCrabForPublication

for full details please refer to https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCrabFaq#Stageout_and_publication

  1. To stage out the output of the jobs (batch jobs or CRAB Jobs) running at Tier-III at TIFR

Note* - It is advised that If you are using more then 5 TB of storage space in your user area, Please inform any of the system administrator via email

Setup and usage instructions for Tier-III account.


At present tier III account has been set up with condor as the batch system, system has been designed in such a way that as per the load on the system and user requirement no of job slots can be increased or decreased. We have dedicated NFS based tier-III storage area, which is separate from the user space at tier II storage. Directory structure and it's details are following

  • With your credentials you can login on User Interface machine at TIFR. Currently ew have two user interface machines.
    • Hostname for SLC6 - ui.indiacms.res.in
    • Hostname for SLC5 - ui2.indiacms.res.in 
  • CMS software is mounted using CERN Virtual Machine File System (CVMFS) , where all supported versions of CMSSW software are available. Directory location is /cvmfs/cms.cern.ch/
  • Tier III storage space is available under your user home directory as. /home/<username>/t3store You are requested to use this location to store your user data.

Test your grid certificate

  1. Is your personal certificate able to generate Grid proxies? To find out, after having setup your environment run this command:
    grid-proxy-init -debug -verify 
    In case of failure, the possible causes are:
    • the certificate/key pair is not installed under $HOME/.globus/
    • the certificate has expired
    • the certificate and the private key do not match
In the first case, you either do not have a certificate at all or have to install it on the UI; in the second case, you should get a new certificate; in the third case you probably have incorrectly installed your certificate.
  1. Are you a member of the CMS VO? To see if this is the case, you can execute this command:
    voms-proxy-init -voms cms 
    If you get an error, chances are that you did not register to the CMS VO, or your registration expired. In this case, please follow the instructions in this page: https://twiki.cern.ch/twiki/bin/view/CMS/WebHome?topic=SWGuideLcgAccess .

Submissions of CMSSW jobs locally - ui.indiacms.res.in

Before submitting jobs to the Grid, it is a good practice to run the CMSSW code locally over a few events to discard problems not related with CRAB.

To run an analysis CMSSW code, one needs to specify an input dataset file directly in the CMSSW parameter-set configuration file (specifying as well how to access/open the file). One could either copy one file of the remote input dataset to a local machine, or, more conveniently, open the file remotely. For both, the recommended tool is the Xrootd service (please refer to the https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookXrootdService to learn the basics about how to use Xrootd). We will choose to open a file remotely. In any case, one firt shas to find out the LFN of such a file. We used DAS to find the files in the dataset.

https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookCRAB3Tutorial#DAS query for files in dataset /GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RECO.

Input dataset

In order to run an analysis over a given dataset, one has to find out the corresponding dataset name and put it in the CRAB configuration file. In general, either someone will tell you which dataset to use or you will use the DAS web interface to find available datasets. To learn how to use DAS, the reader can refer to the https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookLocatingDataSamples and/or read the more complete documentation linked from the DAS web interface. For this tutorial, we will use the datasets pointed out https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookCRAB3Tutorial#About_softwares_versions_dataset Below are screenshots of corresponding DAS query outputs for these datasets, where one can see that:

  • The datasets are in status "VALID" (otherwise we wouldn't be able to run on them).
  • The /GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RECO MC dataset has 1 block, 177 files and 300K events.

    CMSSW configuration file to process an existing dataset

    This section shows the CMSSW parameter-set configuration file that we will use in this tutorial when running over an existing dataset (either MC or Data, either CMS official or user-produced). We call it pset_tutorial_analysis.py.


import ParameterSet .Config as cms

process = cms.Process('NoSplit')

process.source = cms.Source("PoolSource", fileNames = cms.untracked.vstring('root://se01.indiacms.res.in//store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/00CE4E7C-DAAD-E111-BA36-0025B32034EA.root'))
process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(10))
process.options = cms.untracked.PSet(wantSummary = cms.untracked.bool(True))
process.output = cms.OutputModule("PoolOutputModule",
outputCommands = cms.untracked.vstring("drop *", "keep recoTracks_*_*_*"),
fileName = cms.untracked.string('output.root'),
process.out = cms.EndPath(process.output)

This analysis code will produce an output ROOT file called output.root containing only the recoTracks_*_*_* branches for a maximum of 10 events in the input dataset.

Notice that we added the string root://se01.indiacms.res.in/ before the LFN. The first part, root:, specifies that the file should be opened with ROOT.

note.gif Note: When running CMSSW code locally with cmsRun, we suggest to do it in a separate (fresh) shell, where the user sets up the Grid and CMS environments, but skips the CRAB setup. This is to avoid the CRAB environment to interfere with the CMSSW environment.

For the list of supported and production releases of CMSSW software please refer to this link


Please ensure that you select the appropriate architecture for this machine. Machine architecture is following

Description: Scientific Linux release 6.5 (Carbon)

Setting up the environment

For SLC6 at ui.indiacms.res.in ( for bash shell )

#export SCRAM_ARCH=<architecture> 
#source /cvmfs/cms.cern.ch/cmsset_default.sh ( _CMS environment_) 
#scram l CMSSW ( _list of CMSSW versions under this arch_) 
#scram p CMSSW CMSSW_<version> ( _creates a working area ./CMSSW_<version>_) 
#cd CMSSW_<version>/src 
#cmsenv ( _Setting cms environment_ ) 
#edmConfigHash your_python_file.py ( _To test your .py file whether it complies with CMSSW_ ) 

For SLC6 at ui.indiacms.res.in ( for tcsh shell )
#set SCRAM_ARCH=<architecture> 
#source /cvmfs/cms.cern.ch/cmsset_default.csh ( _CMS environment_) 
#scram l CMSSW ( _list of CMSSW versions under this arch_) 
#scram p CMSSW CMSSW_<version> ( _creates a working area ./CMSSW_<version>_) 
#cd CMSSW_<version>/src #cmsenv ( _Setting cms environment_ ) 
#edmConfigHash your_python_file.py ( _To test your .py file whether it complies with CMSSW_ )

Now we run the analysis locally:

#cmsRun pset_tutorial_analysis.py    

Note* After setting up cmsenv. rfio set of commands wil stop working because of environment variable conflict. therefor you have to execute following command for using rfdir and setting up proxy etc etc.*

# source /home/public/env_ui.sh  #(for Bash shell) 
# set  /home/public/env_ui.csh  #(for tcsh shell)
But please remember that In order to <verbatim>again use CMS environment specific commands<font face="#mce_temp_font#">, you have to again execute commands </font></verbatim>to setup cms environment. as above http://indiacms.res.in/twiki/bin/view/CMS/IndiaCMSGridUserGuide#Setting_up_the_environment

Submission of local batch jobs - ui.indiacms.res.in

A set of example testing scripts are available at /home/public/testjob.

Detailed Manual for condor is available here http://research.cs.wisc.edu/htcondor/manual/v8.0/index.html

Following are the commands used to submit condor jobs

#condor_submit myjob.submit ( to submit the jobs)

#condor_q job_id (toknow the status of job)

#condor_status (to know the status of available job slots)


arguments=Example.$(Cluster).$(Process) 100
when_to_transfer_output = ON_EXIT

echo "I'm process id $$ on" `hostname`
echo "This is sent to standard error" 1>&2
echo "Running as binary $0" "$@"
echo "My name (argument 1) is $1"
echo "My sleep duration (argument 2) is $2"
sleep $2
echo "Sleep of $2 seconds finished.  Exiting"

Submission of standard CRAB job

In addtion to the above steps of setting up of environment <Submissions of CMS jobs locally - ui.indiacms.res.in>

A sample CRAB file is

#number_of_jobs = 5

pset = qcd_pt_40_7tev_pythia8_tune_monas1_v0_cff_py_gen9999.pydatasetpath = None

total_number_of_events =4

events_per_job = 2

output_file = qcdevt_pythia8_7tev_tune_monas1_v0_flat9999.root

#to stageout at site T2 storage (return_data = 0 copy_data = 1)

return_data = 0

copy_data = 1

storage_element = T2_IN_TIFR
user_remote_dir = Test

email = yourid@cern.ch


scheduler =remoteGlidein

jobtype = cmssw


with the above file your crab otput will be staged out in


following are the additonal steps for submitting standard CRAB jobs.

#source /cvmfs/cms.cern.ch/crab3/crab.sh (for bash shell) 
#source /cvmfs/cms.cern.ch/crab3/crab.csh (for tcsh shell)
#crab checkwrite ----site=T2_IN_TIFR    ( To check permissions to write on the site storage )
#crab -create -submit -cfg  YOUR_CRAB.cfg

How to open root file

In addtion to the above steps, In order to open root file in your T3 account at ui.indiacms.res.in. Xrootd is now the standard way of openining files using root locally or anywhere ( Any of the CMS T1s or T2s )

Following are the steps to open files locally

  # root -l
  root [0]# TFile *f =TFile::Open("root://se01.indiacms.res.in//store/test/xrootd/CMSSAM//store/mc/SAM/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0013/CE4D66EB-5AAE-E111-96D6-003048D37524.root")

Transferring files To/From T2 and T3 at TIFR

There are various scenerios of data transfers To/From T2 and to your T3 account at TIFR, most of the possibilities can be summed up as following. For details you may check out this page


Transfer using lcg-cp

Ex. 1 from other T1/T2 to TIFR T2

lcg-cp -v -b -n 4 srmv2 srm://cmssrmdisk.fnal.gov:8443/srm/managerv2?SFN=/dcache/uscmsdisk/store/mc/Summer12DR53X/WJetsToLNu_TuneCUETP8M1_8TeV-madgraphMLM-pythia8/AODSIM/PU_S10_TuneCUETP8M1_START53_V19-v1/10000/78448563-478C-E411-ABC8-002618943834.root srm://se01.indiacms.res.in:8446/srm/managerv2?SFN=/dpm/indiacms.res.in/home/cms/store/mc/Summer12DR53X/WJetsToLNu_TuneCUETP8M1_8TeV-madgraphMLM-pythia8/AODSIM/PU_S10_TuneCUETP8M1_START53_V19-v1/10000/78448563-478C-E411-ABC8-002618943834.root

Ex. 2 from TIFR to other T1/T2

lcg-cp -v -b -n 4 srmv2 srm://se01.indiacms.res.in:8446/srm/managerv2?SFN=/dpm/indiacms.res.in/home/cms/store/mc/Summer12DR53X/WJetsToLNu_TuneCUETP8M1_8TeV-madgraphMLM-pythia8/AODSIM/PU_S10_TuneCUETP8M1_START53_V19-v1/10000/78448563-478C-E411-ABC8-002618943834.root srm://cmssrmdisk.fnal.gov:8443/srm/managerv2?SFN=/dcache/uscmsdisk/store/mc/Summer12DR53X/WJetsToLNu_TuneCUETP8M1_8TeV-madgraphMLM-pythia8/AODSIM/PU_S10_TuneCUETP8M1_START53_V19-v1/10000/78448563-478C-E411-ABC8-002618943834.root

Using xrd-cp

Ex from TIFR T2 to your local area

xrdcp root://se01.indiacms.res.in//store/user/pawan/Phys14dr/QCD_Pt-1400to1800_Tune4C_13TeV_pythia8/QCD_Pt-1400to1800_Tune4C_13TeV_pythia8_Phys14DR-AVE20BX25_tsg_castor_PHYS14_25_V3-v1/150623_133444/0000/Ouput_step2_raw_73.root /home/<your-area>

usinf rfcp

Ex. from T2 to local

rfcp /dpm/indiacms.res.in//home/cms/store/mc/SAM/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0013/3E573418-4BAE-E111-B0CD-BCAEC532971D.root /var/tmp/

Ex from local to T2

rfcp /var/tmp/filename.root /dpm/indiacms.res.in//home/cms/store/mc/SAM/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0013/filename.root

Using xrootd to access files at T2s

looging to a xrootd server

#connecting to storage server e.g TIFR

xrd se01.indiacms.res.in

#Listing the files


# Type help to see all the command options.

xrdcp root://xrootd.unl.edu//store/foo /some/local/path 
You will get a progress bar as the file downloads. You may also want to think about using the -R option, which allows you to recursively download a directory.

Example script for multiple file transfers from T2 to T3

Using the above xrootd feature, you can create scripts for transferring multiple files from T2 to T3 or vice-versa.

once such example is available in a test script placed at /home/public/list.sh on T3

Deletion of data from your accounts.

Be carefull while using these instructions.

Checking the utilization of space from UI

after creating a valid proxy, use below command.

#/opt/lcg/bin/dpns-du /dpm/indiacms.res.in/home/cms/store/user/<your-username>

1. - Using xrootd.

create a proxy and log into the server

#xrd se01.indiacms.res.in

your prompt will change as below.

# <root://se01.indiacms.res.in:1094// >

Now go to your directory

#cd /cms/store/user/<your-username>

you will be in your directory

# <root://se01.indiacms.res.in:1094//cms/store/user/<your-username> >

Now to delete a directory

# rmdir <name-of-your-directory>


2 - using rfio command

following command can be used to recursively delete files

#rfrm -r /dpm/indiacms.res.in/home/cms/store/user/<your-username>/<directory-name>

-- TWikiAdminUser - 2014-07-26

Topic revision: r37 - 2017-07-07 - 13:24:59 - TWikiAdminUser
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback