Setting up New T3


we have successfully consolidated network for Indian HEP users over NKN by creating a L3VPN and connecting NKN to TIFR-LHCONE dedicated P2P network.

If your institue is part of collaboration and has active NKN connection then please contact TIFR group ( Prof Kajari Majumdar, Prof Gobinda Majumdar and Brij Kishor Jashal) for further details.

Once you become part of TIFR-LHCONE VPN, you will have seamless connectivity to all of WLCG.

Base requirement:

1 Gbps, if given public IPs are not available then private IPs should be NATed and should be able to connect to Internet.. ( internet via a proxy server will not work )

Hardware and configuration for Compute Cluster


Any generic X86-64 architecture based machine

Processor – Any Intel x86-64 based with N no of cores (N is no of logical cores enabled in your machines, after enabling Hyper Threading

Memory (RAM) – 2xN (N is no of cores) Or Min. 1.5 x N GB

Disk Space --- Job execution directory should be mounted separately on a separate partition - Recommend size =18GB xN or more

Minimum size =10GB x N

Operating System

Scientific Linux 6.7

Do basic installation of OS.

There are many turning parameters and nobes which needs to be done at OS and kernel level

Batch System

4. Batch system

There are multiple options, We have experience in following two

1. Torque/PBS ( implemented as T2_IN_TIFR)

2. HTcondor ( implemented at T3 at TIFR)

Let us start with HTcondor for a T3

There are many components of HTcondor which has been used across WLCG infrastructure, like pilot based submission and many others and after OSG, it is now being adopted and integrated in WLCG.

Before we start with our actual configuration and installation, I would suggest followingondoroduction and tutorials

HT condor week presentations

a. Installation:


Repositories to be configured:

1- OSG



4- Lcg-CA

5- SLC

(Remove all other repositories)

5. CVMFS and Frontier Squid

To get the CMSSW work effectively, we need to configure CVMFS on all the worker nodes and batch system head node. And to efficiently utilize the network bandwidth, it is important to setup a local frontier squid server.

a. Frontier Squid:

We at T2_IN_TIFR have frontier squid server as

I followed the following instructions to setup the squid.

There are various tests suggested at the last of the guide to test if your squid is working fine.

Network Requirements: - a multihued NIC node can be used. With a dedicated Public IP address to connect to outside work and a private IP address to connect local machines.

You host should have a valid FQDN and the Public IP should be able to do reverse and forward lookup from outside the network.

If you are facing problem at any point in configuration, please let me know.

b. CVMFS Mounting

Following instructions can be followed to onfigure cvmfs

Important note: - You should at least provision 20GB for cvmfs partition.

Let me know you you face any problem.

You first needs to install autofs

#yum install autofs

And after configuration restart the #autofs service.

Note- Check /var/log/messages for any errors and let me know if you come across any

. ##### Work Under progress #####

Topic revision: r2 - 2016-08-24 - 06:21:48 - TWikiAdminUser
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback