DLT Installation Instructions

Installing and configuring a Data Logistics Toolkit Storage Depot on a computer running CentOS and RHEL is a fairly straightforward. This page provides step by step instructions for a suitable installation. 

• Note: for further assistance, please email data-logistics@googlegroups.com

Prerequisites

1. DLT can be built on other flavors of UNIX, but this guide assumes CentOS/RHEL  6/7 or equivalent.
2. SUDO privileges or root access.
3. Familiarity with basic UNIX administration.

RPM install for CentOS 7 and RHEL 7

• Add the Data Logistics and EPEL repositories (for extra dependencies)

su -c 'rpm -Uvh http://data-logistics.org/yum/data-logistics-release-1-1.noarch.rpm'
sudo yum install epel-release

RPM install for CentOS 6 and RHEL 6

• Add the Data Logistics and EPEL repositories (for extra dependencies).

su -c 'rpm -Uvh http://data-logistics.org/yum/data-logistics-release-1-1.noarch.rpm'
sudo yum install epel-release

Update repository

In case you already have this repository installed, make sure that your repository cache is up to date.

my_host> yum clean all
my_host> yum check-update

DLT Meta Package

• If you are updating an existing depot and wish to keep your settings, make a backup of /etc/ibp.cfg for later.
• To get all the packages in the DLT, run:

my_host> yum install dlt

• This will install the IBP depot, Periscope monitoring, and Phoebus/XSP components.
• Individual component installation and configuration instructions are below.

DLT Tools installation

my_host> yum install dlt-tools

If the old 'lors' package is installed, you'll need to do 'yum erase lors' first. The dlt-tools package adds "LoRS" which provides the ability to upload and download to and from EODN by command line.

DLT Tools basically contains 2 Clis

• eodn_feed : Listen for new files and download
• eodn_download : Download the existing scenes specified in the list

Arguments

It contains the following arguments:

• -s , --scenes : comma separated list of scenes to look for
• -p, --productcode : comma separated list of products to look for
• -H , --host : host name for the exnode UNIS instance Eg: dev.crest.iu.edu
• -P, --port : the port for the exnode UNIS instance
• -v, --verbose : produce verbose output from the script
• -U, --visualhost : the hostname of the dlt-web client to display visual download information
• -r , --regex : Pass a regex to fetch by /exnodes?metadata.scene=reg=/<Argumnet> Download everything that matches the regex
• -S, --ssl : use ssl for socket connection

IBP server

Configure ibp.cfg

• If you have an existing configuration file, replace /etc/ibp.cfg and start the ibp_server:

my_host> /etc/init.d/ibp-server start
or
my_host> sudo service ibp-server start

• The ibp_server RPM comes with a Python configuration script that can create depot resources and install an initial ibp.cfg:

my_host> ibp_configure.py --help

• Running the script will edit the new blank ibp.cfg and will allow you to enter values interactively. 
• Once the script completes, configuration files will be written to the filesystem and any existing files will be backed up with a .ibp_configure_save extension.
• If you are installing a new depot, run the IBP configuration script (recommended: use the --host option to specify the DNS name of your server, if you have one)"​

my_host> sudo ibp_configure.py --host server.your.domain.edu

• The script will walk you through a number of configuration options, with suggested defaults, including:

  1. IBP server settings
  2. UNIS registration with a BLiPP monitoring option
  3. Phoebus Gateway specification

• The UNIS registration section includes SSL configuration as the global DLT UNIS instance requires that clients present valid certificates to perform queries. The default settings point to a certificate and key pair that are installed along with the RPM. Adjust these settings as necessary if you have different SSL requirements or are registering to a different UNIS service.
• Consult the IBP server configuration guide for more information on customizing the IBP server settings.

Adding Resources

If you did not specify your target storage area while running ibp_configure.py, or if you want to add additional storage, resource entries should be added reflecting storage resources available to the depot. The ideal configuration gives ibp direct access to block devices such as hard drives. Each drive this server will use should be mounted and added to the server configuration.

• Partition and mount each drive in the usual way.
• Create the IBP resource by running this command for each drive:

my_host> mkfs.resource <IBP_id (unique)> dir <path_to_drive> <path_to_drive_DB> 
<-b maximum_size_in_MB> <-d maximum duration for allocation in seconds>

Where:

Verify running ibp_server

• Confirm that the server has started correctly by checking log at /var/log/ibp-server.log
• You can check ibp_server stats using get_version command

my_host> get_version <interface_IP> <port_number>

Verify depot has registered with UNIS

• With the UNIS configuration defaults, the ibp_server will register itself with UNIS automatically. This makes the new depot instance discoverable within the DLT network and allows for status monitoring: http://dlt.crest.iu.edu/status

Open TCP Port

ibp_server must be able to accept packets on TCP IBP port (e.g., 6714). The following commands will open the port and keep it open after reboot.

my_host> sudo firewall-cmd --zone=public --add-port=6714/tcp --permanent

my_host> sudo firewall-cmd --reload

Running BLiPP to report network and IBP depot storage usage.

• Install BLiPP from repository if not already present. This package is included with the DLT meta-package install.

my_host> yum install blipp

• The ibp_configure.py script described above will create a BliPP configuration file if desired. Select 'Y' when prompted. When BLiPP monitoring is selected, a config file will be written to /etc/periscope/blipp_dlt.json by default.

• Start BLiPP:

my_host> sudo service blippd start

• Check that BLiPP has started correctly.

my_host> tail -f /var/log/blippd.log

• Check status of new depot here:

http://dlt.crest.iu.edu

UNIS (optional for local installations)

Installation and configuration

Refer Installation & configuration guide. Refer SSL configuration tutorial to enable SSL on UNIS.

Access SSL enabled UNIS

In order to access UNIS when protected by SSL, you need to install UNIS CA generated client certificate. Most of the modern browser, now, understand PKCS format. Obtain dlt-client.pfx file from dlt mailing list and import in your browser. Now try to access https://dlt.crest.iu.edu:9000/

Tips and Tricks for a healthy and reliable Depot

Auto start IBP on reboot.

my_host> sudo service ibp-server enable

Auto start BLiPP on reboot.

my_host> sudo service blipp enable

Server Clock:

Make sure your server keeps good time. One way to do thiis is to run ntpd:

my_host> sudo service ntpd start

Make sure it starts up again after a system reboot:

my_host> sudo service ntpd enable

Server Auto Restart after Power Outage:

Consider setting your server to return to its last state after power outage. In the BIOS select return to "Last state" for the behavior after loss of AC power