Enterprise Information Catalog (EIC) Installation

1 - About

Enterprise Information Catalog (EIC) uses the Catalog Service and other application services to bring together configured data assets in an enterprise and present a comprehensive view of the data assets and data asset relationships.

This installation occurs:

3 - Minimal topology information

3.1 - Cluster

You can deploy Enterprise Information Catalog either:

  • in an internal Hadoop distribution on Hortonworks (inside the installer) on the same machine
  • or external Hadoop distribution on Cloudera, Hortonworks, or Azure HDInsight

3.1.1 - Internal

  • The Enterprise Information Catalog installer creates an Informatica Cluster Service as an ISP service.
  • Enterprise Information Catalog uses Apache Ambari to manage and monitor the internal Hadoop cluster.

3.1.2 - External

Preparation:

We use the below information

  • <username> : the Informatica domain user
  • <ServiceClusterName> : the name of the service cluster (you enter it when you create the Catalog Service)

in HDFS before creating the Catalog Service:

/Informatica/LDM/<ServiceClusterName>
/user/<username>

Make the owner of the /Informatica/LDM/<ServiceClusterName> and /user/<username> directories.

3.2 - Services

3.3 - Domain

  • The Informatica domain is the administrative unit.
  • Enterprise Information Catalog requires a dedicated domain
  • EIC is installed within the Informatica domain

3.4 - Data Set

  • Small, Medium, Large, Default, and Demo data set sizes (Configured in Informatica Administrator using custom properties). You cannot change the data set size if you had selected a Demo data set size or if the data set size is smaller.
  • are classified based on the amount of metadata to process and the number of nodes used to process metadata

3.5 - Client

Enterprise Information Catalog contains the following client applications:

  • Informatica Administrator
  • Informatica Catalog Administrator
  • Enterprise Information Catalog search tool

3.6 - Repository

The types of repository is based on the type of data and metadata that it stores.

  • Domain configuration repository: A relational database that stores domain configuration and user information.
  • Model repository: A relational database that stores metadata created by Enterprise Information Catalog and application services to enable collaboration between the clients and services. Model repository also stores the resource configuration and data domain information.
  • Profiling warehouse: A relational database that stores profile results. Profile statistics form one part of the comprehensive metadata view that Enterprise Information Catalog provides.
  • Reference data warehouse: A relational database that stores data values for the reference table objects that you define in the Model repository. When you add data to a reference table, the Content Management Service writes the data values to a table in the reference data warehouse.

4 - Prerequisites

4.1 - SQL Server

  • The database
CREATE DATABASE EIC;
ALTER DATABASE EIC SET ALLOW_SNAPSHOT_ISOLATION ON;
ALTER DATABASE EIC SET READ_COMMITTED_SNAPSHOT ON;
USE EIC;
  • the user account name for the domain configuration repository
-- the database user account name for the domain configuration repository
CREATE LOGIN eic_dom WITH PASSWORD = 'pwd', DEFAULT_DATABASE = EIC;
CREATE USER eic_dom FOR LOGIN eic_dom;
CREATE SCHEMA eic_dom AUTHORIZATION eic_dom;
ALTER USER eic_dom WITH DEFAULT_SCHEMA = eic_dom;
 
-- Permission
EXEC sp_addrolemember 'db_ddladmin', 'eic_dom';
EXEC sp_addrolemember 'db_datawriter', 'eic_dom';
EXEC sp_addrolemember 'db_datareader', 'eic_dom';
 
-- Double ?
GRANT CONNECT TO eic_dom;
GRANT CREATE TABLE TO eic_dom;
GRANT CREATE VIEW TO eic_dom;
  • the user account name for the mrs repository
CREATE LOGIN eic_dom WITH PASSWORD = 'pwd', DEFAULT_DATABASE = EIC;
CREATE USER eic_dom FOR LOGIN eic_dom;
CREATE SCHEMA eic_dom AUTHORIZATION eic_dom;
ALTER USER eic_dom WITH DEFAULT_SCHEMA = eic_dom;
 
-- Permission
EXEC sp_addrolemember 'db_ddladmin', 'eic_dom';
EXEC sp_addrolemember 'db_datawriter', 'eic_dom';
EXEC sp_addrolemember 'db_datareader', 'eic_dom';
 
-- Double ?
GRANT CONNECT TO eic_dom;
GRANT CREATE TABLE TO eic_dom;
GRANT CREATE VIEW TO eic_dom;
  • The mrs database account name
CREATE LOGIN eic_mrs WITH PASSWORD = 'pwd', DEFAULT_DATABASE = EIC;
CREATE USER eic_mrs FOR LOGIN eic_mrs;
CREATE SCHEMA eic_mrs AUTHORIZATION eic_mrs;
GRANT CONNECT TO eic_mrs;
GRANT CREATE TABLE TO eic_mrs;
GRANT CREATE VIEW TO eic_mrs;
ALTER USER eic_mrs WITH DEFAULT_SCHEMA = eic_mrs;
EXEC sp_addrolemember 'db_ddladmin', 'eic_mrs';
EXEC sp_addrolemember 'db_datawriter', 'eic_mrs';
EXEC sp_addrolemember 'db_datareader', 'eic_mrs';
CREATE LOGIN eic_pwh WITH PASSWORD = 'pwd', DEFAULT_DATABASE = EIC;
CREATE USER eic_pwh FOR LOGIN eic_pwh;
CREATE SCHEMA eic_pwh AUTHORIZATION eic_pwh;
GRANT CONNECT TO eic_pwh;
GRANT CREATE TABLE TO eic_pwh;
GRANT CREATE VIEW TO eic_pwh;
ALTER USER eic_pwh WITH DEFAULT_SCHEMA = eic_pwh;
EXEC sp_addrolemember 'db_ddladmin', 'eic_pwh';
EXEC sp_addrolemember 'db_datawriter', 'eic_pwh';
EXEC sp_addrolemember 'db_datareader', 'eic_pwh';

4.2 - Hardware Minimum

For a simple topology (p25 and 28 of the installation doc):

The minimum system requirements for the Informatica Domain and Hadoop cluster on the same machine:

  • Disk Space: 75 GB
  • Memory(RAM): 32 GB
  • Number of CPU cores:: 16

The minimum system requirements for the Informatica domain if the Hadoop cluster is not on the Informatica domain machine:

  • Disk Space: 40 GB
  • Memory(RAM): 16 GB
  • Number of CPU cores:: 8

Temp: 8 GB of temporary disk space.

Example on Azure with a machine size of Standard_F8S (8 Cores, 16 Gb Memory, 128 Gb Disk, 284 Euro/month):

az.cmd vm create ^
    --resource-group myGroup ^
    --name INFA-EIC-01 ^
    --image RedHat:RHEL:7.3:latest ^
    --size Standard_F8s ^ 
    --authentication-type password ^
    --admin-username hi-adm ^
    --admin-password pwd ^
    --location westeurope

4.3 - HDFS

4.3.1 - Directory

Create the directory /Informatica/LDM/<ServiceClusterName>

If you do not specify a service cluster name, Enterprise Information Catalog considers DomainName_CatalogServiceName as the default value. You must then have the /Informatica/LDM/<DomainName>_<CatalogServiceName> directory in HDFS.

Create the directory

/Informatica/LDM/DOMAIN_EIC_01_CS_EIC_01

where:

  • DOMAIN_EIC will be the DomainName
  • and CS_EIC_01 will be the catalog service name
hadoop fs -mkdir /Informatica
hadoop fs -mkdir /Informatica/LDM
hadoop fs -mkdir /Informatica/LDM/DOMAIN_EIC_01_CS_EIC_01
hadoop fs -chmod -R 777 /Informatica/LDM/DOMAIN_EIC_01_CS_EIC_01
hadoop fs -chown -R powercenter:powercenter /Informatica
hadoop fs -mkdir /user/powercenter
hadoop fs -chown powercenter:powercenter /user/powercenter

5 - Installation

5.1 - Installation

  • Installer: Enterprise Information Catalog installer install the services.
  • When you install the Enterprise Information Catalog services on a machine, you install all the files for all services.
  • The first time you run the installer, you must create the domain. During the installation on the additional machines, you create worker nodes that you join to the domain.

5.1.1 - As machine admin

mkdir /tmp/infa
cd /tmp/infa
wget https://containerName.blob.core.windows.net/install/informatica_1020_server_linux-x64.tar
# As sudo other, you don't have any permissions
sudo su tar -xvf informatica_1020_server_linux-x64.tar
 
# Other installation files
wget https://containerName.blob.core.windows.net/install/ScannerBinaries.zip
mv ScannerBinaries.zip /tmp/infa/source
 
# Installation user
sudo useradd powercenter
sudo passwd powercenter
 
sudo chown -R powercenter.powercenter .
 
# place the key in the home
sudo mv license.key /home/powercenter/informatica/license.key
sudo chown powercenter.powercenter /home/powercenter/informatica/license.key
 
# resources limit
sudo vi  /etc/security/limits.conf
powercenter    hard   nofile    32000
powercenter    soft   nofile    3000
5.1.1.1 - Firewall

Firewalld

Add the ports:

  • Azure Firewall
az.cmd network nsg rule create ^
    --resource-group myGroup ^
    --nsg-name INFA-BDM-01NSG ^
    --name allow-infa-admin-hi ^
    --protocol tcp ^
    --priority 1021 ^
    --destination-port-range 6008 ^
    --source-address-prefixes publicIP
az.cmd network nsg rule create ^
    --resource-group myGroup ^
    --nsg-name INFA-BDM-01NSG ^
    --name allow-infa-admin-hi ^
    --protocol tcp ^
    --priority 1022 ^
    --destination-port-range 6005 ^
    --source-address-prefixes publicIP
  • As root, Red Hat Firewall
sudo firewall-cmd --zone=public --add-port=8443/tcp --permanent
sudo firewall-cmd --zone=public --add-port=6005-6009/tcp --permanent
sudo firewall-cmd --zone=public --add-port=6014-6114/tcp --permanent
sudo firewall-cmd --reload

5.1.2 - As powercenter

The install.sh and silentinstall.sh program are wrapper of the real installer. It validates the environment for the installer.

  • The install.sh will perform the following: To create a response file, you can add the -r option. See install.bin
./Server/install.bin -DINSTALL_MODE=CONSOLE -DINSTALL_TYPE=0
  • The silentinstall.sh will perform the following:
./Server/install.bin -i silent -DINSTALL_MODE=SILENT

A silent installation:

cd /tmp/infa
 
# for the installation directory
mkdir ~/informatica
mkdir ~/informatica/10.2
 
# For the keystore
mkdir ~/informatica/10.2/isp
mkdir ~/informatica/10.2/isp/config
mkdir ~/informatica/10.2/isp/config/keys
 
 
cp SilentInput.properties SilentInputBackup.properties

Example of modification of the property files for an hdInsight Cluster: The diff was made with winmerge. Tools > generate patch

ENABLE_USAGE_COLLECTION=1
LICENSE_KEY_LOC=/home/powercenter/informatica/license.key
USER_INSTALL_DIR=/home/powercenter/informatica/10.2
INSTALL_LDM=1
ACCEPT_ORACLE_LICENSE=1
HTTPS_PORT=
KEY_DEST_LOCATION=/home/powercenter/informatica/10.2/isp/config/keys
PASS_PHRASE_PASSWD=changeme
DB_TYPE=MSSQLServer
DB_UNAME=eic_dom
DB_PASSWD=changeme
SQLSERVER_SCHEMA_NAME=eic_dom
DB_SERVICENAME=EIC
DB_ADDRESS=msft-db-01:1433
DOMAIN_NAME=DOMAIN_EIC_01
DOMAIN_HOST_NAME=INFA-EIC-01
NODE_NAME=NodeEic01
DOMAIN_PORT=6005
DOMAIN_USER=Administrator
DOMAIN_PSSWD=changeme
DOMAIN_CNFRM_PSSWD=changeme
ADVANCE_PORT_CONFIG=1
MIN_PORT=6014
MAX_PORT=6114
TOMCAT_PORT=6006
AC_PORT=6007
SERVER_PORT=6008
AC_SHUTDWN_PORT=6009
CREATE_SERVICES=1
MRS_DB_TYPE=MSSQLServer
MRS_DB_UNAME=eic_mrs
MRS_DB_PASSWD=changeme
MRS_SQLSERVER_SCHEMA_NAME=eic_mrs
MRS_DB_SERVICENAME=EIC
MRS_DB_ADDRESS=msft-db-01:1433
MRS_SERVICE_NAME=MRS_EIC_01
DIS_SERVICE_NAME=DIS_EIC_01
DIS_PROTOCOL_TYPE=http
DIS_HTTP_PORT=8095
ASSOCIATE_PROFILE_CONNECTION=1
PWH_DB_TYPE=SQLServer
PWH_DB_UNAME=eic_pwh
PWH_DB_PASSWD=changeme
PWH_SQLSERVER_SCHEMA_NAME=eic_pwh
PWH_DB_SERVICENAME=EIC
PWH_DB_ADDRESS=msft-db-01:1433
PWH_DATA_ACCESS_CONNECT_STRING=jdbc:informatica:sqlserver://msft-db-01:1433;databaseName=EIC
LOAD_DATA_DOMAIN=1
CMS_SERVICE_NAME=CMS_EIC_01
CMS_HTTP_PORT=8105
CMS_DB_TYPE=SQLServer
CMS_DB_UNAME=eic_cms
CMS_DB_PASSWD=changeme
CMS_SQLSERVER_SCHEMA_NAME=eic_cms
CMS_DB_SERVICENAME=EIC
CMS_DB_ADDRESS=msft-db-01:1433
CMS_DATA_ACCESS_CONNECT_STRING=jdbc:informatica:sqlserver://msft-db-01:1433;databaseName=EIC
CLUSTER_HADOOP_DISTRIBUTION_TYPE=HortonWorks
IS_CLUSTER_SSL_ENABLE=false
CATALOGUE_SERVICE_NAME=CS_EIC_01
CATALOGUE_SERVICE_PORT=
CLUSTER_HADOOP_DISTRIBUTION_URL=https://clus-spark-01.azurehdinsight.net
CLUSTER_HADOOP_DISTRIBUTION_URL_USER=adm
CLUSTER_HADOOP_DISTRIBUTION_URL_PASSWD=changeme
CLUSTER_NAME=CLUS-SPARK-01
5.1.2.1 - Check

Bi starting the installer, you can check the prerequisites:

./install.sh
******************************************************************************************************
System Check Summary - Step 4 of 4
******************************************************************************************************
[ Type 'back' to go to the previous panel or 'help' to check the help contents for this panel or 'quit' to cancel the installation at any time. ]

Informatica Pre-Installation (i10PreInstallChecker) System Check Tool Results
[Pass] Disk Space: Available disk space is 15,548 MB. Sufficient for the Informatica installation.
[Pass] Processors: Available number of processors is 2. Sufficient for the Informatica installation.
[Pass] Physical Memory: Available physical memory is 16,416 MB. Sufficient for the Informatica installation.
[Pass] Temporary Space: Available temporary disk space is 15,548 MB. Sufficient for the Informatica installation.
[Pass] Ports: Port range is 6,005 - 6,009. All port numbers within the port range are available for the Informatica installation.
[Pass] Locale Environment Variable: The LANG environment variable is set to language en_US.UTF-8. The LC_ALL environment variable is set to language null. Sufficient for the Informatica installation.
[Pass] JRE_HOME Environment Variable: The JRE_HOME environment variable does not contain a value. Sufficient for the Informatica installation.
[Pass] File Descriptor Limits: The file descriptor limits per process is 32000. Sufficient for the Informatica installation.
[Pass] CREATE TABLE Privilege: The database user account has the CREATE TABLE privilege. The installer successfully created a database table.
[Pass] INSERT RECORD Privilage: The installer successfully inserted a record into database table.
[Pass] DELETE RECORD Privilage: The installer successfully deleted a record from database table.
[Pass] CREATE VIEW Privilege: The database user account has the CREATE VIEW privilege. The installer successfully created a database view.
[Pass] DROP VIEW Privilege: The database user account has the DROP VIEW privilege. The installer successfully dropped a database view.
[Pass] DROP TABLE Privilege: The database user account has the DROP TABLE privilege. The installer successfully dropped a database table.
[Pass] SQL Server READ COMMITTED Isolation Level: The SQL Server READ COMMITTED isolation level for the database is set to ON. Sufficient for the Informatica installation.
[Pass] SQL Server Case Sensitivity: The SQL Server instance is not case-sensitive. Sufficient for the Informatica installation.
[Information] Informatica Installation Directory: /home/powercenter
[Information] Informatica Starting Port Number: 6005
[Information] Database Type: SQLServer
[Information] Database User ID: hi_eic_dom
[Information] Database Host Name: hi-msft-db-01
[Information] Database Port Number: 1433
[Information] Database Service Name: hi_eic
[Information] Operating System: Operating system is Linux. Operating system version is 3.10.0-514.28.1.el7.x86_64.
[Information] RAM: The memory module size is 16,416 MB.
[Information] Virtual Memory: Virtual Memory is set to unlimited.
5.1.2.2 - Installation Silent
/tmp/infa/silentinstall.sh

Installation log are in the root installation directory. For instance, /home/powercenter/informatica/10.2/Informatica_10.2.0_InstallLog.log

OS detected is Linux

\***************************************************************************
\* Welcome to the Informatica 10.2.0 Server Installer.  *
\***************************************************************************



Configure the LANG and LC_ALL variables to generate the appropriate code pages and
create and connect to repositories and Repository Services.
Before you continue, read the following documents:
* Informatica 10.2.0 Installation Guide, Informatica Release Guide and Informatica Release Notes.
* B2B Data Transformation 10.2.0 Installation, Configuration Guide and Release Notes.

You can find the 10.2.0 documentation in the Product Documentation section at https://network.informatica.com/.
The installer requires Linux version 2.6.32-431 or later versions of the 2.6.32 series or version 3.10.0-0 or later versions of the 3.10.0 series.
The current operating system Linux version 3.10.0-514.
Current operating system meets minimum requirements.
-----------------------------------------------------------
Checking for an Informatica 10.2.0 installation.
Launching installer in silent mode ...
Installation Completed.

5.2 - Post Installation

5.2.1 - Environment variable

._bash_profile
export INFA_HOME=/home/powercenter/informatica/10.2/
 
# A pair of bin
export PATH=$PATH:$INFA_HOME/java/jre/bin/
export PATH=$PATH:$INFA_HOME/tomcat/bin/

5.2.2 - Start and Stop

  • Upload the below file to /tmp
infainit
#!/bin/sh
# chkconfig: 345 99 10
# description: Informatica auto start-stop script for the init system
 
INFA_OWNER=powercenter
export INFA_HOME=/home/powercenter/informatica/10.2
 
case "$1" in
    'start')
        # Start Informatica
        su - $INFA_OWNER -c "${INFA_HOME}/tomcat/bin/infaservice.sh startup"
        ;;
    'stop')
        # Stop Informatica
        su - $INFA_OWNER -c "${INFA_HOME}/tomcat/bin/infaservice.sh shutdown"
        ;;
esac
 
#
exit
  • As root
sudo su -
mv /tmp/infainit /etc/init.d/infa
chgrp powercenter /etc/init.d/infa
chmod 750 /etc/init.d/infa
chown powercenter /etc/init.d/infa
  • Symlink that gives the level and when to start the script (K=shutdown and S=startup)
ln -s /etc/init.d/infa /etc/rc.d/rc0.d/K01infa
ln -s /etc/init.d/infa /etc/rc.d/rc3.d/S99infa
ln -s /etc/init.d/infa /etc/rc.d/rc5.d/S99infa

Then:

service infa start
service infa stop

Output example after startup:

Starting Informatica services on node 'NodeEic01'
Using CURRENT_DIR:     /home/powercenter/informatica/10.2/tomcat/bin
Using INFA_HOME:       /home/powercenter/informatica/10.2
Using System log directory :   /home/powercenter/informatica/10.2/logs/NodeEic01

5.2.3 - Log

  • Admin: ${INFA_HOME}/logs/NodeEic01/services/AdministratorConsole/_AdminConsole_jsf.log
  • MRS: ${INFA_HOME}/logs/NodeEic01/services/ModelRepositoryService
  • DIS ${INFA_HOME}/10.2/logs/NodeEic01/services/DataIntegrationService/DIS_EIC_01_jsf.log
  • CMS ${INFA_HOME}/logs/NodeEic01/services/ContentManagementService/CMS_EIC_01_jsf.log

5.2.4 - Configuring the Catalog Service for Azure HDInsight

(p123) See also the Hadoop doc for the whole story

After you create the Catalog Service, you need to define in the custom properties:

Example:

5.2.4.1 - Storage Account Key

LdmCustomOptions.deployment.azure.account.key: The key to authenticate the Catalog Service to connect to Azure storage account. The value of the Azure storage account key might be:

  • encrypted. Ie the value from fs.azure.account.key.<storage account name> property in core-site.xml file present in the Azure HDInsight cluster. Location /etc/hadoop/ is encrypted
  • or non encrypted (In the azure portal)

If the key is in encrypted format, add the decrypt script in the LdmCustomOptions.deployment.azure.key.decryption.script.path. It use the key certificate to decrypt it. You must:

  • copy the decrypt shell script and key certificate file to the (same path as cluster machine) domain machine before enabling Catalog Service.
  • Make sure that you maintain the path in the Azure HDInsight cluster machine for the copied files in the domain machine.

The value for the property is the location of the decrypt shell script. For example, /usr/lib/hdinsight-common/scripts/decrypt.sh. The key certificate file, key_decryption_cert.prv, is present in the /usr/lib/hdinsight-common/certs/key_decryption_cert.prv directory of Azure HDInsight cluster.

5.2.4.2 - File System URI
  • LdmCustomOptions.deployment.hdfs.default.fs: Address of the WASB storage account to which the Catalog Service must connect. The address includes the WASB storage container name with the storage account name. The value for the property is the complete WASB address with the container and storage account names. You can retrieve the value for the property from the fs.defaultFS property in the core-site.xml file present in the Azure HDInsight cluster. Example: wasb://[email protected]

Copy the file of the decrypt files:

  • /usr/lib/hdinsight-common/scripts/decrypt.sh
  • /usr/lib/hdinsight-common/certs/key_decryption_cert.prv

from the cluster head to the EIC server in the /tmp directory

Then on the EIC server:

cd /usr/lib
mkdir hdinsight-common
mkdir hdinsight-common/scripts
mkdir hdinsight-common/certs
mv /tmp/azure/decrypt.sh ./hdinsight-common/scripts
mv /tmp/azure/key_decryption_cert.prv ./hdinsight-common/certs
chmod +x ./hdinsight-common/scripts/decrypt.sh
chown powercenter:powercenter ./hdinsight-common/scripts/decrypt.sh

6 - Upgrade to 10.2 update 1

As the installation user (ie powercenter)

export EIC_INSTALL_DIR=/extradrive/update1
cd $EIC_INSTALL_DIR
wget https://YouContainerName.blob.core.windows.net/install/Informatica_1020U1_server_linux-x64.tar
tar -xvf Informatica_1020U1_server_linux-x64.tar
service infa stop
cd $EIC_INSTALL_DIR
./installEBF.sh
Mar 5, 2018 12:05:53 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: start time is - Mon Mar 05 12:05:53 UTC 2018
Mar 5, 2018 12:05:53 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: rollbackAction = 0
Mar 5, 2018 12:05:53 PM com.informatica.installer.logging.InstallLogger logInfo
INFO:  destLoc = /home/powercenter/informatica/10.2
Mar 5, 2018 12:05:53 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: creating - /home/powercenter/informatica/10.2/services/CatalogService/ScannerBinaries/CustomDeployer/upgradeScannerDeployer.sh
Mar 5, 2018 12:05:53 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: no of files created 1
Mar 5, 2018 12:05:53 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//plugins/acplugins/com.infa.products.ldm.adminplugins.ldm-service-10.2.0.599.490-SNAPSHOT.jar  to                                                                             /home/powercenter/informatica/10.2/EBFs/EBF-1020U1//plugins/acplugins/com.infa.products.ldm.adminplugins.ldm-service-10.2.0.599.490-SNAPSHOT.jar_EBF-                                                                           1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:05:53 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//plugins/acplugins/com.infa.products.ldm.adminplugins.ldm-service-10.2.0.599.490-SNAPSHOT.jar  wi                                                                           th /extradrive/update1/EBFs/plugins/acplugins/com.infa.products.ldm.adminplugins.ldm-service-10.2.0.599.490-SNAPSHOT.jar_bak
Mar 5, 2018 12:05:57 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//services/CatalogService/ldmadmin.war  to  /home/powercenter/informatica/10.2/EBFs/EBF-1020U1//ser                                                                           vices/CatalogService/ldmadmin.war_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:05:57 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//services/CatalogService/ldmadmin.war  with /extradrive/update1/EBFs/services/CatalogService/ldma                                                                           dmin.war_bak
Mar 5, 2018 12:05:58 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: deleting - /home/powercenter/informatica/10.2//services/CatalogService/ldmadmin
Mar 5, 2018 12:06:14 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//services/CatalogService/access.war  to  /home/powercenter/informatica/10.2/EBFs/EBF-1020U1//servi                                                                           ces/CatalogService/access.war_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:06:14 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//services/CatalogService/access.war  with /extradrive/update1/EBFs/services/CatalogService/access                                                                           .war_bak
Mar 5, 2018 12:06:15 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: deleting - /home/powercenter/informatica/10.2//services/CatalogService/access
Mar 5, 2018 12:06:21 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//services/CatalogService/ingest.jar  to  /home/powercenter/informatica/10.2/EBFs/EBF-1020U1//servi                                                                           ces/CatalogService/ingest.jar_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:06:21 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//services/CatalogService/ingest.jar  with /extradrive/update1/EBFs/services/CatalogService/ingest                                                                           .jar_bak
Mar 5, 2018 12:06:31 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//services/CatalogService/ldmcatalog.war  to  /home/powercenter/informatica/10.2/EBFs/EBF-1020U1//s                                                                           ervices/CatalogService/ldmcatalog.war_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:06:31 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//services/CatalogService/ldmcatalog.war  with /extradrive/update1/EBFs/services/CatalogService/ld                                                                           mcatalog.war_bak
Mar 5, 2018 12:06:32 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: deleting - /home/powercenter/informatica/10.2//services/CatalogService/ldmcatalog
Mar 5, 2018 12:06:34 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//services/CatalogService/Binaries/slider-bin.tar.gz  to  /home/powercenter/informatica/10.2/EBFs/E                                                                           BF-1020U1//services/CatalogService/Binaries/slider-bin.tar.gz_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:06:34 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//services/CatalogService/Binaries/slider-bin.tar.gz  with /extradrive/update1/EBFs/services/Catal                                                                           ogService/Binaries/slider-bin.tar.gz_bak
Mar 5, 2018 12:06:34 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//services/CatalogService/Binaries/Deployment.sh  to  /home/powercenter/informatica/10.2/EBFs/EBF-1                                                                           020U1//services/CatalogService/Binaries/Deployment.sh_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:06:34 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//services/CatalogService/Binaries/Deployment.sh  with /extradrive/update1/EBFs/services/CatalogSe                                                                           rvice/Binaries/Deployment.sh_bak
Mar 5, 2018 12:06:34 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//services/CatalogService/ScannerBinaries/Catalog_Agent_Profiles_10_2.zip  to  /home/powercenter/in                                                                           formatica/10.2/EBFs/EBF-1020U1//services/CatalogService/ScannerBinaries/Catalog_Agent_Profiles_10_2.zip_EBF-1020U1_2018-03-05_12-05-53 with status as                                                                            true
Mar 5, 2018 12:06:34 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//services/CatalogService/ScannerBinaries/Catalog_Agent_Profiles_10_2.zip  with /extradrive/update                                                                           1/EBFs/services/CatalogService/ScannerBinaries/Catalog_Agent_Profiles_10_2.zip_bak
Mar 5, 2018 12:06:39 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//services/CatalogService/ScannerBinaries/Catalog_Agent_10_2.zip  to  /home/powercenter/informatica                                                                           /10.2/EBFs/EBF-1020U1//services/CatalogService/ScannerBinaries/Catalog_Agent_10_2.zip_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:06:39 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//services/CatalogService/ScannerBinaries/Catalog_Agent_10_2.zip  with /extradrive/update1/EBFs/se                                                                           rvices/CatalogService/ScannerBinaries/Catalog_Agent_10_2.zip_bak
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//services/CatalogService/ScannerBinaries/scanner.sh  to  /home/powercenter/informatica/10.2/EBFs/E                                                                           BF-1020U1//services/CatalogService/ScannerBinaries/scanner.sh_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//services/CatalogService/ScannerBinaries/scanner.sh  with /extradrive/update1/EBFs/services/Catal                                                                           ogService/ScannerBinaries/scanner.sh_bak
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//isp/bin/plugins/ldm/com.infa.products.ldm.service.isp.cli.jar  to  /home/powercenter/informatica/                                                                           10.2/EBFs/EBF-1020U1//isp/bin/plugins/ldm/com.infa.products.ldm.service.isp.cli.jar_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//isp/bin/plugins/ldm/com.infa.products.ldm.service.isp.cli.jar  with /extradrive/update1/EBFs/isp                                                                           /bin/plugins/ldm/com.infa.products.ldm.service.isp.cli.jar_bak
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: renaming - /home/powercenter/informatica/10.2//isp/bin/plugins/ihs/com.infa.products.ihs.isp.cli.jar  to  /home/powercenter/informatica/10.2/EBF                                                                           s/EBF-1020U1//isp/bin/plugins/ihs/com.infa.products.ihs.isp.cli.jar_EBF-1020U1_2018-03-05_12-05-53 with status as true
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: replacing - /home/powercenter/informatica/10.2//isp/bin/plugins/ihs/com.infa.products.ihs.isp.cli.jar  with /extradrive/update1/EBFs/isp/bin/plu                                                                           gins/ihs/com.infa.products.ihs.isp.cli.jar_bak
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: EBF History path is/home/powercenter/informatica/10.2//server/bin/ebfHistory.info
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: EBF History path is/home/powercenter/informatica/10.2//server/bin/ebfHistory.info
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: deleting** - /home/powercenter/informatica/10.2/services/AdministratorConsole/webapps/administrator/console/js/infa/InfaHadoopService
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: File does not exist for delete - /home/powercenter/informatica/10.2/services/AdministratorConsole/webapps/administrator/console/js/infa/InfaHado                                                                           opService
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: deleting** - /home/powercenter/informatica/10.2/services/AdministratorConsole/webapps/administrator/console/js/infa/CatalogService
Mar 5, 2018 12:06:40 PM com.informatica.installer.logging.InstallLogger logInfo
INFO: End time is - Mon Mar 05 12:06:40 UTC 2018
./Input.properties: line 17: Valid.Folder.List=isp: command not found
./Input.properties: line 17: server: command not found
./Input.properties: line 17: tomcat: command not found
./Input.properties: line 17: services: command not found
./Input.properties: line 17: tools: command not found
./Input.properties: line 17: plugins: command not found
Succesfully modified scannerId
Creating temp file

Modified scannerDeployer.xml
removed temp File upgrade.tmp
  • Due to the bug below, create this extra install script and run it
extraInstall.sh
DEST_DIR=/home/powercenter/informatica/10.2
 
cd $DEST_DIR/services/CatalogService/ScannerBinaries/CustomDeployer
chmod 755 $DEST_DIR/services/CatalogService/ScannerBinaries/CustomDeployer/upgradeScannerDeployer.sh
sh $DEST_DIR/services/CatalogService/ScannerBinaries/CustomDeployer/upgradeScannerDeployer.sh
Succesfully modified scannerId
Creating temp file

Modified scannerDeployer.xml
removed temp File upgrade.tmp
  • Start the services again
service infa start

7 - Annexes

7.1 - Uninstall

  • Drop Informatica
sudo service infa stop
cd ~/informatica/10.2/Uninstaller_Server/
./uninstaller
  • Drop the database
DROP DATABASE EIC;

7.2 - install.bin

INSTALL_HOME/Server/install.bin -?
Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...
Usage: install [-f <path_to_installer_properties_file> | -options]
            (to execute the installer)

where options include:
    -?          show this help text
    -h          show this help text
    -help       show this help text
    --help      show this help text
    -i [swing | console | silent]
            specify the user interface mode for the installer
    -D<name>=<value>
            specify installer properties
    -r <path_to_generate_response_file>
            Generates response file.
JVM heap size options are only applicable to Installers
    -jvmxms <size>
            Specify JVM initial heap size.
    -jvmxmx <size>
            Specify JVM maximum heap size.
The options field may also include the following in case of uninstaller
if it is enabled for Maintenance Mode
    -add <feature_name_1> [<feature_name_2 ...]
            Add Specified Features
    -remove <feature_name_1> [<feature_name_2 ...]
            Remove Specified Features
    -repair
            Repair Installation
    -uninstall
            Uninstall

notes:
    1. the path to the installer properties file may be either absolute,
       or relative to the directory in which the installer resides.
    2. if an installer properties file is specified and exists, all other
       command line options will be ignored.
    3. if a properties file named either 'installer.properties' or
       <NameOfInstaller>.properties resides in the same directory as the
       installer, it will automatically be used, overriding all other command
       line options, unless the '-f' option is used to point to another valid
       properties file.
    4. if an installer properties file is specified but does not exist, the
       default properties file, if present, will be used.  Otherwise, any
       supplied command line options will be used, or if no additional
       options were specified, the installer will be run using the default
       settings.

7.3 - Test cluster access

curl --basic --user user:pwd https://clus-spark-01.azurehdinsight.net

8 - More

9 - Support

9.1 - Error decrypting CMS structure

On an Azure cluster, when launching EIC, you may see this error in LDM.log

org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.KeyProviderException: ExitCodeException exitCode=4: Error decrypting CMS structure
140293906716320:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:604:

	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:938)
	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:438)
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1048)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

It seems that BDM still want the azure storage account key in a encrypted format.

You get this kind of error when the key cannot be decrypted. Example:

/usr/lib/hdinsight-common/scripts/decrypt.sh theKeyFoundInCoreSiteXml 
Error decrypting CMS structure
140656402700192:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:604:

Normally, you would get the key of your storage account that you can see on the Azure Portal.

9.2 - Cleaning a start of EIC

9.2.1 - Cleaning Informatica

On the EIC machine as the installation user (ie powercenter)

  • INFA_HOME verification
echo $INFA_HOME
/home/powercenter/informatica/10.2
  • Suppress temporary files
rm -r -f $INFA_HOME/10.2/logs/NodeEic01/services/CatalogService/CS_EIC_01
rm -r $INFA_HOME/tomcat/temp/CS_EIC_01

9.2.2 - Cleaning Cluster

On the cluster:

zookeeper-client -server zkNode:2181
rmr /Informatica
rmr /registry
rmr /services
dit/powercenter/eic.txt · Last modified: 2018/10/09 13:50 by gerardnico