InfiniDB®

Enterprise Manager

Guide

 

 

 

 

 

 

 

 

 

                             

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Release:  4.6

Document Version:  4.6-1


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

InfiniDB Enterprise Manager Guide

July 2014

Copyright © 2014 InfiniDB Corporation. All Rights Reserved.

 

InfiniDB, the InfiniDB logo and any other product or service names or slogans contained in this document are trademarks of InfiniDB and its suppliers or licensors, and may not be copied, imitated or used, in whole or in part, without the prior written permission of InfiniDB or the applicable trademark holder.

 

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of InfiniDB.

 

InfiniDB may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering  subject matter in this document. Except as expressly provided in any written license agreement from InfiniDB, the furnishing of this document does not give you any license to these patents, trademarks copyrights, or other intellectual property. The information in this document is subject to change without notice. InfiniDB shall not be liable for any damages resulting from technical errors or omissions which may be present in this document, or from use of this document.


Contents

1       Introduction. 4

1.1         Audience. 4

1.2         List of Documentation. 4

1.3         Obtaining documentation. 4

1.4         Documentation feedback. 4

1.5         Additional resources. 4

2       InfiniDB Enterprise Manager Overview. 5

2.1         What is Enterprise Manager 5

2.1.1      Major functional categories. 5

2.1.2      Components/Roles. 5

3       Installation of Enterprise Manager 6

3.1         Prerequisites. 6

3.1.1      General Server Requirements. 6

3.1.2      EM Server Requirements. 6

3.2         Enterprise Manager Installation. 7

3.2.1      RPM Installation. 7

3.2.2      DEB Installation. 7

3.2.3      Binary Installation. 7

3.2.4      Accessing Enterprise Manager 7

4       Upgrading Enterprise Manager 9

5       Provisioning. 10

5.1         New Cluster Installation. 10

5.1.1      Preparation. 10

5.1.2      Install 10

5.2         Attach Existing Cluster 15

5.3         Detach Cluster 18

5.4         Management 19

5.4.1      Cluster Summary. 19

5.4.2      Cluster Configuration. 20

5.4.3      Monitoring. 21

6       Removal of Enterprise Manager 25

6.1         Saving off configuration/data files. 25

6.2         EM Removal 25

6.2.1      General 25

6.2.2      EM Host 25

6.2.3      OAM Host 25

7       Appendix A. 26

7.1.1      Logging. 26

 


1         Introduction

This guide includes information for installing, configuring, maintaining, performance tuning, and troubleshooting InfiniDB through the Enterprise Manager.

 

The content in this guide is only available with InfiniDB Enterprise.

1.1      Audience

This guide is intended for database administrators and IT administrators who are responsible for setting up, maintaining and managing InfiniDB.

1.2      List of Documentation

The InfiniDB Database Platform documentation consists of several guides intended for different audiences. The documentation is described in the following table:

 

Document

Description

InfiniDB Administrator’s Guide

Provides detailed steps for maintaining InfiniDB.

InfiniDB Apache HadoopTM Configuration Guide

Installation and Administration of an InfiniDB for Apache Hadoop system.

InfiniDB Concepts Guide

Introduction to the InfiniDB analytic database.

InfiniDB Minimum Recommended Technical Specifications

Lists the minimum recommended hardware and software specifications for implementing InfiniDB.

InfiniDB Installation Guide

Contains a summary of steps needed to perform an install of InfiniDB.

InfiniDB Multiple UM Configuration Guide

Provides information for configuring multiple User Modules.

InfiniDB SQL Syntax Guide

Provides syntax native to InfiniDB.

Performance Tuning for the InfiniDB Analytics Database

Provides help for tuning the InfiniDB analytic database for parallelization and scalability.

InfiniDB Windows Installation and Administrator’s Guide

Provides information for installing and maintaining InfiniDB for Windows.

1.3      Obtaining documentation

These guides reside on our http://www.infinidb.co website.  Contact support@infinidb.co for any additional assistance.

1.4      Documentation feedback

We encourage feedback, comments, and suggestions so that we can improve our documentation. Send comments to support@infinidb.co along with the document name, version, comments, and page numbers.

1.5      Additional resources

If you need help installing, tuning, or querying your data with InfiniDB, you can contact support@infinidb.co.

 


2         InfiniDB Enterprise Manager Overview

2.1      What is Enterprise Manager

The Enterprise Manager is a web-based administration and management console that assists in the installation, configuration, administration and monitoring of InfiniDB.  It allows a user to easily manage one or more InfiniDB clusters.

 

2.1.1    Major functional categories

The Enterprise Manager provides the following capabilities for InfiniDB:

 

2.1.2    Components/Roles

 


3         Installation of Enterprise Manager

This chapters describes the prerequisites needed and how to install the InfiniDB Enterprise Manager software.

3.1      Prerequisites

3.1.1    General Server Requirements

Before installing Enterprise Manager (EM), the following must exist or be performed:

 

vi /etc/sudoers

#Defaults requiretty

3.1.2    EM Server Requirements

In addition to the general requirements, these capabilities need to be present on the node where the Enterprise Manager is installed. The Enterprise Manager can be installed on any node whether or not it is also a PM or UM in the InfiniDB cluster.  In most installations in is appropriate to deploy the EM on the primary UM node (UM1).

 

Example commands for use on a CentOS system:

·         YUM commands to get a node up to the levels needed:

 

sudo yum install python-setuptools

sudo yum install java-1.7.0-openjdk

sudo easy_install pip

sudo pip install virtualenv

 

Note:  There will be the capability to designate a non-InfiniDB node for the Enterprise Manager in the future.

3.2      Enterprise Manager Installation

Enterprise Manager comes in various packages to allow different types of installs:

 

·      RPM Install for Single/Multiple Server

·      DEB Install for Single/Multiple Server

·      Binary Install for Single/Multiple Server

3.2.1    RPM Installation

Download and install the RPM for the InfiniDB Enterprise Manager:

 

rpm -ivh infinidb-entmgr-release#.el6.x86_64.rpm

 

After the installation of the RPM, a service named infinidb-em will be made and started.

3.2.2    DEB Installation

Download and install the DEB for the InfiniDB Enterprise Manager:

 

dpkg -i infinidb-entmgr-release#.el6.amd64.deb

 

After the installation of the DEB, a service named infinidb-em will be made and started.

3.2.3    Binary Installation

Download and unpack the tarball for InfiniDB Enterprise Manager, which will create the /opt/infinidb/em directory.

 

# tar -zxvf infinidb-em-release#.tar.gz

 

From the /opt/infinidb/em directory, run the install script install.sh.  After the installation , a service named infinidb-em will be made and started.

3.2.4    Accessing Enterprise Manager

In order to access the Enterprise Manager, the Enterprise Manager server should be addressable from the browser.  You may need to modify the hosts file on the user machine to add something similar to the following information:

 

10.100.1.167 EMServerName

 

In addition, you may need to allow the following ports to any firewall software running on the user machine:

 

·         EM Server - 9090

·         OAM Server - 8180

 

Once set up, Enterprise Manager can be accessed in a browser using the hostname or ipaddress.  Example based on the above hosts information:

 

http://EMServerName:9090  or http://10.100.1.167:9090


4         Upgrading Enterprise Manager

The recommendation is to remove the existing installation (See “Removal of Enterprise Manager”) and perform a new installation (See ”Enterprise Manager Installation”).

 

You can remove only the EM components on the node the EM was installed on and the installation will work. The components on the remote nodes did not change in the build and are compatible.

 

The upgrade procedure is being worked on currently to alleviate the manual steps described above of removal and re-installation.

 


5         Provisioning

The Enterprise Manager has the ability to perform new cluster installations or attach to existing installations of InfiniDB.   The following sections describe each case of installation.  These steps may be performed for each cluster that you want Enterprise Manager to manage and monitor.

 

Note:  Alarms will not be enabled when performing an install with the Enterprise Manager.

5.1      New Cluster Installation

Follow these steps for creating a new InfiniDB cluster through Enterprise Manager:

5.1.1    Preparation

 

5.1.2    Install

On the Home/Clusters page, Select the Provision Cluster button and select “New” (if there are no clusters listed, you may also choose the “New” link in the list):

 

 

The Provisioning Cluster page is displayed to enter information about the new cluster:

 

 

Inputs:

o   Cluster Name - Name to use for the cluster. This is the name the EM will display and refer to.  Must be a unique name within EM (example above – test_infinidb_cluster).

o   Cluster Hosts – A list of all the nodes to be used for the cluster.  Hosts entries may go on separate lines

o   User - The user to perform the provisioning as. Must have passwordless SSH for any user.  If non-root, must have passwordless sudo. Can use the root user.

o   Key - The SSH Key that has been setup for the nodes for passwordless SSH.

 


Once the Continue button is selected, EM gathers information about the cluster and presents the following page:

 

 

 

The specifics about the cluster and hosts can be configured here.  Information on the Cluster tab:

 

 

Information on the Configuration tab.  Currently this information is defaulted based on the host information and may be changed for specific tuning.

 

 

 

·         TotalUmMemory – Constrains the maximum amount of memory available for managing intermediate results for join, aggregations and select operations on the User Module.  This setting is shared across all queries active on a User Module.  Expressed as 1-100% or expressed in bytes such as 1G.

·         LocalQuery – Defines whether the local query interface for PMs will be enabled.  This allows queries to be limited to the local PM instead of querying all PMs in the cluster.

·         DBRootsPerPm – Defines how many DB Roots per PM will be allocated.  Setting the value to 1 with 4 PMs defined will result in 4 DB Roots.

·         NumBlocksPct – Defines the amount of memory dedicated to the data buffer cache within each Performance Module. The setting is expressed as a percentage of a Performance Module's total memory. Valid values are 1 through 100.

·         PmMaxMemorySmallSide - Defines the maximum size of a small side join result set for a PM join. If the size of the data set is smaller than the PmMaxMemorySmallSide, the data will be sent to the PM(s) to allow creation of a distributed hash map. Otherwise, it will be created on the UM. The setting is individual to each small side result set.  Expressed as 1-100% or expressed in bytes such as 64M.

·         VersionBufferFileSize - This setting defines the amount of disk per DBRoot dedicated to the version buffer. The version buffer is used for rollbacks of DML statements. Valid value is 1G for 1 GB, 2G for 2GB, etc.  Minimum value is 1G.

·         CoreFileFlag - Determines whether core files will be produced upon abnormal termination of InfiniDB processes. Valid value is y or n.

 

 

Once the Attach button is selecting, EM will proceed with the provisioning of the configuration.  Once completed, the cluster will appear in the Cluster list:

 

 

 

On this page the following options are available:

 

 


5.2      Attach Existing Cluster

Follow these steps for attaching an existing InfiniDB cluster to Enterprise Manager:

 

The EM can attach to any installed InfiniDB instance version 4.5.1 or later as long as it’s on a supported platform:

 

Click Add -> Existing

Provisioning Existing Cluster screen is displayed:

 

 

Inputs:

 


Example fields entered:

 

 


Click continue and the server will connect to the node and discover the cluster. A report of the cluster will come back:

 

 

For attaching to a cluster, the OAM Server will need to be designated. This can be any node of the system.  Recommendation is to use UM1 as the OAM Server. Here in the future we will support a separate node to be in the cluster that can be the EM / OAM Server node only and not affect the UM / PM nodes for database operations.

 

Should a node have an issue, such as passwordless SUDO missing, there will be a message on the node and the status of the node will be Offline. Fix the issue on the node and hit back and continue to refresh the report.

 


Click Attach to provision the EM components to the system:

 

Installation Successful:

 

The Cluster should appear in the cluster list now.

5.3      Detach Cluster

Select the cluster to detach and click the delete icon ( ).  This will remove the cluster from being managed by the EM.  Detaching a cluster will have the telemetry removed for that cluster, but the installed components will remain.  Follow the uninstallation steps to remove software changes.

 


5.4      Management

The Enterprise Manager has the capability to manage several key areas for InfiniDB Cluster.  Selecting the cluster name navigates to the Cluster Details page:

 

5.4.1    Cluster Summary

Information about the cluster is collected on a 10 second poll by the EM to retrieve the status of the system along with a list of any alarms and locks present.

5.4.1.1  Status

Status is shown on a per module basis along with System. This is the same information that can be seen by calling calpontConsole getsystemstatus.

 

5.4.1.2  Alarms

List of the active alarms on the system.  This is the same information that can be seen by calling calpontConsole getactivealarms.

5.4.1.3  Locks

List of active table locks on the system.  This is the same information that can be seen by calling calpontConsole gettablelocks.

 

5.4.2    Cluster Configuration

The Enterprise Manager can manage certain key configuration parameters for each type of InfiniDB node.

 

5.4.2.1  PM

PmMaxMemorySmallSide

This setting defines the maximum size of a small side join result set for a PM join. If the size of the data set is smaller than the PmMaxMemorySmallSide, the data will be sent to the PM(s) to allow creation of a distributed hash map. Otherwise, it will be created on the UM. The setting is individual to each small side result set.

Expressed as 1-100% or expressed in bytes such as 64M.

 

NumBlocksPct

This setting defines the amount of memory dedicated to the data buffer cache within each

Performance Module. The setting is expressed as a percentage of a Performance Module's total

memory. Valid values are 1 through 100.

 

VersionBufferFileSize

This setting defines the amount of disk per DBRoot dedicated to the version buffer. The version buffer is used for rollbacks of DML statements. Valid value is 1G for 1 GB, 2G for 2GB, etc.  Minimum value is 1G.

 

CoreFileFlag

This setting determines whether core files will be produced upon abnormal termination of InfiniDB processes. Valid value is y or n.

5.4.2.2  UM

TotalUmMemory

This setting constrains the maximum amount of memory available for managing intermediate results

for joins, aggregations, and set operations on the User Module. This setting is shared across all queries

active on a User Module. Expressed as 1-100% or expressed in bytes such as 1G.

 

5.4.3    Monitoring

The Enterprise Manage can monitor key memory and statistical elements of an InfiniDB cluster.

 

5.4.3.1  System Utilization

Each node of the cluster will be displayed at the same time for the metric chosen. The metrics available are:

 

Category

Metric

CPU

User

System

Idle

Memory

Free

Used

Buffered

Cached

Network

Bytes

Packets

Errors

Storage

Free

Used

Reserved

Disk

Bytes

OPS

Merged

Time

Swap

Free

Used

Cached

IO

In cases where there are multiple instances of a metric, such as CPU with a multi core, the metric has been

aggregated to the system.

 

The metrics have been normalized to a 0-100 scale when possible to provide similar scale across nodes for viewing.  Network and Disk will be displayed as absolute values as it cannot be determined what the range of values could be. 

 

The type of graph being shown is a horizon graph. The range of values for the metric is divided into 4 bands and as the value reaches the top of the graph, it will cycle back at the bottom and use a darker intensity of the coloring.  This allows seeing the groups of nodes as a heat map like graph as you can quickly identify the intensity levels based on coloring. In the cases of the normalized metrics, the coloring will always be based on the 0-100 range, while for the absolute metrics, the range will be dynamically determined based on the data returned for the time range.

 

The difference granularity selections are:

 

By adjusting the granularity you can change the time range of the window seen (how far back in time from real

time).

 

e.g. Changing from 10s to 30s when having a resolution of 1440 pixels would change the time range window from 4 hours to 12 hours.

 

By default it is 10s, which is the same value as the granularity that is stored server side (no aggregations are needed).

 

In order to aggregate the metrics to the granularity selected, the following aggregates are applied to the metrics:

Category

Metric

Aggregation

CPU

User

System

Idle

AVG

AVG

AVG

Memory

Free

Used

Buffered

Cached

AVG

AVG

AVG

AVG

Network

Bytes

Packets

Errors

SUM

SUM

SUM

Storage

Free

Used

Reserved

AVG

AVG

AVG

Disk

Bytes

OPS

Merged

Time

SUM

SUM

SUM

SUM

Swap

Free

Used

Cached

IO

AVG

AVG

AVG

SUM

 

 

5.4.3.2  Jobs

The telemetry data for queries and loads can be seen under the Job tab. This data originates from InfiniDB and is

sent to the EM as they occur.

 

Each type can be seen in a similar manner of a summary table that provides sorting, filtering, and paging

capabilities.  You may select the Details link at the right to drill down into the details of the telemetry.

 

5.4.3.3  Paging

The number of items per page can be changed by the Item Per Page on the bottom right, the available options are 10, 25 and 50.

 

As multiple pages of data occur, the page navigation on the bottom left will be available.

5.4.3.4  Sorting

Query Telemetry supports sorting by all columns except Note.  Import Telemetry supports sorting by all columns.

 

With the explicit sorting defined by the user a secondary implicit sort of StartTime and ID is applied to give

deterministic ordering of the result set. This occurs when multiple objects have the same value, the secondary sort will guarantee the same result ordering each time the query is ran.

 

e.g. Sorting by Progress will typically result in multiple items having the same value (such as 100), the items will then be sorted by Start Time and then by ID.

5.4.3.5  Filtering

By Status

Can toggle by All or Active for filtering. Objects which have no End Time are considered Active.

 

By Search

Performs a substring filter by search term of designated fields for the Telemetry object:

 

By Date

Performs a date range filter of objects.

 

Context filters for date range are available to provide easy selection of time range

5.4.3.6  Query Telemetry

Queries are shown at the summary and can be drilled into for details. In the details of a query, you can see the telemetry information for the query along with the steps of the query.

 

In addition to the common filtering options, the queries can be filtered by their statement type to allow filtering to only SELECT or similar types. The types can be toggled on and off in combination with each other.

5.4.3.7  Import Telemetry

Imports can be seen at the summary level and the number of rows that have been imported so far as part of the

import.  Import telemetry is aggregated for all modules.  The rows imported is updated periodically during the active import.


6         Removal of Enterprise Manager

If you have an existing build of the Enterprise Manager and wish to uninstall it, here are the procedures. Some of

these may overlap as a node could be acting as multiple roles.

6.1      Saving off configuration/data files

If you would like to keep the configuration and telemetry files before removing Enterprise Manager in order to use them when re-installing/upgrading Enterprise Manager, the following directories should be copied to a backup location and restored once EM has been re-installed:

 

/opt/infinidb/em/conf

/opt/infinidb/em/data

6.2      EM Removal

6.2.1    General

Every node in the cluster has components installed on it during the attach process.

 

service infinidb-em-collectd stop

chkconfig --del infinidb-em-collectd

userdel -r infinidb-em

rm -rf /opt/infinidb/em

6.2.2    EM Host

The EM was installed via a RPM which then installed the components needed for the EM host. It also has the

components installed (collectd) on all nodes as part of the EM attach. Currently the RPM does not contain a full

uninstall script, so we will do it manually and then remove the entry from the RPM DB.

 

service infinidb-em stop

chkconfig --del infinidb-em

userdel -r infinidb-em

rm -rf /opt/infinidb/em

rpm -e --justdb infinidb-entmgr

6.2.3    OAM Host

The OAM Server contains the graphite installation which has two processes, graphite and carbon, to be removed.

 

service infinidb-em-graphite stop

service infinidb-em-carbon stop

chkconfig --del infinidb-em-graphite

chkconfig --del infinidb-em-carbon

userdel -r infinidb-em

rm -rf /opt/infinidb/em

                                                                                      


7         Appendix A

7.1.1    Logging

The Enterprise Manager server has two logs being generated. These are configurable in the YAML file in the server directory.

 

·         Request log

o   The HTTP access logs to the EM

o   Rolled up daily and archived for 5 days

o   WARN and higher are sent to STDOUT also

o    /opt/infinidb/em/server-access.log

·         Server log

o   The application logs of the EM

o   Rolled up daily and archived for 5 days

o   WARN and higher are sent to STDOUT also

o    /opt/infinidb/em/server.log

 

Collectd, if it is present on the machine (after attach of cluster), will generate a log:

o   The log for the collectd agent running on the machine

o    /opt/infinidb/em/log/collectd