Greenplum Command Center Features and references

Configuration parameters for Greenplum Command Center are stored in these files:

1. $MASTER_DATA_DIRECTORY/gpperfmon/conf/gpperfmon.conf — stores configuration parameters for the Greenplum Command Center agents.

2. $GPPERFMONHOME/instances/instance_name/conf/gpperfmonui.conf and lighttpd.conf — stores configuration parameters for the Command Center web application and web server.

3. $MASTER_DATA_DIRECTORY/postgresql.conf — stores configuration parameters to enable the Greenplum Command Center feature for Greenplum Database server.
Any system user with write permissions to these directories can edit these configuration files.

Command Center Database Reference

posted Nov 8, 2012, 1:01 PM by Sachi Ojha

The Command Center database consists of three sets of tables; now tables store data on current system metrics such as active queries, history tables store data on historical metrics, and tail tables are for data in transition. Tail tables are for internal use only and should not be queried by users. The now and tail data are stored as text files on the master host file system, and accessed by the Command Center database via external tables. The history tables are regular database tables stored within the Command Center (gpperfmon) database.

The database consists of three sets of tables:
•now tables store data on current system metrics such as active queries.
•history tables store data historical metrics.
•tail tables are for data in transition. These tables are for internal use onlz and should not be queried by endusers.

The now and tail data are stored as text files on the master file system, and accessed by Command Center database (gpperfmon) via external tables. The history tables are regular database tables stored within the Command Center database.

There are the following categories of tables:

1. The database_* tables store query workload information for a Greenplum Database instance.

2. The emcconnect_history table displays information about ConnectEMC events and alerts. ConnectEMC events are triggered based on a hardware failure, a fix to a failed hardware component, or a Greenplum Database startup. Once an ConnectEMC event is triggered, an alert is sent to EMC Support.

3. The filerep_* tables store health and status metrics for the file replication process. This process is how high-availability/mirroring is achieved in Greenplum Database instance. Statistics are maintained for each primary-mirror pair.

4. The health_* tables store system health metrics for the EMC Data Computing Appliance.

5. The interface_stats_* tables store statistical metrics for each active interface of a Greenplum Database instance. Note: These tables are in place for future use and are not currently populated.

6. The iterators_* tables store information about query plan iterators and their metrics. A query iterator refers to a node or operation in a query plan.

7. The queries_* tables store high-level query status information.

8. The segment_* tables store memory allocation statistics for the Greenplum Database segment instances.

9. The socket_stats_* tables store statistical metrics about socket usage for a Greenplum Database instance. Note: These tables are in place for future use and are not currently populated.

10. The system_* tables store system utilization metrics.

11. The tcp_stats_* tables store statistical metrics about TCP communications for a Greenplum Database instance. Note: These tables are in place for future use and are not currently populated.

12. The udp_stats_* tables store statistical metrics about UDP comm unications for a Greenplum Database instance. Note: These tables are in place for future use and are not currently populated.

The Command Center database also contains the following views:

1. The dynamic_memory_info view shows an aggregate of all the segments per host and the amount of dynamic memory used per host.

2. The iterators_*_rollup set of views summarize the query iterator metrics across all segments in the system.

3. The memory_info view shows per-host memory information from the system_history and segment_history tables.

Command Center Utility Reference

posted Nov 8, 2012, 12:57 PM by Sachi Ojha

Greenplum provides the following utilities to install and maintain Greenplum Command Center:

1. gpperfmon_install - Enables the data collection agents.

2. gpcmdr - Sets up and manages the web application.

gpperfmon_install
Installs the Command Center database (gpperfmon) and optionally enables the data collection agents.

Synopsis
gpperfmon_install
[--enable --password gpmon_password --port gpdb_port]
[--pgpass path_to_file]
[--verbose]

gpperfmon_install --help | -h | -?

Description
The gpperfmon_install utility automates the steps required to enable the Command Center data collection agents. You must be the Greenplum system user (gpadmin) in order to run this utility. If using the --enable option, Greenplum Database instance must be restarted after the utility completes.

When run without any options, the utility will just create the Command Center database (gpperfmon). When run with the --enable option, the utility will also run the following additional tasks necessary to enable the Command Center data collection agents:

1.Creates the gpmon superuser role in Greenplum Database. The Command Center data collection agents require this role to connect to the database and write their data. The gpmon superuser role uses MD5-encrypted password authentication by default. Use the --password option to set the gpmon superuser’s password. Use the --port option to supply the port of the Greenplum Database master instance.

2.Updates the $MASTER_DATA_DIRECTORY/pg_hba.conf file. The utility will add the following line to the host-based authentication file (pg_hba.conf). This allows the gpmon user to locally connect to any database using MD5-encrypted password authentication:

local all gpmon md5

3.Updates the password file (.pgpass). In order to allow the data collection agents to connect as the gpmon role without a password prompt, you must have a password file that has an entry for the gpmon user. The utility add the following entry to your password file (if the file does not exist, the utility will create it):
*:5432:gpperfmon:gpmon:gpmon_password
If your password file is not located in the default location (~/.pgpass), use the --pgpass option to specify the file location.

4.Sets the server configuration parameters for Command Center. The following parameters must be enabled in order for the data collection agents to begin collecting data. The utility will set the following parameters in the postgresql.conf configuration files:
gp_enable_gpperfmon=on (in all postgresql.conf files)
gpperfmon_port=8888 (in all postgresql.conf files)
gp_external_enable_exec=on (in the master postgresql.conf file)

Options
--enable
In addition to creating the gpperfmon database, performs the additional steps
required to enable the Command Center data collection agents. When --enable is
specified the utility will also create and configure the gpmon superuser account and
set the Command Center server configuration parameters in the postgresql.conf
files.

--password gpmon_password
Required if --enable is specified. Sets the password of the gpmon superuser.

--port gpdb_port
Required if --enable is specified. Specifies the connection port of the Greenplum
Database master.

--pgpass path_to_file
Optional if --enable is specified. If the password file is not in the default location
of ~/.pgpass, specifies the location of the password file.

--verbose
Sets the logging level to verbose.

--help | -h | -?
Displays the online help.

Examples
Create the Command Center database (gpperfmon) only:

$ su - gpadmin

$ gpperfmon_install

Create the Command Center database (gpperfmon), create the gpmon superuser, and enable the Command Center agents:
$ su - gpadmin
$ gpperfmon_install --enable --password p@$$word --port 5432
$ gpstop -r

gpcmdr
Configures and manages instances of the Command Center Console.

Synopsis
gpcmdr --setup [instance_name]
| --start [instance_name]
| --stop [instance_name]
| --restart [instance_name]
| --status [instance_name]

Description
The gpcmdr utility sets up and configures Command Center Console instances, starts and stops instances, and provides status information.
For all actions, including starting and stopping, you can specify a console instance name. If you do not specify a name, the action applies to all existing console instances.

Options
--setup
Configures console components on the installation host. With this option, gpcmdr prompts for values to configure the components and writes the values to gpperfmonui.conf and lighttpd.conf. 

--start
Starts the specified instance (or all instances by default) and its associated web service.
--stop
Stops the specified instance (or all instances by default) and its associated web service.
--restart
Restarts the specified instance (or all instances by default) and its associated web service.
--status
Displays the status, either Running or Stopped, of the web service.
--version
Displays the version of the gpperfmon utility and the lighttpd web service.

Examples
Start the utility in order to install and configure a new instance:
$ gpcmdr --setup

Check the status of Command Center:
$ gpcmdr --status

Accessing Command Center Data using SNMP

posted Nov 8, 2012, 12:51 PM by Sachi Ojha

You can configure a Greenplum system to trigger SNMP alerts or send email notifications to system administrators whenever certain database events occur.

Greenplum’s gpsnmpd agent is an SNMP (Simple Network Management Protocol) daemon that supports SNMP requests on the health and state of a Greenplum system by using a set of MIBs (Management Information Bases).

MIBs are a collection of objects that describe an SNMP-manageable entity — in this case, a Greenplum system. MIBs can be used by enterprise monitoring systems such
as IBM Tivoli, HP Openview, and Nagios to monitor our Greenplum system.

SNMP on the Greenplum Data Computing Appliance
For more information about SNMP on the DCA, see the Greenplum Data Computing Appliance Getting Started Guide. 

Starting with Greenplum Data Computing Appliance version 1.2, we have a new MIB definition that will help you monitor software and hardware as follows:
•Software
•Version information
•Database
•Hadoop
•Hardware
•Master Node
•Segment Nodes
•Interconnect switches
•DIA/ETL loads
•Hadoop Master Nodes
•Hadoop Worker Nodes
•Admin switches

SNMP on Greenplum Database
For more information about SNMP on the Greenplum Database software-only, see chapter 18, Monitoring a Greenplum System, in Greenplum Database Administrator Guide.
There are the following two MIB definitions for Greenplum Database:

•RDBMS-MIB
•NETWORK-SERVICES-MIB

Viewing Historical Query Metrics using SQLin Greenplum Command Center

posted Nov 8, 2012, 12:48 PM by Sachi Ojha

Historical tables in the Command Center database store all query and iterator data collected by Command Center. You can analyze this data to identify trends in the performance of Greenplum system over time.

Calculating the average, minimum and maximum values for system-wide query statistics aggregated over time can yield a useful view of database utilization. The following information provides an example SQL view for viewing query statistics in this way.

In this example view named database_metrics_1hour, rows of statistics from the table database_history are averaged over hour-long intervals:

DROP VIEW if exists database_metrics_1hour;

CREATE VIEW database_metrics_1hour AS (
SELECT date_trunc('hour', ctime) AS sample_time, avg(queries_total) AS queries_total,
min(queries_total) AS queries_total_min,
max(queries_total) AS queries_total_max, avg(queries_running) AS queries_running, min(queries_running) AS queries_running_min, max(queries_running) AS queries_running_max, avg(queries_queued) AS queries_queued,
min(queries_queued) AS queries_queued_min, max(queries_queued) AS queries_queued_max
FROM database_history
GROUP BY sample_time );


To change the interval for averaging results, adjust the value specified for date_trunc. Valid values are microseconds, milliseconds, second, minute, hour, day, week, month, quarter, year, decade, century and millennium.

Note: You use can also the Command Center Console to view historical query metrics. See the Command Center Console online help for more information.

Viewing Historical System Metrics using SQLin Greenplum command Center

posted Nov 8, 2012, 12:47 PM by Sachi Ojha

Historical tables in the Command Center database store all system data collected by Command Center. You can analyze this data to identify trends in the performance of the Greenplum system over time.

With a default quantum setting of 15 seconds, Command Center collects very large amounts of historical system data. To view meaningful groupings of system data records, you can view the data by selected time intervals with a SQL view (as described below).

The following provides an example SQL view for viewing aggregated system metrics. In this example view named system_metrics_aggregate_1min, rows of system metrics from system_history are averaged over one-minute intervals and aggregated across all segments in the array:

DROP VIEW IF EXISTS system_metrics_aggregate_1min;

CREATE VIEW system_metrics_aggregate_1min AS (
SELECT date_trunc('minute', ctime) AS sample_time, hostname, avg(mem_total) AS mem_total,
avg(mem_used) AS mem_used,
avg(mem_actual_used) AS mem_actual_used,
avg(mem_actual_free) AS mem_actual_free,
avg(swap_total) AS swap_total, avg(swap_used) AS swap_used, avg(swap_page_in) AS swap_page_in,
avg(swap_page_out) AS swap_page_out,
avg(cpu_user) AS cpu_user, avg(cpu_sys) AS cpu_sys, avg(cpu_idle) AS cpu_idle,
avg(load0) AS load0, avg(load1) AS load1,
avg(load2) AS load2,
avg(disk_ro_rate) AS disk_ro_rate,
avg(disk_wo_rate) AS disk_wo_rate,
avg(disk_rb_rate) AS disk_rb_rate,
avg(disk_wb_rate) AS disk_wb_rate,
avg(net_rp_rate) AS net_rp_rate,
avg(net_wp_rate) AS net_wp_rate,
avg(net_rb_rate) AS net_rb_rate,
avg(net_wb_rate) AS net_wb_rate
FROM system_history
GROUP BY hostname, sample_time
);


To change the interval for averaging results, adjust the value specified for date_trunc. Valid values are microseconds, milliseconds, second, minute, hour, day, week, month, quarter, year, decade, century and millennium.

Command Center Web Server (lighttpd)Administration

posted Nov 8, 2012, 12:42 PM by Sachi Ojha


The Lighttpd web server and web service middleware are installed in the www directory of your Greenplum Command Center installation. For detailed information on Lighttpd administration, see http://www.lighttpd.net/.

Configuring the Web Server
The Lighttpd web server configuration file is stored in $GPPERFMONHOME/instances/instance_name/conf/lighttpd.conf. Some of the parameters in this configuration file are set by the gpcmdr setup utility, including the web server port and SSL options. See “Web Server Parameters (lighttpd.conf)” for a description of the parameters in this file.

You should not need to manually edit this file, if you do, you may break some functionality. Contact EMC Support if you want to make custom modifications to this file.

Viewing and Maintaining Web Server Log Files

Web server access and error logs are written to $GPPERFMONHOME/instances/instance_name/logs. These two logs are:
•lighttpd-access.log
•lighttpd-error.log

If you experience errors viewing the Greenplum Command Center Console, refer to these logs for more information.

To prevent web server logs from growing to excessive size, you can set up log file rotation using logrotate or cronolog, both of which are widely used with Lighttpd.

Command Center Database Administration

posted Nov 8, 2012, 12:39 PM by Sachi Ojha

Data collected by Command Center agents is stored in a dedicated database called gpperfmon within the Greenplum Database instance. This database requires the typical database maintenance tasks such as clean up of old historical data and periodic ANALYZE.

See “Command Center Database Reference”  for a reference of the tables and views in the gpperfmon database.

Connecting to the Command Center Database
Database administrators can connect directly to the Command Center database (gpperfmon) using any Greenplum Database-compatible client program (such as psql). For example:

$ psql -d gpperfmon -h master_host -p 5432 -U gpadmin

Backing Up and Restoring the Command Center Database
The history tables of the Command Center database (gpperfmon) can be backed up and restored using the Greenplum Database parallel backup and restore utilities (gp_dump, gp_restore, gpcrondump, gpdbrestore). See the Greenplum Database Administrator Guide for more information.

Because the Command Center database has a low number of tables, you may prefer to devise a backup plan using the table-level backup features of gp_dump. For example, you can create scripts to run gp_dump to back up the monthly partitions of the historical data tables on a monthly schedule. Alternately, you can back up your Command Center database at the database level.

Maintaining the Historical Data Tables
All of the *_history tables stored in the Command Center database (gpperfmon) are partitioned into monthly partitions. A January 2010 partition is created at installation time as a template partition (it can be deleted once some current partitions are created). The Command Center agents automatically create new partitions in two month increments as needed. Administrators must periodically drop partitions for the months that are no longer needed in order to maintain the size of the Command Center database. See the Greenplum Database Administrator Guide for more information on dropping partitions of a partitioned table.

Command Center Agent Administration

posted Nov 8, 2012, 12:37 PM by Sachi Ojha

Adding and Removing Hosts
Segment agents on new hosts are detected automatically by the master agent. Whenever gp_enable_gpperfmon is enabled on the master, the master monitor agent automatically detects, starts, and begins harvesting data from new segment agents.

To verify the addition of a new monitored host, you can check for the new hostname in the Greenplum Command Center Console System Metrics view described in the Greenplum Command Center Console online help. Alternately, you can query the system_now table for the row containing current metrics for each host. For example:

=# SELECT * FROM system_now WHERE hostname='new_hostname';

Viewing and Maintaining Master Agent Log Files
Log messages for the master agent are written to $MASTER_DATA_DIRECTORY/gpperfmon/logs/gpmmon.log by default. To change the log file location, edit the log_location parameter in gpperfmon.conf.

On the segment hosts, agent log messages are written to a gpsmon.log file in the segment instance’s data directory. For a host with multiple segments, the agent log file is located in the data directory of the first segment, as listed in the gp_configuration table by dbid. If the segment agent is unable to log into this directory, it will log messages to the home directory of the user running Command Center (typically gpadmin).

Configuring Log File Rollover
At higher logging levels, the size of the log files may grow dramatically. To prevent the log files from growing to excessive size, you can add an optional log rollover parameter to gpperfmon.conf. The value of this parameter is measured in bytes. For example:
max_log_size = 10485760

With this setting, the log files will grow to 10MB before the system rolls over the log file. The timestamp is added to the log file name when it is rolled over. Administrators must periodically clean out old log files that are no longer needed.

Configuring Greenplum Command Center

posted Nov 8, 2012, 12:35 PM by Sachi Ojha

Configuration parameters for Greenplum Command Center are stored in the following configuration files:

Agent Configuration
Changes to these files require a restart of the Greenplum Database instance
(gpstop -r).

•$MASTER_DATA_DIRECTORY/gpperfmon/conf/gpperfmon.conf

•$MASTER_DATA_DIRECTORY/postgresql.conf

Console Configuration
Changes to these files require a restart of Command Center Console (gpcmdr --restart).

•$GPPERFMONHOME/instances/instance_name/conf/gpperfmonui.conf
•$GPPERFMONHOME/instances/instance_name/conf/lighttpd.conf

You should not need to manually edit any of the files. Running the Command Center setup utility will make all the necessary modifications to these configuration files.

Starting and Stopping Greenplum Command Center

posted Nov 8, 2012, 12:34 PM by Sachi Ojha

Starting and Stopping Command Center Agents
Whenever the Greenplum Database server configuration parameter gp_enable_gpperfmon is enabled in the master postgresql.conf file, the Command Center agents will run and collect data. These agents are automatically stopped and started together along with the Greenplum Database instance.

If you wish to disable the Command Center data collection agents, you must disable the gp_enable_gpperfmon parameter, and restart the Greenplum Database instance.

Starting and Stopping Command Center Console
You can start, stop and restart Greenplum Command Center Console instances with the following commands:
$ gpcmdr --start ["instance name"]
$ gpcmdr --stop ["instance name"]
$ gpcmdr --restart ["instance name"]

If you do not specify an instance name, all instances are started, stopped or restarted at once. You can check the status of instances using:
$ gpcmdr --status ["instance name"]

1-10 of 17