Troubleshooting Greenplum Database System and DCA issues


Standby Master DCA showing "not synchronized" . what to do?

posted Aug 29, 2014, 11:56 AM by Sachchida Ojha   [ updated Nov 3, 2014, 7:20 PM ]

Problem: running gpstat show standby master not synchronized.
gpadmin@master$gpstate -f
The gadget spec URL could not be found
20120829:12:31:02:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:-Obtaining Segment details from master...
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:-Standby master details
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:-----------------------
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby address = smdw
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby data directory = /data/master/gpseg-1
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby port = 1589
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby PID = 12498
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby status = Standby host passive
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--------------------------------------------------------------
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--gp_master_mirroring table
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--------------------------------------------------------------
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--Summary state: Not Synchronized
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--Detail state: Unexpected error
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--Log time: 2012-08-29 31:44:21-03
20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--------------------------------------------------------------

Solution:run the following command in the master from gpadmin account to fix the problem,

gpadmin@master$gpinitstandby -n -M fast

Note: this command will shutdown GPDB firstly, so please run it in outage window.
The gadget spec URL could not be found
gpinitstandby
Adds and/or initializes a standby master host for a Greenplum Database system.


Synopsis
gpinitstandby { -s standby_hostname | -r | -n }
[-M smart | -M fast] [-a] [-q] [-D] [-L]
[-l logfile_directory]
gpinitstandby -? | -v

Description
The gpinitstandby utility adds a backup master host to your Greenplum Database system. If your system has an existing backup master host configured, use the -r option to remove it before adding the new standby master host.

Before running this utility, make sure that the Greenplum Database software is installed on the backup master host and that you have exchanged SSH keys between hosts. Also make sure that the master port is set to the same port number on the master host and the backup master host. See the Greenplum Database Installation Guide for instructions. This utility should be run on the currently active primary master host.

The utility will perform the following steps:

•Shutdown your Greenplum Database system
•Update the Greenplum Database system catalog to remove the existing backup master host information (if the -r option is supplied)
•Update the Greenplum Database system catalog to add the new backup master host information (use the -n option to skip this step)
•Edit the pg_hba.conf files of the segment instances to allow access from the newly added standby master.
•Setup the backup master instance on the alternate master host
•Start the synchronization process
•Restart your Greenplum Database system

A backup master host serves as a ‘warm standby’ in the event of the primary master host becoming unoperational. The backup master is kept up to date by a transaction log replication process (gpsyncagent), which runs on the backup master host and keeps the data between the primary and backup master hosts synchronized. If the primary master fails, the log replication process is shut down, and the backup master can be activated in its place by using the utility. Upon activation of the backup master, the replicated logs are used to reconstruct the state of the master host at the time of the last successfully committed transaction.
The activated standby master effectively becomes the Greenplum Database master, accepting client connections on the master port and performing normal master operations such as SQL command processing and workload management.

Options
-a (do not prompt)
Do not prompt the user for confirmation.

-D (debug)
Sets logging level to debug.

-l logfile_directory
The directory to write the log file. Defaults to ~/gpAdminLogs.

-L (leave database stopped)
Leave Greenplum Database in a stopped state after removing the warm standby master.

-M fast (fast shutdown - rollback)
Use fast shut down when stopping Greenplum Database at the beginning of the standby initialization process. Any transactions in progress are interrupted and rolled back.

-M smart (smart shutdown - warn)
Use smart shut down when stopping Greenplum Database at the beginning of the standby initialization process. If there are active connections, this command fails with a warning. This is the default shutdown mode.

-n (resynchronize)
Use this option if you already have a standby master configured, and just want to resynchronize the data between the primary and backup master host. The Greenplum system catalog tables will not be updated.

-q (no screen output)
Run in quiet mode. Command output is not displayed on the screen, but is still written to the log file.

-r (remove standby master)
Removes the currently configured standby master host from your Greenplum Database system.

-s standby_hostname
The host name of the standby master host.

-v (show utility version)
Displays the version, status, last updated date, and check sum of this utility.

-? (help)
Displays the online help.

Examples
Add a backup master host to your Greenplum Database system and start the synchronization process:
gpinitstandby -s host09

Remove the existing backup master from your Greenplum system configuration:
gpinitstandby -r
The gadget spec URL could not be found
Start an existing backup master host and synchronize the data with the primary master host - do not add a new Greenplum backup master host to the system catalog:
gpinitstandby -n

Note: Do not specify the -n and -s options in the same command.


Can't create or drop a table in schema

posted Mar 20, 2014, 7:37 AM by Sachchida Ojha   [ updated Mar 20, 2014, 11:02 AM ]

Problem: Create table command fails with error ERROR: relation "abc" already exists (seg4 sdw1:40004 pid=13649). drop table command fails with error relation does not exists. 

Cause of the problem: A function crashed on a segment and left a inconsistent catalog. you can check on sdw4:40004 in pg_class and you should see the table there. The question is: where else does the table exist. You can run gpcheckcat to check all catalogs.

Resolution: To fix the issue access the master in utility mode and add that table definition. then access normally and drop it. That assumes that the table exists on all other segments, just not the master.

Step 1: Run gpcheckcat - Run as gpadmin $GPHOME/bin/lib/gpcheckcat -p 5432 -A>checkcat_A.out 2>&1 &
Step 2. Look at the checkcat_A.out and log from /home/gpadmin/gpAdminLogs/gpcheckcat_20140315.log **20140315 is the date when you ran the utility.
Step 3: Shutdown the database
Step 4: Starts Greenplum Database in restricted mode $gptart -R
(only database superusers are allowed to connect).
step 5: Run gpcheckcat - Run as gpadmin $GPHOME/bin/lib/gpcheckcat -p 5432 -B>checkcat_B.out 2>&1 &
Step 6. Run gpcheckcat repair script
Step 7: Run gpcheckcat - Run as gpadmin $GPHOME/bin/lib/gpcheckcat -p 5432 -C>checkcat_C.out 2>&1 &
Step 8: Fix Catalog issue
  1. Stop the database
  2. Start the database (master only) $gpstart -m
  3. Connect to the database in utility mode (you need to access the master in utility mode so it only creates relations on the master. Then exit and re-connect normally to drop.)
  4. PGOPTIONS='-c gp_session_role=utility' psql  sachi (For help run [gpadmin@sachi ~]$ gpstart -?/PGO) will give you syntex for utility mode.
  5. Create table abc. (it will create the table abc in the master only)
  6. Exit the utility mode ($\q)
  7. Restart the database ($gpstap -arf)
  8. Connect to the database
  9. Drop  table abc.
Step 9: Run gpcheckcat - Run as gpadmin $GPHOME/bin/lib/gpcheckcat -p 5432 -D>checkcat_D.out 2>&1 &


Note: When you run create table script inside an stored procedure/function in Greenplum and function fails then it rolls back only master but table definition in the segments exist. This is the reason of the above problem.

DCA shutdown

posted Nov 8, 2012, 6:04 AM by Sachi Ojha   [ updated Nov 6, 2014, 1:27 PM by Sachchida Ojha ]

The gadget spec URL could not be found
The dca_shutdown utility will safely power down all servers in a DCA. The utility can be run with no parameters, and will use the system inventory generated by DCA Setup during an installation or Regenerate DCA Config Files operation. If the utility is run with a hostfile or hostname specified, only those hosts will be shutdown. This utility will not shut down the administration, Interconnect or aggregation switches.
The utility should be run as the user root. Prior to running the dca_shutdown, the following steps should be performed to ensure a clean shutdown:
1.Stop Greenplum Database:
$ gpstop -af
2.Stop Command Center:
$ gpcmdr --stop
3.Stop health monitoring as the user root:
$ su -
# dca_healthmon_ctl -d

The gadget spec URL could not be found
Usage:

dca_shutdown { -f hostfile | -h hostname } [ --ignoredb ] [ --password= password ] [ --passfile= password_file ] [--statusonly]
dca_shutdown
dca_shutdown --help

The gadget spec URL could not be found
Options
-?, --help
Print usage and help information
-i, --ignoredb
Do not check if Greenplum Database, health monitoring or Command Center are running. Shut down all servers immediately.
-h, --host hostname
Perform a shutdown on the host specified.
-f, --hostfile hostfile
Perform a shutdown on the hosts listed in the hostfile. This option can not be used with the --host option.
-p, --password password
Specify a password to connect to the server’s IPMI (iDRAC) to perform the shutdown. The password is originally set during installation with DCA Setup - if an installation through DCA Setup has never been run, the user will be prompted for a password.
-s, --passfile password_file
Specify a file containing the password to use to connect to the server’s IPMI (iDRAC) to perform the shutdown. This file is generated during installation with DCA Setup, and is located in /opt/dca/etc/ipmipasswd.
-o, --statusonly
Print the power status (ON | OFF) of all servers. This will not power off any servers.
The gadget spec URL could not be found
Examples
Shut down all servers in a DCA:
dca_shutdown
Shut down servers listed in the file hostfile:
dca_shutdown -f /home/gpadmin/gpconfigs/hostfile
The gadget spec URL could not be found

How to enable and disable healthmon?

posted Nov 8, 2012, 6:01 AM by Sachi Ojha

To disable healthmond 
"dca_healthmon_ctl -d" 

To enable healthmond 
"dca_healthmon_ctl -e".

How to diagnose DCA controller x status: error: degraded

posted Nov 7, 2012, 1:38 PM by Sachchida Ojha

To diagnose  controller issues,Run the following commands and look at the output.

omreport system esmlog
omreport system alertlog
omreport storage battery
omreport storage pdisk controller=0
omreport storage vdisk
cat /etc/gpdb-appliance-version
omreport storage controller

For example

[gpadmin@mdw ~]$ omreport system esmlog
Embedded System Management (ESM) Log

Health : Ok

Severity      : Ok
Date and Time : Wed Jul 13 17:50:45 2011
Description   : Log cleared.

Severity      : Ok
Date and Time : Wed Jul 13 17:51:08 2011
Description   : This is an OEM record.

Severity      : Ok
Date and Time : Wed Jul 13 17:51:08 2011
Description   : An OS graceful shut-down occured.

*********************************************************************************************

 [gpadmin@mdw ~]$ omreport storage battery
List of Batteries in the System

Controller PERC H700 Integrated (Slot Embedded)
ID                        : 0
Status                    : Ok
Name                      : Battery 0
State                     : Ready
Recharge Count            : Not Applicable
Max Recharge Count        : Not Applicable
Predicted Capacity Status : Ready
Learn State               : Idle
Next Learn Time           : 82 days 20 hours
Maximum Learn Delay       : 7 days 0 hours
Learn Mode                : Auto

*******************************************************************************************

[gpadmin@mdw ~]$ omreport storage pdisk controller=0
List of Physical Disks on Controller PERC H700 Integrated (Embedded)

Controller PERC H700 Integrated (Embedded)
ID                        : 0:0:0
Status                    : Ok
Name                      : Physical Disk 0:0:0
State                     : Online
Power Status              : Spun Up
Bus Protocol              : SAS
Media                     : HDD
Revision                  : FM08
Failure Predicted         : No
Certified                 : Yes
Encryption Capable        : No
Encrypted                 : Not Applicable
Progress                  : Not Applicable
Mirror Set ID             : Not Applicable
Capacity                  : 558.38 GB (599550590976 bytes)
Used RAID Disk Space      : 558.38 GB (599550590976 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare                 : No
Vendor ID                 : DELL(tm)
Product ID                : ST9600204SS     
Serial No.                : 3WN0GSYQ            
Part Number               : SG07T0DW125310BP02H1A00 
Negotiated Speed          : 6.00 Gbps
Capable Speed             : 6.00 Gbps
Manufacture Day           : 07
Manufacture Week          : 48
Manufacture Year          : 2010
SAS Address               : 5000C50028C725B1

ID                        : 0:0:1
Status                    : Ok
Name                      : Physical Disk 0:0:1
State                     : Online
Power Status              : Spun Up
Bus Protocol              : SAS
Media                     : HDD
Revision                  : FM08
Failure Predicted         : No
Certified                 : Yes
Encryption Capable        : No
Encrypted                 : Not Applicable
Progress                  : Not Applicable
Mirror Set ID             : Not Applicable
Capacity                  : 558.38 GB (599550590976 bytes)
Used RAID Disk Space      : 558.38 GB (599550590976 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare                 : No
Vendor ID                 : DELL(tm)
Product ID                : ST9600204SS     
Serial No.                : 3WN0GTQK            
Part Number               : SG07T0DW125310BP018SA00 
Negotiated Speed          : 6.00 Gbps
Capable Speed             : 6.00 Gbps
Manufacture Day           : 07
Manufacture Week          : 48
Manufacture Year          : 2010
SAS Address               : 5000C50028C70169

ID                        : 0:0:2
Status                    : Ok
Name                      : Physical Disk 0:0:2
State                     : Online
Power Status              : Spun Up
Bus Protocol              : SAS
Media                     : HDD
Revision                  : FM08
Failure Predicted         : No
Certified                 : Yes
Encryption Capable        : No
Encrypted                 : Not Applicable
Progress                  : Not Applicable
Mirror Set ID             : Not Applicable
Capacity                  : 558.38 GB (599550590976 bytes)
Used RAID Disk Space      : 558.38 GB (599550590976 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare                 : No
Vendor ID                 : DELL(tm)
Product ID                : ST9600204SS     
Serial No.                : 3WN0GTP4            
Part Number               : SG07T0DW125310BP0175A00 
Negotiated Speed          : 6.00 Gbps
Capable Speed             : 6.00 Gbps
Manufacture Day           : 07
Manufacture Week          : 48
Manufacture Year          : 2010
SAS Address               : 5000C50028C70069

ID                        : 0:0:3
Status                    : Ok
Name                      : Physical Disk 0:0:3
State                     : Online
Power Status              : Spun Up
Bus Protocol              : SAS
Media                     : HDD
Revision                  : FM08
Failure Predicted         : No
Certified                 : Yes
Encryption Capable        : No
Encrypted                 : Not Applicable
Progress                  : Not Applicable
Mirror Set ID             : Not Applicable
Capacity                  : 558.38 GB (599550590976 bytes)
Used RAID Disk Space      : 558.38 GB (599550590976 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare                 : No
Vendor ID                 : DELL(tm)
Product ID                : ST9600204SS     
Serial No.                : 3WN0GSTC            
Part Number               : SG07T0DW125310BP02D0A00 
Negotiated Speed          : 6.00 Gbps
Capable Speed             : 6.00 Gbps
Manufacture Day           : 07
Manufacture Week          : 48
Manufacture Year          : 2010
SAS Address               : 5000C50028C72389

ID                        : 1:0:4
Status                    : Ok
Name                      : Physical Disk 1:0:4
State                     : Online
Power Status              : Spun Up
Bus Protocol              : SAS
Media                     : HDD
Revision                  : FM08
Failure Predicted         : No
Certified                 : Yes
Encryption Capable        : No
Encrypted                 : Not Applicable
Progress                  : Not Applicable
Mirror Set ID             : Not Applicable
Capacity                  : 558.38 GB (599550590976 bytes)
Used RAID Disk Space      : 558.38 GB (599550590976 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare                 : No
Vendor ID                 : DELL(tm)
Product ID                : ST9600204SS     
Serial No.                : 3WN0G0X5            
Part Number               : SG07T0DW125310BP01BLA00 
Negotiated Speed          : 6.00 Gbps
Capable Speed             : 6.00 Gbps
Manufacture Day           : 07
Manufacture Week          : 48
Manufacture Year          : 2010
SAS Address               : 5000C50028C7068D

ID                        : 1:0:5
Status                    : Ok
Name                      : Physical Disk 1:0:5
State                     : Ready
Power Status              : Spun Up
Bus Protocol              : SAS
Media                     : HDD
Revision                  : FM08
Failure Predicted         : No
Certified                 : Yes
Encryption Capable        : No
Encrypted                 : Not Applicable
Progress                  : Not Applicable
Mirror Set ID             : Not Applicable
Capacity                  : 558.38 GB (599550590976 bytes)
Used RAID Disk Space      : 558.38 GB (599550590976 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare                 : Global
Vendor ID                 : DELL(tm)
Product ID                : ST9600204SS     
Serial No.                : 3WN0GV1P            
Part Number               : SG07T0DW125310BP01LYA00 
Negotiated Speed          : 6.00 Gbps
Capable Speed             : 6.00 Gbps
Manufacture Day           : 07
Manufacture Week          : 48
Manufacture Year          : 2010
SAS Address               : 5000C50028C70D59

*********************************************************************************************

[gpadmin@mdw ~]$ omreport storage vdisk
List of Virtual Disks in the System

Controller PERC H700 Integrated (Embedded)
ID                  : 0
Status              : Ok
Name                : boot
State               : Ready
Encrypted           : Not Applicable
Layout              : RAID-5
Size                : 48.00 GB (51539607552 bytes)
Device Name         : /dev/sda
Bus Protocol        : SAS
Media               : HDD
Read Policy         : Adaptive Read Ahead
Write Policy        : Write Back
Cache Policy        : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy   : Disabled

ID                  : 1
Status              : Ok
Name                : swap
State               : Ready
Encrypted           : Not Applicable
Layout              : RAID-5
Size                : 48.00 GB (51539607552 bytes)
Device Name         : /dev/sdb
Bus Protocol        : SAS
Media               : HDD
Read Policy         : Adaptive Read Ahead
Write Policy        : Write Back
Cache Policy        : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy   : Disabled

ID                  : 2
Status              : Ok
Name                : data
State               : Ready
Encrypted           : Not Applicable
Layout              : RAID-5
Size                : 2,137.50 GB (2295123148800 bytes)
Device Name         : /dev/sdc
Bus Protocol        : SAS
Media               : HDD
Read Policy         : Adaptive Read Ahead
Write Policy        : Write Back
Cache Policy        : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy   : Disabled

**********************************************************************************************

[gpadmin@mdw ~]$ cat /etc/gpdb-appliance-version
1.2.0.0s

*********************************************************************************************

[gpadmin@mdw ~]$ omreport storage controller
 Controller  PERC H700 Integrated (Embedded)

Controllers
ID                                            : 0
Status                                        : Ok
Name                                          : PERC H700 Integrated
Slot ID                                       : Embedded
State                                         : Ready
Firmware Version                              : 12.10.2-0004
Minimum Required Firmware Version             : Not Applicable
Driver Version                                : 00.00.05.38-rh1

Minimum Required Driver Version               : Not Applicable
Storport Driver Version                       : Not Applicable
Minimum Required Storport Driver Version      : Not Applicable
Number of Connectors                          : 2
Rebuild Rate                                  : 30%
BGI Rate                                      : 30%
Check Consistency Rate                        : 30%
Reconstruct Rate                              : 30%
Alarm State                                   : Not Applicable
Cluster Mode                                  : Not Applicable
SCSI Initiator ID                             : Not Applicable
Cache Memory Size                             : 512 MB
Patrol Read Mode                              : Auto
Patrol Read State                             : Stopped
Patrol Read Rate                              : 30%
Patrol Read Iterations                        : 5
Abort Check Consistency on Error              : Disabled
Allow Revertible Hot Spare and Replace Member : Enabled
Load Balance                                  : Not Applicable
Auto Replace Member on Predictive Failure     : Disabled
Redundant Path view                           : Not Applicable
CacheCade Capable                             : Not Applicable
Persistent Hot Spare                          : Disabled
Encryption Capable                            : Yes
Encryption Key Present                        : No
Encryption Mode                               : None
Spin Down Unconfigured Drives                 : Disabled
Spin Down Hot Spares                          : Disabled

How to diagnose DCA memory device x status: error: critical

posted Nov 7, 2012, 1:34 PM by Sachchida Ojha

Run the following command

omreport chassis memory
[gpadmin@mdw ~]$ omreport chassis memory

You will see output like this

Memory Information

Health : Ok

Memory Operating Mode

Fail Over State                     : Inactive
Memory Operating Mode Configuration : Optimizer

Attributes of Memory Array(s)

Attributes of Memory Array(s)
Location           : System Board or Motherboard
Use                : System Memory
Installed Capacity : 49152  MB
Maximum Capacity   : 196608  MB
Slots Available    : 12
Slots Used         : 6
Error Correction   : Multibit ECC

Total of Memory Array(s)
Total Installed Capacity                     : 49152  MB
Total Installed Capacity Available to the OS : 48161  MB
Total Maximum Capacity                       : 196608  MB

Details of Memory Array 1
Index          : 0
Status         : Ok
Connector Name : DIMM_A1 
Type           : DDR3 - Synchronous Registered (Buffered)
Size           : 8192  MB

Index          : 1
Status         : Ok
Connector Name : DIMM_A2 
Type           : DDR3 - Synchronous Registered (Buffered)
Size           : 8192  MB

Index          : 2
Status         : Ok
Connector Name : DIMM_A3 
Type           : DDR3 - Synchronous Registered (Buffered)
Size           : 8192  MB

Index          : 
Status         : Unknown
Connector Name : DIMM_A4 
Type           : [Not Occupied]
Size           : 

Index          : 
Status         : Unknown
Connector Name : DIMM_A5 
Type           : [Not Occupied]
Size           : 

Index          : 
Status         : Unknown
Connector Name : DIMM_A6 
Type           : [Not Occupied]
Size           : 

Index          : 3
Status         : Ok
Connector Name : DIMM_B1 
Type           : DDR3 - Synchronous Registered (Buffered)
Size           : 8192  MB

Index          : 4
Status         : Ok
Connector Name : DIMM_B2 
Type           : DDR3 - Synchronous Registered (Buffered)
Size           : 8192  MB

Index          : 5
Status         : Ok
Connector Name : DIMM_B3 
Type           : DDR3 - Synchronous Registered (Buffered)
Size           : 8192  MB

Index          : 
Status         : Unknown
Connector Name : DIMM_B4 
Type           : [Not Occupied]
Size           : 

Index          : 
Status         : Unknown
Connector Name : DIMM_B5 
Type           : [Not Occupied]
Size           : 

Index          : 
Status         : Unknown
Connector Name : DIMM_B6 
Type           : [Not Occupied]
Size           : 

How to diagnose Greenplum appliance issue using the 'omreport' tool

posted Nov 7, 2012, 12:00 PM by Sachchida Ojha   [ updated Nov 7, 2012, 12:31 PM ]

Disks:

omreport storage controller | grep -e "^ID" -e "^Status"
omreport storage pdisk controller=0 | grep -e "^ID" -e "^State"
omreport storage vdisk controller=0 | grep -e "^ID" -e "^State"

You want to see that the Status is OK for the controller and the State is "Online" for all physical and virtual disks.
 
Memory:

omreport chassis memory | grep ^Health - Should show OK.
 
omreport chassis memory | grep -C 1 "Total Installed" | head -3 - Should indicate that 49152 MB are installed.

System Events:

omreport system alertlog - Lists alerts issued.  Review list for anything that's unexpected.
 
omreport system esmlog - Hardware events logged. Review list for anything that's unexpected.
omreport system postlog - Lists the power on self test log.  Review list for anything that's unexpected. 

Chassis commands:

omreport chassis biossetup  (must be root to run) - Reports all the non-default BIOS settings to validate that Arrow has the BIOS to spec.
 
omreport chassis firmware - Reports the iDRAC firmware  version.
 
omreport chassis bios - Reports the server BIOS version.
 
omreport chassis memory  - Lists the installed memory by board slot.
 
omreport chassis processors - Lists the installed processors by processor slot.
 
omreport chassis nics  - Lists the NICs and includes the slot each NIC occupies (embedded, PCIE3, etc.).
 
omreport chassis slots | more - Lists the expansion slot info.  Look for cards that should be in the list but aren't.
 
omreport chassis pwrsupplies | grep -e "^Power Supply Redundancy" -e ^Index -e ^Status -e "^Online Status" - Shows the overall status of the power supply redundancy and the status of each individual power supply.
 
omreport chassis temps -  Lists the temps of various bits.
 
omreport chassis batteries - Lists the status of the CMOS battery.
 
omreport chassis volts | grep -e ^Index -e ^Status -e ^Probe - Lists the voltages of the individual parts and whether that's OK.
 
omconfig chassis leds led=identify flash=on timeout=10 - Flashes the local machine's identity light for 10 seconds. If no timeout is given, sets i to flash permanently.
 
omconfig chassis leds led=hdfault action=clear - Clears an HD fault light.
 
omreport chassis fans | grep -e Index -e Status - Lists the status of the fans.
 
omreport chassis processors | grep -e Index -e Status - Lists the status of the processors.


ipmitool -H [hostname or IP address] -U root -P calvin sel list - Shows the server event list of the host owning the iDRAC named in the -H parameter

1-7 of 7