Gathering Information for Greenplum Support

The gpdetective utility collects information from a running Greenplum Database system and creates a bzip2-compressed tar output file. You can then send the output file to Greenplum Customer Support to aid the diagnosis of Greenplum Database errors or system failures.
The gadget spec URL could not be found
 Run gpdetective on your master host, for example:

[08:53 sachi@sachi ~] > su - gpadmin
Password: 
[gpadmin@sachi ~]$ gpdetective
Collecting information
  + getConfiguration
  + copyLogs
  +     get master logs
  +     get primary segment logs
  +     get gpAdminLogs
  +     get primary segment pg_changetracking FILEREP_CONFIG_LOG and FILEREP_LOG  logs
  +     get mirror segment pg_changetracking FILEREP_CONFIG_LOG and FILEREP_LOG  logs
  + systemInfo
  + checkCatalog
  + pg_dumpall
  + crashReport
    + localhost
ls: cannot access |/usr/libexec/abrt-hook-ccpp: No such file or directory
ls: cannot access *: No such file or directory
ls: cannot access *: No such file or directory
ls: cannot access *: No such file or directory
ls: cannot access *: No such file or directory
ls: cannot access *: No such file or directory
ls: cannot access *: No such file or directory
ls: cannot access localhost: No such file or directory
ls: cannot access postgres: No such file or directory
ls: cannot access 636f726500*: No such file or directory
    + sachi
  + makeTar
Done
The gadget spec URL could not be found
[gpadmin@sachi ~]$ ls -ltr
total 111232
-r--r--r--. 1 root    root        7008 Oct 17  2012 README_INSTALL
-rwxr-xr-x. 1 root    root    55684023 Oct 17  2012 greenplum-db-4.2.2.4-build-1-CE-RHEL5-x86_64.bin
-rw-r--r--. 1 root    root    54377435 Apr 23  2013 greenplum-db-4.2.2.4-build-1-CE-RHEL5-x86_64.zip
-rwxr-xr-x. 1 gpadmin gpadmin        6 Apr 23  2013 hostlist_singlenode
drwxrwxr-x. 3 gpadmin gpadmin     4096 Oct 27 23:56 share
-rwxrwxr-x. 1 gpadmin gpadmin     1983 Oct 27 23:56 pgcrypto_install.sh
drwxrwxr-x. 3 gpadmin gpadmin     4096 Oct 27 23:56 lib
-rw-rw-r--. 1 gpadmin gpadmin   144092 Nov 17 18:09 0
-rw-rw-r--. 1 gpadmin gpadmin       81 Nov 18 08:40 table_list.txt
-rw-r--r--. 1 gpadmin gpadmin   501760 Nov 19 20:07 pgcrypto-1.1.3.0-4609.x86_64.tar
drwxrwxr-x. 2 gpadmin gpadmin     4096 Nov 23 13:15 sachi
-rw-rw-r--. 1 gpadmin gpadmin     4515 Nov 27 11:41 gpinitsystem_singlenode.orig
-rw-rw-r--. 1 gpadmin gpadmin      292 Nov 27 12:50 gpinitsystem_singlenode
drwxrwxr-x. 2 gpadmin gpadmin     4096 Jan 23 08:53 gpAdminLogs
-rw-rw-r--. 1 gpadmin gpadmin  3133719 Jan 23 08:54 gpdetective20140123085342.tar.bz2
[gpadmin@sachi ~]$ 

Let's look at the content of the file.

[gpadmin@sachi ~]$ tar jxf gpdetective20140123085342.tar.bz2

It will create a directory gpdetective20140123085342 and bunch of files and directories under it.
The gadget spec URL could not be found
[gpadmin@sachi ~]$ cd gpdetective20140123085342
[[gpadmin@sachi gpdetective20140123085342]$ ls -ltr
total 1268
-rw-rw-r--. 1 gpadmin gpadmin     231 Jan 23 08:53 version_at_initdb.log
-rw-rw-r--. 1 gpadmin gpadmin     391 Jan 23 08:53 resqueue.log
-rw-rw-r--. 1 gpadmin gpadmin     433 Jan 23 08:53 pg_stat_resqueue.log
-rw-rw-r--. 1 gpadmin gpadmin     696 Jan 23 08:53 pg_stat_database.log
-rw-rw-r--. 1 gpadmin gpadmin    1152 Jan 23 08:53 pg_roles.log
-rw-rw-r--. 1 gpadmin gpadmin     133 Jan 23 08:53 pg_filespace.log
-rw-rw-r--. 1 gpadmin gpadmin     305 Jan 23 08:53 pg_filespace_entry.log
-rw-rw-r--. 1 gpadmin gpadmin     340 Jan 23 08:53 pgdatabase.log
-rw-rw-r--. 1 gpadmin gpadmin     251 Jan 23 08:53 mastermirror.log
-rw-rw-r--. 1 gpadmin gpadmin     133 Jan 23 08:53 interfaces.log
-rw-rw-r--. 1 gpadmin gpadmin       6 Jan 23 08:53 hosts
-rw-rw-r--. 1 gpadmin gpadmin     377 Jan 23 08:53 dbsizes.log
-rw-rw-r--. 1 gpadmin gpadmin     134 Jan 23 08:53 db_interfaces.log
-rw-rw-r--. 1 gpadmin gpadmin     659 Jan 23 08:53 config.log
-rw-rw-r--. 1 gpadmin gpadmin     109 Jan 23 08:53 config_history.log
drwxrwxr-x. 2 gpadmin gpadmin    4096 Jan 23 08:53 backupmaster
-rw-rw-r--. 1 gpadmin gpadmin     393 Jan 23 08:53 autovacuum.log
drwxrwxr-x. 4 gpadmin gpadmin    4096 Jan 23 08:53 sachi
drwxrwxr-x. 3 gpadmin gpadmin    4096 Jan 23 08:53 master
-rw-rw-r--. 1 gpadmin gpadmin     927 Jan 23 08:53 gpstate_mirrors.log
-rw-rw-r--. 1 gpadmin gpadmin    1352 Jan 23 08:53 gpstate_standby_master.log
-rw-rw-r--. 1 gpadmin gpadmin   76998 Jan 23 08:53 gpcheckos_master.log
-rw-rw-r--. 1 gpadmin gpadmin   71825 Jan 23 08:53 gpcheckos_segments.log
-rw-rw-r--. 1 gpadmin gpadmin     645 Jan 23 08:53 gpcheckcat_sachi.log
-rw-rw-r--. 1 gpadmin gpadmin     648 Jan 23 08:53 gpcheckcat_postgres.log
-rw-rw-r--. 1 gpadmin gpadmin     647 Jan 23 08:53 gpcheckcat_gpadmin.log
-rw-rw-r--. 1 gpadmin gpadmin    5854 Jan 23 08:54 pg_dumpall.log
-rw-rw-r--. 1 gpadmin gpadmin 1026325 Jan 23 08:54 catalog_q1.log
-rw-rw-r--. 1 gpadmin gpadmin    8552 Jan 23 08:54 error.log

Let's look at some of the important information included in this file.

[gpadmin@sachi gpdetective20140123085342]$ cat version_at_initdb.log
SQL: 
            SELECT * FROM pg_catalog.gp_version_at_initdb
        
schemaversion | productversion                    | 
--------------+-----------------------------------+
2             | 4.2.2.4 build 1 Community Edition | 
[gpadmin@sachi gpdetective20140123085342]$ cat resqueue.log
SQL: 
            SELECT * FROM pg_catalog.pg_resqueue
        
rsqname    | rsqcountlimit | rsqcostlimit | rsqovercommit | rsqignorecostlimit | 
-----------+---------------+--------------+---------------+--------------------+
perf_test  | 2.0           | 100000000.0  | t             | 100000.0           | 
pg_default | 20.0          | -1.0         | f             | 0.0                | 
[gpadmin@sachi gpdetective20140123085342]$ cat pg_stat_resqueue.log
SQL: 
            SELECT * FROM pg_catalog.pg_stat_resqueues            
        
queueid | queuename  | n_queries_exec | n_queries_wait | elapsed_exec | elapsed_wait | 
--------+------------+----------------+----------------+--------------+--------------+
16995   | perf_test  | 0              | 0              | 0            | 0            | 
6055    | pg_default | 0              | 0              | 0            | 0            | 
[gpadmin@sachi gpdetective20140123085342]$ cat pg_stat_database.log
SQL: 
            SELECT * FROM pg_catalog.pg_stat_database            
        
datid | datname   | numbackends | xact_commit | xact_rollback | blks_read | blks_hit | 
------+-----------+-------------+-------------+---------------+-----------+----------+
16992 | gpadmin   | 1           | 0           | 0             | 0         | 0        | 
10900 | postgres  | 1           | 0           | 0             | 0         | 0        | 
16993 | sachi     | 0           | 0           | 0             | 0         | 0        | 
1     | template1 | 0           | 0           | 0             | 0         | 0        | 
10899 | template0 | 0           | 0           | 0             | 0         | 0        | 
The gadget spec URL could not be found
[gpadmin@sachi gpdetective20140123085342]$ cat pg_roles.log
SQL: 
            SELECT * FROM pg_catalog.pg_roles
        
rolname | rolsuper | rolinherit | rolcreaterole | rolcreatedb | rolcatupdate | rolcanlogin | rolconnlimit | rolpassword | rolvaliduntil | rolconfig | rolresqueue | oid   | rolcreaterextgpfd | rolcreaterexthttp | rolcreatewextgpfd | rolcreaterexthdfs | rolcreatewexthdfs | 
--------+----------+------------+---------------+-------------+--------------+-------------+--------------+-------------+---------------+-----------+-------------+-------+-------------------+-------------------+-------------------+-------------------+-------------------+
gpadmin | t        | t          | t             | t           | t            | t           | -1           | ********    | None          | None      | 6055        | 10    | t                 | t                 | t                 | t                 | t                 | 
sachi   | f        | t          | f             | f           | f            | t           | -1           | ********    | None          | None      | 6055        | 16994 | f                 | f                 | f                 | f                 | f                 | 
[gpadmin@sachi gpdetective20140123085342]$ cat pg_filespace.log
SQL: 
            SELECT * FROM pg_catalog.pg_filespace
        
fsname    | fsowner | 
----------+---------+
pg_system | 10      | 
[gpadmin@sachi gpdetective20140123085342]$ cat dbsizes.log
SQL: 
            SELECT datname,pg_size_pretty(pg_database_size( datname )) 
            FROM ( SELECT datname 
                   FROM pg_database 
                   WHERE datname NOT IN ('template0', 'postgres')) d
        
datname   | pg_size_pretty | 
----------+----------------+
gpadmin   | 9821 kB        | 
sachi     | 9796 kB        | 
template1 | 9805 kB        | 
[gpadmin@sachi gpdetective20140123085342]$ cat error.log
gpdetective version 1.01

TEST: getConfiguration
SQL: 
           SELECT hostname, fselocation 
           FROM   gp_segment_configuration
           INNER JOIN pg_filespace_entry on (fsedbid=dbid)
           INNER JOIN pg_filespace fs on (fs.oid = fsefsoid and fsname = 'pg_system')
           WHERE  content = -1  AND preferred_role = 'p'
        
SQL: 
           SELECT hostname, fselocation
           FROM   gp_segment_configuration
           INNER JOIN pg_filespace_entry on (fsedbid=dbid)
           INNER JOIN pg_filespace fs on (fs.oid = fsefsoid and fsname = 'pg_system')
           WHERE  content < 0 AND preferred_role != 'p'
        
SQL: 
           SELECT distinct hostname
           FROM   gp_segment_configuration
           WHERE  content >= 0
        
SQL: 
           SELECT dbid, hostname || ':' || fselocation
           FROM   gp_segment_configuration
           INNER JOIN pg_filespace_entry on (fsedbid=dbid)
           INNER JOIN pg_filespace fs on (fs.oid = fsefsoid and fsname = 'pg_system')
           WHERE  content >= 0  AND  preferred_role = 'p'
        
SQL: 
           SELECT dbid, hostname || ':' || fselocation
           FROM   gp_segment_configuration
           INNER JOIN pg_filespace_entry on (fsedbid=dbid)
           INNER JOIN pg_filespace fs on (fs.oid = fsefsoid and fsname = 'pg_system')
           WHERE  content >= 0  AND  preferred_role != 'p'
        
SQL: 
           SELECT datname FROM pg_database
           WHERE  datname not in ('template0', 'template1')
        
SQL: 
           SELECT * from gp_segment_configuration
        
SQL: 
           SELECT * from gp_configuration_history
        
SQL: 
            SELECT * FROM gp_configuration_history
        
SQL: 
            SELECT * FROM pg_catalog.pg_resqueue
        
SQL: 
            SELECT * FROM pg_catalog.gp_master_mirroring
        
SQL: 
            SELECT * FROM pg_catalog.gp_version_at_initdb
        
SQL: 
            SELECT * FROM pg_catalog.gp_interfaces
        
SQL: 
            SELECT * FROM pg_catalog.gp_db_interfaces
        
SQL: 
            SELECT * FROM pg_catalog.pg_autovacuum
        
SQL: 
            SELECT * FROM pg_catalog.gp_pgdatabase
        
SQL: 
            SELECT datname,pg_size_pretty(pg_database_size( datname )) 
            FROM ( SELECT datname 
                   FROM pg_database 
                   WHERE datname NOT IN ('template0', 'postgres')) d
        
SQL: 
            SELECT * FROM pg_catalog.pg_stat_resqueues            
        
SQL: 
            SELECT * FROM pg_catalog.pg_stat_database            
        
SQL: 
            SELECT * FROM pg_catalog.pg_roles
        
SQL: 
            SELECT * FROM pg_catalog.pg_filespace
        
SQL: 
            SELECT * FROM pg_catalog.pg_filespace_entry
        

TEST: copyLogs

TEST:     get master logs
localhost$ scp /home/gpmaster/gpsne-1/postgresql.conf master/postgresql.conf 
localhost$ scp /home/gpmaster/gpsne-1/global/pg_database master/pg_database 
localhost$ scp /home/gpmaster/gpsne-1/global/pg_auth master/pg_pgauth 

TEST:     get primary segment logs
localhost$ scp -q sachi:/disk2/gpdata2/gpsne1/postgresql.conf sachi/postgresql_gpsne1.conf 
localhost$ scp -q sachi:/disk1/gpdata1/gpsne0/postgresql.conf sachi/postgresql_gpsne0.conf 

TEST:     get gpAdminLogs
localhost$ cp -R /home/gpadmin/gpAdminLogs/ master/gpAdminLogs 

TEST:     get primary segment pg_changetracking FILEREP_CONFIG_LOG and FILEREP_LOG  logs
localhost$ mkdir sachi/pg_changetracking_gpsne1 
localhost$ scp -qr sachi:/disk2/gpdata2/gpsne1/pg_changetracking/FILEREP_CONFIG_LOG sachi/pg_changetracking_gpsne1/FILEREP_CONFIG_LOG 
scp: /disk2/gpdata2/gpsne1/pg_changetracking/FILEREP_CONFIG_LOG: No such file or directory
localhost$ scp -qr sachi:/disk2/gpdata2/gpsne1/pg_changetracking/FILEREP_LOG sachi/pg_changetracking_gpsne1/FILEREP_LOG 
scp: /disk2/gpdata2/gpsne1/pg_changetracking/FILEREP_LOG: No such file or directory
localhost$ mkdir sachi/pg_changetracking_gpsne0 
localhost$ scp -qr sachi:/disk1/gpdata1/gpsne0/pg_changetracking/FILEREP_CONFIG_LOG sachi/pg_changetracking_gpsne0/FILEREP_CONFIG_LOG 
scp: /disk1/gpdata1/gpsne0/pg_changetracking/FILEREP_CONFIG_LOG: No such file or directory
localhost$ scp -qr sachi:/disk1/gpdata1/gpsne0/pg_changetracking/FILEREP_LOG sachi/pg_changetracking_gpsne0/FILEREP_LOG 
scp: /disk1/gpdata1/gpsne0/pg_changetracking/FILEREP_LOG: No such file or directory

TEST:     get mirror segment pg_changetracking FILEREP_CONFIG_LOG and FILEREP_LOG  logs
localhost$ scp -qr sachi:/disk2/gpdata2/gpsne1/gp_transaction_files_filespace sachi/pg_changetracking_gpsne1/gp_transaction_files_filespace 
scp: /disk2/gpdata2/gpsne1/gp_transaction_files_filespace: No such file or directory
localhost$ scp -qr sachi:/disk2/gpdata2/gpsne1/gp_temporary_files_filespace sachi/pg_changetracking_gpsne1/gp_temporary_files_filespace 
scp: /disk2/gpdata2/gpsne1/gp_temporary_files_filespace: No such file or directory
localhost$ scp -qr sachi:/disk1/gpdata1/gpsne0/gp_transaction_files_filespace sachi/pg_changetracking_gpsne0/gp_transaction_files_filespace 
scp: /disk1/gpdata1/gpsne0/gp_transaction_files_filespace: No such file or directory
localhost$ scp -qr sachi:/disk1/gpdata1/gpsne0/gp_temporary_files_filespace sachi/pg_changetracking_gpsne0/gp_temporary_files_filespace 
scp: /disk1/gpdata1/gpsne0/gp_temporary_files_filespace: No such file or directory
localhost$ cp /home/gpmaster/gpsne-1/gp_transaction_files_filespace master/gp_transaction_files_filespace 
cp: cannot stat `/home/gpmaster/gpsne-1/gp_transaction_files_filespace': No such file or directory
localhost$ cp /home/gpmaster/gpsne-1/gp_temporary_files_filespace master/gp_temporary_files_filespace 
cp: cannot stat `/home/gpmaster/gpsne-1/gp_temporary_files_filespace': No such file or directory
The gadget spec URL could not be found
TEST: systemInfo
localhost$ gpstate -m 
localhost$ gpstate -f 
localhost$ gpcheckos -vm -h localhost 
/usr/local/greenplum-db/./bin/gpcheckos:19: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  import getopt, popen2, pickle, time
localhost$ gpcheckos -vf hosts 
/usr/local/greenplum-db/./bin/gpcheckos:19: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  import getopt, popen2, pickle, time

TEST: checkCatalog
localhost$ /usr/local/greenplum-db-4.2.2.4/bin/lib/gpcheckcat -h localhost -p 5432 -U gpadmin gpadmin 
Error: option -h not recognized
localhost$ /usr/local/greenplum-db-4.2.2.4/bin/lib/gpcheckcat -h localhost -p 5432 -U gpadmin postgres 
Error: option -h not recognized
localhost$ /usr/local/greenplum-db-4.2.2.4/bin/lib/gpcheckcat -h localhost -p 5432 -U gpadmin sachi 
Error: option -h not recognized

TEST: pg_dumpall
localhost$ p g _ d u m p a l l   - - s c h e m a - o n l y   - - g p - s y n t a x   - h   l o c a l h o s t   - p   5 4 3 2   - U   g p a d m i n 
SQL: 
           SELECT 
             nspname || '.' || relname, 
             attname, 
             starelid,
             staattnum,
             stanullfrac,
             stawidth,
             stadistinct,
             stakind1,
             stakind2,
             stakind3,
             stakind4,
             staop1,
             staop2,
             staop3,
             staop4,
             array_to_string(stanumbers1, ','),
             array_to_string(stanumbers2, ','),
             array_to_string(stanumbers3, ','),
             array_to_string(stanumbers4, ','),
             array_to_string(stavalues1, ','),
             array_to_string(stavalues2, ','),
             array_to_string(stavalues3, ','),
             array_to_string(stavalues4, ',')
           FROM   pg_class c, 
                  pg_namespace n, 
                  pg_attribute a, 
                  pg_statistic s 
           WHERE  n.oid = c.relnamespace 
             AND  c.oid = a.attrelid 
             AND  c.oid = s.starelid 
             AND  a.attnum = s.staattnum 
          ORDER BY 1, 2
        

TEST: crashReport
localhost$ uname
localhost$ /sbin/sysctl -n kernel.core_pattern 
localhost$ /sbin/sysctl -n kernel.core_uses_pid 
localhost$ ls -1 |/usr/libexec/abrt-hook-ccpp /var/spool/abrt * * * * * * localhost postgres 636f726500*
localhost$ /usr/bin/gdb -batch -n -c /var/spool/abrt:
/usr/bin/gdb: symbol lookup error: /usr/bin/gdb: undefined symbol: PyUnicodeUCS4_FromEncodedObject
sachi$ uname
sachi$ /sbin/sysctl -n kernel.core_pattern 
sachi$ /sbin/sysctl -n kernel.core_uses_pid 
sachi$ ls -1 |/usr/libexec/abrt-hook-ccpp /var/spool/abrt * * * * * * sachi postgres 636f726500*

TEST: makeTar
[gpadmin@sachi gpdetective20140123085342]$ 

The gadget spec URL could not be found
The gadget spec URL could not be found

The gadget spec URL could not be found

The gadget spec URL could not be found

The gadget spec URL could not be found

The gadget spec URL could not be found

The gadget spec URL could not be found
Comments