Standby Master DCA showing "not synchronized" . what to do?

20120829:12:31:02:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:-Obtaining Segment details from master...

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:-Standby master details

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:-----------------------

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby address = smdw

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby data directory = /data/master/gpseg-1

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby port = 1589

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby PID = 12498

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:- Standby status = Standby host passive

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--------------------------------------------------------------

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--gp_master_mirroring table

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--------------------------------------------------------------

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--Summary state: Not Synchronized

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--Detail state: Unexpected error

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--Log time: 2012-08-29 31:44:21-03

20120829:12:31:05:025122 gpstate:abc0122pl000200:gpadmin-[INFO]:--------------------------------------------------------------

Solution:run the following command in the master from gpadmin account to fix the problem,

gpadmin@master$gpinitstandby -n -M fast

Note: this command will shutdown GPDB firstly, so please run it in outage window.

gpinitstandby

Adds and/or initializes a standby master host for a Greenplum Database system.

Synopsis

gpinitstandby { -s standby_hostname | -r | -n }

[-M smart | -M fast] [-a] [-q] [-D] [-L]

[-l logfile_directory]

gpinitstandby -? | -v

Description

The gpinitstandby utility adds a backup master host to your Greenplum Database system. If your system has an existing backup master host configured, use the -r option to remove it before adding the new standby master host.

Before running this utility, make sure that the Greenplum Database software is installed on the backup master host and that you have exchanged SSH keys between hosts. Also make sure that the master port is set to the same port number on the master host and the backup master host. See the Greenplum Database Installation Guide for instructions. This utility should be run on the currently active primary master host.

The utility will perform the following steps:

•Shutdown your Greenplum Database system

•Update the Greenplum Database system catalog to remove the existing backup master host information (if the -r option is supplied)

•Update the Greenplum Database system catalog to add the new backup master host information (use the -n option to skip this step)

•Edit the pg_hba.conf files of the segment instances to allow access from the newly added standby master.

•Setup the backup master instance on the alternate master host

•Start the synchronization process

•Restart your Greenplum Database system

A backup master host serves as a ‘warm standby’ in the event of the primary master host becoming unoperational. The backup master is kept up to date by a transaction log replication process (gpsyncagent), which runs on the backup master host and keeps the data between the primary and backup master hosts synchronized. If the primary master fails, the log replication process is shut down, and the backup master can be activated in its place by using the utility. Upon activation of the backup master, the replicated logs are used to reconstruct the state of the master host at the time of the last successfully committed transaction.

The activated standby master effectively becomes the Greenplum Database master, accepting client connections on the master port and performing normal master operations such as SQL command processing and workload management.

Options

-a (do not prompt)

Do not prompt the user for confirmation.

-D (debug)

Sets logging level to debug.

-l logfile_directory

The directory to write the log file. Defaults to ~/gpAdminLogs.

-L (leave database stopped)

Leave Greenplum Database in a stopped state after removing the warm standby master.

-M fast (fast shutdown - rollback)

Use fast shut down when stopping Greenplum Database at the beginning of the standby initialization process. Any transactions in progress are interrupted and rolled back.

-M smart (smart shutdown - warn)

Use smart shut down when stopping Greenplum Database at the beginning of the standby initialization process. If there are active connections, this command fails with a warning. This is the default shutdown mode.

-n (resynchronize)

Use this option if you already have a standby master configured, and just want to resynchronize the data between the primary and backup master host. The Greenplum system catalog tables will not be updated.

-q (no screen output)

Run in quiet mode. Command output is not displayed on the screen, but is still written to the log file.

-r (remove standby master)

Removes the currently configured standby master host from your Greenplum Database system.

-s standby_hostname

The host name of the standby master host.

-v (show utility version)

Displays the version, status, last updated date, and check sum of this utility.

-? (help)

Displays the online help.

Examples

Add a backup master host to your Greenplum Database system and start the synchronization process:

gpinitstandby -s host09

Remove the existing backup master from your Greenplum system configuration:

gpinitstandby -r