Recovering from Segment Failures

Greenplum Database can perform three types of segment recovery: full, differential, and incremental (the default).

Full recovery: Full recovery recovers all segments. Specifically, it erases all data files and directories on the current mirror segment and copies to the mirror segment the exact contents of the current primary segment. Full recovery uses the pg_basebackup utility to copy files.

With full recovery, you may recover:

to the current host -- known as "in-place recovery"
to a different host within the current cluster
to a new host outside of the current cluster

Differential recovery

Differential recovery performs a filesystem-level diff between the primary and mirror segments, and copies from the primary to the mirror only those files that have changed on the primary. With differential recovery, you may only do in-place recovery. Differential recovery uses the rsync command to copy files.

Note

Differential recovery is not supported when using input configuration files (gprecoverseg -i).

Incremental recovery (default)

Incremental recovery brings the mirror segment contents into sync with the primary segment contents with the aid of write-ahead log files (WAL files). With incremental recovery, you may only do in-place recovery. Incremental recovery uses the pg_rewind utility to copy files.

By default, gprecoverseg performs an incremental recovery, placing the mirror into Synchronizing mode, which starts to replay the recorded changes from the primary onto the mirror. If the incremental recovery cannot be completed, the recovery fails and you should run gprecoverseg again with the -F option, to perform full recovery. This causes the primary to copy all of its data to the mirror.

Note

After a failed incremental recovery attempt you must perform a full recovery.

Whenever possible, you should perform an incremental recovery rather than a full recovery, as incremental recovery is substantially faster. If you do need to perform an in-place full recovery, you can speed up in-place full recovery with gprecoverseg's --differential option, which causes gprecoverseg to skip recovery of any files and directories that are unchanged.

This topic walks you through what to do when one or more segments or hosts are down and you want to recover the down segments. The recovery path you follow depends primarily which of these 3 scenarios fits your circumstances:

you want to recover in-place to the current host
you want to recover to a different host, within the cluster
you want to recover to a new host, outside of the cluster

The steps you follow within these scenarios can vary, depending on:

whether you want to do an incremental or a full recovery
whether you want to recover all segments or just a subset of segments

Note

Incremental recovery is only possible when recovering segments to the current host (in-place recovery).

This topic is divided into the following sections:

Prerequisites
Recovery Scenarios
Post-Recovery Tasks

Prerequisites

Mirroring is enabled for all segments.
You've already identified which segments have failed. If necessary, see the topic Checking for Failed Segments.
The master host can connect to the segment host.
All networking or hardware issues that caused the segment to fail have been resolved.

Recovery Scenarios

This section documents the steps for the 3 distinct segment recovery scenarios. Follow the link to instructions that walk you through each scenario.

Recover In-Place to Current Host
- Incremental Recovery
- Full Recovery
- Differential Recovery
Recover to A Different Host within the Cluster
Recover to A New Host, Outside of the Cluster

Recover In-Place to Current Host

When recovering in-place to the current host, you may choose between incremental recovery (the default), full recovery, and differential recovery.

Incremental Recovery

Follow these steps for incremental recovery:

To recover all segments, run gprecoverseg with no options:
```
gprecoverseg
```
To recover a subset of segments:
1. Manually create a recover_config_file file in a location of your choice, where each segment to recover has its own line with format failedAddress|failedPort|failedDataDirectory or failedHostname|failedAddress|failedPort|failedDataDirectory
  
  For multiple segments, create a new line for each segment you want to recover, specifying the hostname the address, port number and data directory for each down segment. For example:
```
failedAddress1|failedPort1|failedDataDirectory1
failedAddress2|failedPort2|failedDataDirectory2
failedAddress3|failedPort3|failedDataDirectory3
```
  or
```
failedHostname1|failedAddress1|failedPort1|failedDataDirectory1
failedHostname2|failedAddress2|failedPort2|failedDataDirectory2
failedHostname2|failedAddress3|failedPort3|failedDataDirectory3
```
2. Alternatively, generate a sample recovery file using the following command; you may edit the resulting file if necessary:
```
$ gprecoverseg -o /home/gpadmin/recover_config_file
```
3. Pass the recover_config_file to the gprecoverseg -i command:
```
$ gprecoverseg -i /home/gpadmin/recover_config_file
```
Perform the post-recovery tasks summarized in the section Post-Recovery Tasks.

Full Recovery

To recover all segments, run gprecoverseg -F:
```
gprecoverseg -F
```
To recover specific segments:
1. Manually create a recover_config_file file in a location of your choice, where each segment to recover has its own line with following format:
```
failedAddress1|failedPort1|failedDataDirectory1<SPACE>failedAddress2|failedPort2|failedDataDirectory2
```
  or
```
failedHostname1|failedAddress1|failedPort1|failedDataDirectory1<SPACE>failedHostname2|failedAddress2|failedPort2|failedDataDirectory2
```
  Note the literal SPACE separating the lines.
2. Alternatively, generate a sample recovery file using the following command and edit the resulting file to match your desired recovery configuration:
```
$ gprecoverseg -o /home/gpadmin/recover_config_file
```
3. Run the following command, passing in the config file generated in the previous step:
```
$ gprecoverseg -i recover_config_file
```
Perform the post-recovery tasks summarized in the section Post-Recovery Tasks.

Differential Recovery

Follow these steps for differential recovery:

Run gprecoverseg --differential

Recover to A Different Host within the Cluster

Note

Only full recovery is possible when recovering to a different host in the cluster.

Follow these steps to recover all segments or just a subset of segments to a different host in the cluster:

Manually create a recover_config_file file in a location of your choice, where each segment to recover has its own line with following format:
```
failedAddress|failedPort|failedDataDirectory<SPACE>newAddress|newPort|newDataDirectory
```
or
```
failedHostname|failedAddress|failedPort|failedDataDirectory<SPACE>newHostname|newAddress|newPort|newDataDirectory
```
Note the literal SPACE separating the details of the down segment from the details of where the segment will be recovered to.

Alternatively, generate a sample recovery file using the following command and edit the resulting file to match your desired recovery configuration:
```
$ gprecoverseg -o -p /home/gpadmin/recover_config_file
```
Run the following command, passing in the config file generated in the previous step:
```
$ gprecoverseg -i recover_config_file
```
Perform the post-recovery tasks summarized in the section Post-Recovery Tasks.

Recover to A New Host, Outside of the Cluster

Follow these steps if you are planning to do a hardware refresh on the host the segments are running on.

Note

Only full recovery is possible when recovering to a new host.

Requirements for New Host

The new host must:

have the same Greenplum Database software installed and configured as the failed host
have the same hardware and OS configuration as the failed host (same hostname, OS version, OS configuration parameters applied, locales, gpadmin user account, data directory locations created, ssh keys exchanged, number of network interfaces, network interface naming convention, and so on)
have sufficient disk space to accommodate the segments
be able to connect password-less with all other existing segments and Greenplum master.

Steps to Recover to a New Host

Bring up the new host
Run the following command to recover all segments to the new host:
```
gprecoverseg -p <new_host_name>
```
You may also specify more than one host. However, be sure you do not trigger a double-fault scenario when recovering to two hosts at a time.
```
gprecoverseg -p <new_host_name1>,<new_host_name2>
```
Note

In the case of multiple failed segment hosts, you can specify the hosts to recover to with a comma-separated list. However, it is strongly recommended to recover to one host at a time. If you must recover to more than one host at a time, then it is critical to ensure that a double fault scenario does not occur, in which both the segment primary and corresponding mirror are offline.
Perform the post-recovery tasks summarized in the section Post-Recovery Tasks.

Post-Recovery Tasks

Follow these steps once gprecoverseg has completed:

Validate segement status and preferred roles:
```
select * from gp_segment_configuration
```
Monitor mirror synchronization progress:
```
gpstate -e
```
If necessary, run the following command to return segments to their preferred roles:
```
gprecoverseg -r
```