Oracle RAC: Cluster reconfiguration steps

Cluster reconfiguration steps

The cluster reconfiguration process triggers IMR, and a seven-step process ensures complete reconfiguration.

1. Name service is frozen. The CGS contains an internal database of all the members/ instances in the cluster with all their configuration and servicing details. The name service provides a mechanism to address this configuration data in a structured and synchronized manner.
2. Lock database (IDLM) is frozen. The lock database is frozen to prevent processes from obtaining locks on resources that were mastered by the departing/dead instance.
3. Determination of membership and validation and IMR.
4. Bitmap rebuild takes place, instance name and uniqueness verification. CGS must synchronize the cluster to be sure that all members get the reconfiguration event and that they all see the same bitmap.
5. Delete all dead instance entries and republish all names newly configured.
6. Unfreeze and release name service for use.
7. Hand over reconfiguration to GES/GCS.

Now that you know when IMR starts and node evictions take place, let's look at the corresponding messages in the alert log and LMON trace files to get a better picture. (The logs have been edited for brevity. Note all the lines in boldface define the most important steps in IMR and the handoff to other recovery steps in CGS.)

Problem with a Node Assume a four-node cluster (instances A, B, C, and D), in which instance C has a problem communicating with other nodes because its private link is down. All other services on this node are assumed to be working normally.

Alter log on instance C:

Alter log on instance A:

LMON trace file on instance A:

Here's the LMON trace file on instance A:

Death of a Member For the same four-node cluster (A, B, C, and D), instance C has died unexpectedly:

kjxgrnbrisalive: (3, 4) not beating, HB: 561027672, 561027672 *** 2005-06-19 00:30:52.018 kjxgrnbrdead: Detected death of 3, initiating reconfig kjxgrrcfgchk: Initiating reconfig, reason 2 *** 2005-06-19 00:30:57.035 kjxgmrcfg: Reconfiguration started, reason 2 kjxgmcs: Setting state to 6 0. *** 2005-06-19 00:30:57.037 Name Service frozen kjxgmcs: Setting state to 6 1. *** 2005-06-19 00:30:57.239 Obtained RR update lock for sequence 6, RR seq 6 *** 2005-06-19 00:33:27.261 Voting results, upd 0, seq 7, bitmap: 0 2 Evicting mem 3, stat 0x0007 err 0x0001 kjxgmps: proposing substate 2 kjxgmcs: Setting state to 7 2. Performed the unique instance identification check kjxgmps: proposing substate 3 kjxgmcs: Setting state to 7 3. Name Service recovery started Deleted all dead-instance name entries kjxgmps: proposing substate 4 kjxgmps: proposing substate 4 kjxgmcs: Setting state to 7 4. Multicasted all local name entries for publish Replayed all pending requests kjxgmps: proposing substate 5 kjxgmcs: Setting state to 7 5. Name Service normal Name Service recovery done *** 2005-06-19 00:33:27.266 kjxgmps: proposing substate 6 … … … kjxgmps: proposing substate 2

GES/GCS recovery starts here:

Use the following table of contents to navigate to chapter excerpts or click here to view RAC Troubleshooting in its entirety.

Oracle Database 10g: Real Application Clusters Handbook
  Home: Oracle RAC troubleshooting: Introduction
  1: Oracle RAC: Log directory structure in cluster ready services
  2: Oracle RAC: Log directory structure in Oracle RDBMS
  3: Oracle RAC and the Lamport algorithm
 4: Oracle RAC: ON and OFF
  5: Oracle RAC: Database performance issues
  6: Oracle RAC: Debugging node eviction issues
  7: Oracle RAC: Member voting
  8: Oracle RAC: Cluster reconfiguration steps
  9: Oracle RAC: Debugging CRS and GSD using DTRACING