On different Oracle Clusterware releases, storage for OCR and voting disk varies: raw devices on 10gR2, block devices on 11gR1. Starting with 11g R2, OCR file and voting disk files can be stored in ASM. Unless you are upgrading your Clusterware from 11gR1 to 11gR2 where you can keep the voting disk in the block device, you should put the voting disk file in ASM. OCR and voting disk files can be stored in the same ASM diskgroup as Oracle Database files or in a separate ASM diskgroup. The ASM diskgroup that stores voting disk files can be configured with one of the following three redundancy level settings. These are similar to a regular diskgroup for database files, but their requirement for the minimum numbers of failure groups is higher than those diskgroups that only store database files. The number of the voting disk files is determined by the number of the failure groups of the ASM diskgroup that stores the voting disk files:
1. External Redundancy: no mirroring; only one failure group is needed. This provides only one copy of the voting disk file. It is strongly recommended to have an external RAID configuration for this setting.
2. Normal Redundancy: at least three failure groups are needed. This provides three copies of the voting disk files.
3. High Redundancy: at least five failure groups needed. This provides five copies of the voting disk files.
If you decide to store the voting disk file in the same ASM diskgroup as the data files, adding an additional failure group could mean adding a huge disk space, which you may not have. To solve this problem, a new quorum failure group is introduced. When this new failure group is added to a diskgroup, it is only used to store an extra copy of voting disk file, and non-database files will be stored in this failure group. As a result, the size of the disk for this failure group can be much smaller than other disks in the ASM diskgroup. For example, the ASM disk for a quorum failure group can be as small as 300MB to cover the 200MB voting disk file. The following example shows how to use the quorum failgroup clause to include a quorum failure group in a diskgroup:
CREATE DISKGROUP data NORMAL REDUDANCY FAILGROUP fg1 DISK 'ORCL:DATA1' FAILGROUP fg1 DISK 'ORCL:DATA2' QUORUM FAILGROIUP fg3 DISK 'ORCL:OCR1' ATTRIBUTE 'compatible.asm' = 12.1.0.0.0;
In this case DISK ORCL:DATA1 and ORCL:DATA2 can be 200GB each, DISK ORCL:OCR1 can be small as 300MB.
During the Grid Infrastructure installation you will be asked to provide the disks to create the ASM diskgroup for OCR and voting disk files (Figure 5-10). You have an option to select the redundancy level setting of the ASM diskgroup for OCR and voting disks.
Oracle 12c R1 introduced the Grid Infrastructure Management repository (also called Cluster Health Monitor (CHM) repository), which is a central database used to store all the metrics database collected by the System Monitor Service process of the Clusterware in all the cluster nodes. Currently, this repository is configured during Oracle Grid Infrastructure installation. Refer to the CHM section in Chapter 2. Figure 2-6 shows the Grid Infrastructure Management Repository Option that you can select during Grid Infrastructure installation. If you select the Grid Infrastructure Management Repository option, the diskgroup you are going to create needs to be big enough to store the CHM repository as well as OCR and voting disks; otherwise you will get the error message shown in Figure 5-11.
In my test run, I allocated a 20GB diskgroup to store CHM repository, OCR, and voting disks. Once I made this ASM diskgroup a bigger size such as 100 GB, this issue was solved.
Figure 5-10. Select three-ASM disk for the high-redundancy diskgroup
Figure 5-11. The diskgroup has insufficient space for the CHM repository
As part of a successful Grid Infrastructure installation, the VOCR diskgroup was created with three failure diskgroups: VOCR1, VOCR2, and VOCR3. Three copies of voting disk files are stored in this diskgroup, as shown in the following queries:
SQL> SELECT d.PATH, g.NAME, d.FAILGROUP FROM V$ASM_DISK d, V$ASM_DISKGROUP g where d.GROUP_NUMBER = g.GROUP_NUMBER and g.name='OCRVOTDSK';
PATH NAME FAILGROUP --- --- ---OCRL:VOCR1 VOCRVOTDSK VOCR1 OCRL:VOCR2 VOCRVOTDSK VOCR2 OCRL:VOCR3 VOCRVOTDSK VOCR3
Three copies of voting disk files are as follows:
[grid@kr720n1 bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group -- --- --- -- 1. ONLINE 8b562c9b2ec34f88bfe8343142318db7 (ORCL:VOCR1) [VOCR]
2. ONLINE 1c88a98d5c9a4f88bf55549b6c2dc298 (ORCL:VOCR2) [VOCR]
3. ONLINE 7581ef0020094fccbf1c8d3bca346eb1 (ORCL:VOCR3) [VOCR]
]
Located three voting disk(s).
Unlike the voting disk files, only one copy of OCR can be stored in one ASM diskgroup or a cluster file system.
Oracle requires at least one OCR copy with the ability to add up to five copies of OCR. If you want to store multiple copies of the OCR, you can store them on multiple ASM diskgroups or a combination of ASM diskgroups and cluster file systems.
In order to add an additional copy of OCR on a different ASM diskgroup (for example, DATA2), you can use the following command:
$ocrconfig -add +DATA2
This example shows how to add an additional copy of your OCR on a cluster file system labeled /u01/data:
$ocrconfig –add /u01/data/ocr.dbf
Another example showing how to move the original OCR to a different ASM diskgroup DATA3 is as follows:
$ocrconfig –replace +VOCR replacement +DATA3
The following example shows how to list all copies of OCR:
$ ocrcheck
Status of Oracle Cluster Registry is as follows : Version : 3
Total space (kbytes) : 262120 Used space (kbytes) : 2860 Available space (kbytes) : 259260 ID : 2322227200
Device/File Name :+VOCR Device/File integrity check succeeded
Device/File Name :+DATA1 Device/File integrity check succeeded Device/File Name :+DATA2 Device/File integrity check succeeded Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
In this example, there are three copies of OCR. They are stored in three ASM diskgroups: +VOC, +DATA1, and +DATA2, respectively.