Oracle … as usual

Oracle by Laurent Leturgez

Category Archives: ASM / ADVM

How Oracle 11.2 clusterware starts, find voting file and ASM SPfile

On a recent discussion on linkedin RACSIG page, Jim Williams explained how ASM SPFILE and Voting files are discovered during OHASD startup phase.

With one of my colleague (Pierre Labrousse), we have built a little video which explain this.

You can see this video here :

Advertisements

How ASM disk header block repair works

In this post, I will explain one of my last works about the ASM disk header block.

First, I will create a TEST tablespace in my orcl database. The TEST’s datafile will be managed by ASM.

[oracle@oel ~]$ . oraenv
ORACLE_SID = [+ASM] ? orcl
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel ~]$ sqlplus / as sysdba
SQL> create tablespace test datafile '+data' size 5M autoextend on maxsize unlimited;
Tablespace created.

SQL>  select instance_name from v$instance;
INSTANCE_NAME
----------------
orcl

SQL> select file_id,tablespace_name,file_name,status,online_status from dba_data_files
  2  /

   FILE_ID TABLESPACE_NAME                FILE_NAME                                          STATUS    ONLINE_
---------- ------------------------------ -------------------------------------------------- --------- -------
         4 USERS                          /u02/oradata/orcl/users01.dbf                      AVAILABLE ONLINE
         3 UNDOTBS1                       /u02/oradata/orcl/undotbs01.dbf                    AVAILABLE ONLINE
         2 SYSAUX                         /u02/oradata/orcl/sysaux01.dbf                     AVAILABLE ONLINE
         1 SYSTEM                         /u02/oradata/orcl/system01.dbf                     AVAILABLE SYSTEM
         5 EXAMPLE                        /u02/oradata/orcl/example01.dbf                    AVAILABLE ONLINE
         6 TEST                           +DATA/orcl/datafile/test.258.798578613             AVAILABLE ONLINE

6 rows selected.

My ASM instance manages two diskgroups DATA (external redundancy) and RL (normal redundancy).

DATA is a 2 disks diskgroup managed by asmlib.

SQL> select group_number,name,type,state from v$asm_diskgroup
  2  /

GROUP_NUMBER NAME TYPE   STATE
------------ ---- ------ -----------
           1 DATA EXTERN MOUNTED
           2 RL   NORMAL MOUNTED
SQL> select path from v$asm_disk where group_number=1;

PATH
---------------------------------
ORCL:ASM1
ORCL:ASM2

And now, we erase the header block of each disk. Header block is the first one of an ASM disk device, and its default size is 4096 bytes (This size is available in _asm_blksize undocumented parameter).

[oracle@oel ~]$ dd if=/dev/zero of=/dev/oracleasm/disks/ASM1 bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 2.2e-05 seconds, 186 MB/s
[oracle@oel ~]$ dd if=/dev/zero of=/dev/oracleasm/disks/ASM2 bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000336 seconds, 12.2 MB/s

Even my block headers destroyed, I can still write in my diskgroup and allocate extents in it.

SQL> alter database datafile 6 resize 10M;

Database altered.

SQL> create table t tablespace TEST as select * from dba_source;

Table created.

SQL> alter system checkpoint;

System altered.

SQL> select file#,checkpoint_time from v$datafile_header where file#=6;

     FILE# CHECKPOINT_TIME
---------- -------------------
         6 05/11/2012 19:27:43

Well, let’s try to restart the rdbms instance:

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 1068937216 bytes
Fixed Size                  2235208 bytes
Variable Size             377488568 bytes
Database Buffers          683671552 bytes
Redo Buffers                5541888 bytes
Database mounted.
Database opened.
SQL> select count(*) from t;

  COUNT(*)
----------
    702070

Damned ! I can still restart it and read all file extents.

There’s something strange in this demo, my ASM disk header is invalid and I still can read files. I will verify with kfed my headers state (only one ASM disk is shown below):

oracle@oel ~]$ kfed read /dev/oracleasm/disks/ASM1
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
2B469E002400 00000000 00000000 00000000 00000000  [................]
  Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[oracle@oel ~]$ sudo /u01/app/oracle/product/11.2.0.3/dbhome_1/bin/kfed read /dev/sdb1
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
2B413559C400 00000000 00000000 00000000 00000000  [................]
  Repeat 255 times
KFED-00322: file not found; arguments: [kfbtTraverseBlock] [Invalid OSM block type] [] [0]

So KFED has confirmed my header blocks are invalid. Now, I will try to unmount and remount the diskgroup to see if there’s any effect on diskgroup mount operation.

[oracle@oel ~]$ sqlplus / as sysasm

SQL> alter diskgroup DATA dismount;

Diskgroup altered.

SQL> alter diskgroup DATA mount;
alter diskgroup DATA mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

So ASM disk header seems to be important for mounting the diskgroup but without effect on asm extent allocation etc.

Now I will repair asm disks with kfed and repair operation to restore my asm header blocks:

[oracle@oel ~]$ kfed repair /dev/oracleasm/disks/ASM1
[oracle@oel ~]$ kfed repair /dev/oracleasm/disks/ASM2

[oracle@oel ~]$ sqlplus / as sysasm

SQL> alter diskgroup DATA mount;

Diskgroup altered.

So KFED is able to restore the header (with the repair operation).

[oracle@oel ~]$ . oraenv
ORACLE_SID = [+ASM] ? orcl
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel ~]$ sqlplus / as sysdba

Connected to an idle instance.

SQL> startup
...
Database opened.
SQL> select count(*) from T;

  COUNT(*)
----------
    702070

And datas in the T table are still there … no problem !

Well, let’s summarize … I erased the header block of both asm disks in my diskgroup without any impact on file extent allocation. The only impact I had was on the mount capability. Next, I have restored the header block of each disk with a kfed repair operation … yes, but without any backup of the disk or the disk header. So I wonder … where was the backup of my header block?

First, I will try a very simple method. I will have a look at all metadata blocks in my asm disk device. I hope I will find another header block. For this purpose, i will use a basic shell script to analyze each block. If the script finds a block with the type KFBTYP_DISKHEAD, it will keep the block position, and print them at the end of script execution:

The basic script:

#!/bin/bash
#set -x
export ORAENV_ASK=NO
export ORACLE_SID=+ASM
. oraenv

i=0
ret=0
tab_cnt=0;
blk_typ='';

while [ "$blk_typ" != "KFBTYP_INVALID" ];
do
  blk_typ=`kfed read $1 blkn=$i | grep kfbh.type | awk '{print $5;}' | sed 's/^ *//g' | sed 's/ *$//g' `
  ret=$?
  if [ "$blk_typ" = "KFBTYP_DISKHEAD" ]; then
    t[$i]=$i
  fi
  let i=$i+1
done
echo "list of header block with KFBTYP_DISKHEAD type"
echo ${t[@]}

Execution results:

[oracle@oel ~]$ ./asm_surgery.sh /dev/oracleasm/disks/ASM1
The Oracle base remains unchanged with value /u01/app/oracle
list of header block with KFBTYP_DISKHEAD type
0 510
[oracle@oel ~]$ ./asm_surgery.sh /dev/oracleasm/disks/ASM2
The Oracle base remains unchanged with value /u01/app/oracle
list of header block with KFBTYP_DISKHEAD type
0 510

Ok, so there is a copy of the header block in the 510th block in my disk. Indeed, as Bane Radulović mentioned it in its blog, a backup copy of ASM disk header is in the second last block of allocation unit 1. So, if my AU is 1Mb and my block size 4096 bytes, a copy of my header block will be available in ((1048576 / 4096) * 2 – 1  = 511), as the block# starts at 0, it is located in the 510th block.

To double check this, I will erase the header block of my first ASM disk, and use kfed repair and strace to see what really happens:

[oracle@oel ~]$ dd if=/dev/zero of=/dev/oracleasm/disks/ASM1 bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 2.3e-05 seconds, 178 MB/s

[oracle@oel ~]$ strace kfed repair /dev/oracleasm/disks/ASM1
.../...

stat("/dev/oracleasm/disks/ASM1", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 17), ...}) = 0
access("/dev/oracleasm/disks/ASM1", F_OK) = 0
statfs("/dev/oracleasm/disks/ASM1", {f_type=0x958459f6, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096}) = 0
open("/dev/oracleasm/disks/ASM1", O_RDWR) = 7
lseek(7, 2088960, SEEK_SET)             = 2088960
read(7, "\1\202\1\1\376\200<\206\371\7"..., 4096) = 4096
lseek(7, 0, SEEK_SET)                   = 0
read(7, ""..., 4096) = 4096
lseek(7, 0, SEEK_SET)                   = 0
write(7, "\1\202\1\1\200\302\206\371\7"..., 4096) = 4096
close(7)                                = 0

This is very interesting.

First operation, it opens the device and read the block after 2088960 bytes. But 2088960/4096 = 510, so it reads the 510th block of the disk. Next it reads the block at position 0 (header block), and then writes in it the content of the 510th block.

Well, now I know there’s a copy of each header block in a secret position !!! (block #510) and kfed uses this block to repair the main header block.

During my tests, I noticed that sometimes, asm disks lost their asm label (I don’t know why). As a consequence, repaired disks won’t be recreated by the oracleasm scandisks (or after a reboot).

[root@oel ~]# ls /dev/oracleasm/disks/*
/dev/oracleasm/disks/ASM1  /dev/oracleasm/disks/ASM2  /dev/oracleasm/disks/ASM3  /dev/oracleasm/disks/ASM4
[root@oel ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Cleaning disk "ASM1"
Scanning system for ASM disks...
[root@oel ~]# ls /dev/oracleasm/disks/*
/dev/oracleasm/disks/ASM2  /dev/oracleasm/disks/ASM3  /dev/oracleasm/disks/ASM4

According the fact that you know which device is corresponding to asmlib disk device, you can relabel this disk with oracleasm renamedisk command (be very careful with this command):

[root@oel ~]# oracleasm renamedisk -f /dev/sdb1 ASM1
Writing disk header: done
Instantiating disk "ASM1": done

[root@oel ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@oel ~]# ls /dev/oracleasm/disks/*
/dev/oracleasm/disks/ASM1  /dev/oracleasm/disks/ASM2  /dev/oracleasm/disks/ASM3  /dev/oracleasm/disks/ASM4

Dump ASM disk header … method #2

In a previous post, I explained how to dump ASM Disk header and how to analyze it. This method, which used kfed binary, is a little bit complex because of fields with a complex name extracted from the internal structure.

Another way to proceed and to see main information about asm disks is to use an undocumented binary located in $ORACLE_HOME/bin named amdu.

If amdu is not available in your $ORACLE_HOME/bin directory, you can build it with the next make command:

[oracle@oel ~]$ make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk iamdu

To dump your ASM diskheader, you just have to execute this command with the -diskstring option and specify the location of your ASM disk devices (default uses ASM lib directory):

[oracle@oel ~]$ amdu -diskstring='/dev/oracleasm/disks/*'
amdu_2012_03_21_11_14_36/

Now, change directory to the output generated and have a look to the report.txt file:

[oracle@oel ~]$ cd amdu_2012_03_21_11_14_36/
[oracle@oel amdu_2012_03_21_11_14_36]$ cat report.txt
-*-amdu-*-
******************************* AMDU Settings ********************************
ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/dbhome_1
System name: Linux
Node name: oel
Release: 2.6.18-274.18.1.0.1.el5
Version: #1 SMP Thu Feb 9 19:07:16 EST 2012
Machine: x86_64
amdu run: 21-MAR-12 11:14:36
Endianess: 1
--------------------------------- Operations ---------------------------------
------------------------------- Disk Selection -------------------------------
 -diskstring '/dev/oracleasm/disks/*'
------------------------------ Reading Control -------------------------------
------------------------------- Output Control -------------------------------
********************************* DISCOVERY **********************************
----------------------------- DISK REPORT N0001 ------------------------------
 Disk Path: /dev/oracleasm/disks/ASM1
 Unique Disk ID:
 Disk Label:
 Physical Sector Size: 512 bytes
 Disk Size: 1019 megabytes
 Group Name: DATA
 Disk Name: DATA_0000
 Failure Group Name: DATA_0000
 Disk Number: 0
 Header Status: 3
 Disk Creation Time: 2011/05/27 10:57:54.822000
 Last Mount Time: 2012/03/21 08:54:10.221000
 Compatibility Version: 0x0b200000(11020000)
 Disk Sector Size: 512 bytes
 Disk size in AUs: 1019 AUs
 Group Redundancy: 1
 Metadata Block Size: 4096 bytes
 AU Size: 1048576 bytes
 Stride: 113792 AUs
 Group Creation Time: 2011/05/27 10:57:54.456000
 File 1 Block 1 location: AU 2
 OCR Present: NO
----------------------------- DISK REPORT N0002 ------------------------------
 Disk Path: /dev/oracleasm/disks/ASM2
 Unique Disk ID:
 Disk Label:
 Physical Sector Size: 512 bytes
 Disk Size: 1019 megabytes
 Group Name: DATA
 Disk Name: DATA_0001
 Failure Group Name: DATA_0001
 Disk Number: 1
 Header Status: 3
 Disk Creation Time: 2011/05/27 10:57:54.822000
 Last Mount Time: 2012/03/21 08:54:10.221000
 Compatibility Version: 0x0b200000(11020000)
 Disk Sector Size: 512 bytes
 Disk size in AUs: 1019 AUs
 Group Redundancy: 1
 Metadata Block Size: 4096 bytes
 AU Size: 1048576 bytes
 Stride: 113792 AUs
 Group Creation Time: 2011/05/27 10:57:54.456000
 File 1 Block 1 location: AU 0
 OCR Present: NO
******************************* END OF REPORT ********************************

As mentioned, now you know if your disk is part of a disk group, the DG Allocation Unit size, disk size, last mounted time etc.

Of course, this tool can be used regarless of the asm instance status.

 

 

Dump ASM disk header

If you want to dump ASM disk header, you can use an Oracle internal tool to obtain information about your disk, diskgroup etc. even if the disk is offline.

This tool is named KFED (Kernel File EDitor). It is fitted by default with an Oracle 11g installation, but you’ll need to build it  if you want to use it with Oracle 10g :

[oracle@oel ~]$ make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk ikfed

Well, now have a closer look to a feature of this tool.

If you want to read information stored on the ASM Disk header, you can use it like this :

[oracle@oel ~]$ kfed read /dev/oracleasm/disks/ASM3 dsk1.dump

Now, you have in the dsk1.dump file the content of your ASM file header “/dev/oracleasm/disks/ASM3”. This file can be easily read by a text editor

 
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:              2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
kfbh.check:                  2930000864 ; 0x00c: 0xaea443e0
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:     ORCLDISKASM3 ; 0x000: length=12
kfdhdb.driver.reserved[0]:    860705601 ; 0x008: 0x334d5341
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum:                        0 ; 0x024: 0x0000
kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:          MIRROR_DG_0000 ; 0x028: length=14
kfdhdb.grpname:               MIRROR_DG ; 0x048: length=9
kfdhdb.fgname:           MIRROR_DG_0000 ; 0x068: length=14
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             32959021 ; 0x0a8: HOUR=0xd DAYS=0x11 MNTH=0xa YEAR=0x7db
kfdhdb.crestmp.lo:           3500063744 ; 0x0ac: USEC=0x0 MSEC=0x3af SECS=0x9 MINS=0x34
kfdhdb.mntstmp.hi:             32959382 ; 0x0b0: HOUR=0x16 DAYS=0x1c MNTH=0xa YEAR=0x7db
kfdhdb.mntstmp.lo:            505578496 ; 0x0b4: USEC=0x0 MSEC=0xa1 SECS=0x22 MINS=0x7
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact:                    113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize:                    1019 ; 0x0c4: 0x000003fb
kfdhdb.pmcnt:                         2 ; 0x0c8: 0x00000002
kfdhdb.fstlocn:                       1 ; 0x0cc: 0x00000001
kfdhdb.altlocn:                       2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn:                      2 ; 0x0d4: 0x00000002
kfdhdb.redomirrors[0]:                0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]:                0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]:                0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]:                0 ; 0x0de: 0x0000

Now, we can read some information about the file : on the structure “kfdhdb”, at the offset 0x048, and coded on 9 bytes, the name of the diskgroup which owns this ASM disk  file.

Most important information are detailed below :

* kfbh.endian: Endian used on this disk : 1 for little endian.

* kfdhdb.driver.provstr: Provision String used for ASM (which means in our case : ORCL:DISKASM3)

* kfdhdb.grptyp: type of diskgroup the disk is attached to.

* kfdhdb.hdrsts: header status. Here, the disk is a member of the diskgroup.

* kfdhdb.dskname: disk name in the disk group

* kfdhdb.grpname: disk group name

* kfdhdb.fgname: failure group name which owns the disk

* kfdhdb.secsize: sector size

* kfdhdb.blksize:  block size

* kfdhdb.ausize: allocation unit size

If you want to rename the diskgroup the disk belongs to, you can edit the dumpfile and use the “merge” command of KFED to apply changes to the disk header.

[oracle@oel ~]$ kfed merge /dev/oracleasm/disks/ASM3 text=dsk1.dump

Be careful when you use the “merge” command because, it seems the diskgroup name, or disk name is coded with a fixed length, so if you change the name, and this one is based on a 4 bytes word, rename it to a 4 bytes word.

Off course, using kfed is not supported by Oracle.

Create or modify your virtualBox disks to be used in a RAC Cluster

If you want to create a disk to be used in a RAC Cluster, you have to flag your disk to be shared by many virtual machines.

Currently, it is not possible to create a shared disk by using the GUI.So, you must use the VirtualBox command line tool : VBoxManage.

Your disk has be a fixed size disk, and you have to create it first and modify it next to make it shareable.

To create a shared disk, use the createhd command :

VBoxManage createhd --filename /VDisks/asmdisk1.vdi --size 5120 --format VDI --variant Fixed

Next,  use the option –type of the modifyhd command. You can use this command to transform a disk created with the GUI to be shareable.

VBoxManage modifyhd /VDisks/asmdisk1.vdi --type shareable

Now, you can attach your disk to multiple virtual machines and use it as a shared storage device.