Oracle … as usual

Oracle by Laurent Leturgez

Oracle 12c, VMWare Fusion and the perl binary’s segmentation fault

Recently I have installed a new Virtual Machine in VM Ware Fusion 7 Pro under Oracle Enterprise Linux 6.6 for running Oracle 12c Grid Infrastructure and database (12.1.0.2). Mac OS X is a 10.9.5 (Mavericks). As usual for my tests environment, I deploy the oracle-rdbms-server-12cR1-preinstall rpm package to be sure everything will be correctly configured on my system. During the installation process, there was a first error during link edition of rman (“error invoking target irman ioracle of makefile /u01/app/oracle/product/12.1.0/grid/rdbms/lib/ins_rdbms.mk”). This problem can be fixed by copying the libjavavm12.a library located in $ORACLE_HOME/javavm/jdk/jdk7/lib to $ORACLE_HOME/lib directory, and then retry the link edition by clicking on “Retry” button. So, if you don’t use Grid Infrastructure here or if you install only the database kernel … you won’t have any problem. Maybe, you will encountered the problems detailed above if you create a mutitenant (or single tenant) database (because it needs to run catcon.pl script … but we will see why later). In my case, I was deploying Grid Infrastructure and at the end of the installation process, you have to run root.sh script to configure the CRS stack. At this step, the root.sh script fails with an error on  roothas.pl script and a cute “Segmentation fault (core dumped)” error:


Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
/u01/app/oracle/product/12.1.0/grid/crs/config/rootconfig.sh: line 131: 4562 Segmentation fault (core dumped) $ROOTSCRIPT $ROOTSCRIPT_ARGS
The command '/u01/app/oracle/product/12.1.0/grid/perl/bin/perl -I/u01/app/oracle/product/12.1.0/grid/perl/lib -I/u01/app/oracle/product/12.1.0/grid/crs/install /u01/app/oracle/product/12.1.0/grid/crs/install/roothas.pl ' execution failed

After analyzing the error, I get the cause of my problem which is located on the perl binary delivered with Oracle. If I run a simple “perl -v”, I got the error.


[oracle@oel6 bin]$ pwd
/u01/app/oracle/product/12.1.0/grid/perl/bin
[oracle@oel6 bin]$ ./perl -v
Segmentation fault (core dumped)

If we go deeper, gdb shows us an error located on PerlIO functions and specifically on PerlIO_default_layers function:


[oracle@oel6 bin]$ gdb perl
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /u01/app/oracle/product/12.1.0/grid/perl/bin/perl...(no debugging symbols found)...done.
(gdb) r
Starting program: /u01/app/oracle/product/12.1.0/grid/perl/bin/perl
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000531399 in PerlIO_default_layers ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.7.x86_64 libgcc-4.4.7-11.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64
(gdb) bt
#0 0x0000000000531399 in PerlIO_default_layers ()
#1 0x000000000053070e in PerlIO_resolve_layers ()
#2 0x000000000053050f in PerlIO_openn ()
#3 0x0000000000530206 in PerlIO_fdopen ()
#4 0x0000000000530198 in PerlIO_stdstreams ()
#5 0x0000000000530150 in Perl_PerlIO_stdin ()
#6 0x0000000000435f09 in S_parse_body ()
#7 0x00000000004343bb in perl_parse ()
#8 0x000000000041cf13 in main ()

IMHO, it’s a specific problem on WMWare fusion because on a VM hosted on Virtualbox there’s no problem, and more :

  • If I copy the working perl binary from the virtualbox VM to the VMWare VM … execution failed on VMWare
  • If I copy the failing perl binary from the VMWare VM to the virtualbox VM … it works fine

At the time I write this post, I have asked to a friend of mine if he can reproduced the problem on a VM hosted on an ESX server, I will update this post as soon as I will get his results. I made a lot of different tests (Downgrading VMWare fusion to Fusion 6 Pro, Installing OEL 6.4, OEL 7.0, Redhat 6.5 and 7, creating my vmdk’s on another disk, changing scsi driver etc … always the same problem). I read on many blogs, people have faced the same problem and have symlinked the failing perl to the system perl binary which runs fine. Doing this can produce different issues because of the INC directory which is not the same, you can encountered some errors due to version compatibility between modules in $OH/perl/lib directory and the perl binary. Indeed Oracle 12cR1 uses a perl v5.14.1 and system perl binary is 5.10 on OEL/RHEL 6, and 5.16 on OEL/RHEL7. I workarounded the problem by recreating the perl binary from sources, but you need to be careful of the $OH/perl/lib directory, because it contains all the perl modules needed by Oracle. See above, the different steps to compile perl binary and replace it in $ORACLE_HOME:


export ORACLE_HOME=/u01/app/oracle/product/12.1.0/grid
cd ~
rm -rf perl
mkdir perl
cd perl/
curl -O http://www.cpan.org/src/5.0/perl-5.14.1.tar.gz
tar -xvzf perl-5.14.1.tar.gz

cd $ORACLE_HOME
mv perl/ perl.OLD
mkdir perl
cd /home/oracle/perl/perl-5.14.1
./Configure -des -Dprefix=$ORACLE_HOME/perl -Doptimize=-O3 -Dusethreads -Duseithreads -Duserelocatableinc ; make clean ; make ; make install
cd $ORACLE_HOME/perl
rm -rf lib/ man/
cp -r ../perl.OLD/lib/ .
cp -r ../perl.OLD/man/ .
cp ../perl.OLD/bin/dbilogstrip bin/
cp ../perl.OLD/bin/dbiprof bin/
cp ../perl.OLD/bin/dbiproxy bin/
cp ../perl.OLD/bin/ora_explain bin/

Once these steps are done and you have a running perl binary, you can safely run your root.sh script, create multitenant databases, and all the stuff that needs $ORACLE_HOME/bin/perl.

UPDATE: I have just downloaded the last release of Oracle Virtualbox (5.0) on my Macbook and the same problems occurs. They can be resolved by using the same method described in this post.

UPDATE2 : I would like to thank Deiby Gomez for his trick. Indeed, in my original post I forgot to keep some perl binaries that could be useful 😉 : dbilogstrip, dbiprof, dbiproxy and ora_explain. The script above has been modified.

UPDATE3:  Danny Bryant from Enkitec has worked closely with Oracle VM and virtual box team and they discovered the origin of the bug. This one should be fixed in a next release. But, if you use Virtualbox 5.0, Danny published a workaround, you can read it following this link: http://dbaontap.com/2016/01/13/vbox5/

 

Advertisements

22 responses to “Oracle 12c, VMWare Fusion and the perl binary’s segmentation fault

  1. Pingback: Virtual Box 5 Broke my 12c RAC | Stojan's Oracle Blog

  2. mpm_dba (@mpm_dba) August 11, 2015 at 1:05 PM

    Hi, thanks for this post, it ended several days of frustration trying to install 12.1.0.2 on Linux 6 in Virtualbox 5.

  3. Pingback: Oracle DB12c on OL7 with Virtual Box 5 installation issues - My Experiences with Oracle Technology

  4. Pingback: Can’t Compile GI 12.1.0.2 and Segmentation Fault » Peasland Database Blog

  5. Vic September 7, 2015 at 12:09 PM

    It was the same under Mac OS X 10.10.5 Yosemite+Parallels+Oracle Linux 6 and 7+Oracle 12.1.0.2
    thank you

  6. Jazo September 24, 2015 at 4:28 PM

    Thank you for your post. Very good and helpful.
    VMware Fusion 8
    Mac OS X 10.10.5 Yosemite
    Oracle Linux 6.6

  7. Brad November 5, 2015 at 4:32 PM

    Hello, I encountered this issue yesterday with VirtualBox 5.0.8 on OSX Yosemite with guest OS Oracle Linux 6u7. In my case, shutting down the VM, changing the paravirtualization interface from “Default” to “Legacy” resolved the issue. I’m not sure if this is a VirtualBox bug or an Oracle DB embedded perl bug.

  8. Neil Johnson November 7, 2015 at 7:53 PM

    Got to say thanks. Recently upgraded to a Mac and this post was so helpful when applying PSU under VirtualBox.
    Thanks

  9. Pingback: ins_rdbms.mk and Segmentation fault errors with Oracle 12c - Oracle - Oracle - Toad World

  10. benson yang January 15, 2016 at 1:38 PM

    thanks very much

  11. benson yang January 15, 2016 at 1:39 PM

    thanks , it is very helpful .

  12. Oliver Nelson April 15, 2016 at 6:06 PM

    Can be fixed in VMWare Fusion by adding this to your VMX:

    cpuid.4.4.eax = “00000000000000000000000000000000”
    cpuid.4.4.ebx = “00000000000000000000000000000000”
    cpuid.4.4.ecx = “00000000000000000000000000000000”
    cpuid.4.4.edx = “00000000000000000000000000000000”

  13. Murth May 18, 2016 at 3:10 PM

    How do you do the last code in a windows environment?

  14. zephyr59fr June 14, 2016 at 2:21 PM

    Thank you so much for this code that saved me a lot of time !
    But I have to say something, it’s incredible :
    I prepared this environment in a VM that failed each time, and I applied your patch and it worked fine.
    Today I do it again, exactly the same VM ( a copy ) on another computer ( same architecture, a few différences ) and it’s fine !!!! No need to patch
    I did just a difference, I changed the network from Bridged to NAT using VMware
    I am really bored about Oracle products and those wastes of time….

    • Laurent June 14, 2016 at 2:53 PM

      If it’s the same VM which have been copied after you patched it .. IMO it’s normal.
      Because, the problem is located on the perl binary delivered during the installation process. If you copied the VM after you patched it, you copied the patch and the workaround as well …

  15. Pingback: Installation d’Oracle E-Business Suite 12.2.5 sous deux noeuds Oracle Linux 7.2: installing the datatiers! | b2train.com

  16. Pingback: Vagrant for you RAC test environment | Future Veterans

  17. Pingback: Installation d’Oracle E-Business Suite 12.2.5 sous deux noeuds Oracle Linux 7.2: installing the datatiers! |

  18. Kranthi September 7, 2016 at 1:28 PM

    Thank you so much. It helped a lot. Just did blindly and worked like charm..

  19. Pingback: Install Cloud Control 13.2 on Mac and Virtual Box | hias222

  20. Pingback: Perl and new Intel CPUs with 4 level cache – Linux & Unix & Database

  21. siripala66 March 19, 2017 at 3:45 PM

    Encountered the same 12c R2 while creating DB using DBCA and the above sol is not going to help
    Any work around is greatly appriciated…..

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: