Data … as usual

All things about data by Laurent Leturgez

Category Archives: Cloud

Dealing with Oracle Cloud Infrastructure and Python

Oracle provides various SDK to create resources in the OCI.

Recently, I played with the Python SDK for OCI. In this blog post, I will show you the basics to create a simple bucket in the Object Storage part of OCI, and simply put a file on this Bucket.

OCI Client configuration

First, you will need to install the python OCI package. The best for that is to create a python virtual environment, activate it, and install all the packages you need inside.


mbp:python_venv $ python -m virtualenv oci
Using base prefix '/Users/leturgezl/miniconda3/envs/general'
New python executable in /Users/leturgezl/python_venv/oci/bin/python
Installing setuptools, pip, wheel...
done.

mbp:python_venv $ source oci/bin/activate


(oci) mbp:python_venv leturgezl$ pip install oci numpy pandas

Now that packages are installed, we have to configure the client to access OCI.

To do that, we need many things :

  • User OCID: this can be found in the User’s Page in OCI
  • Tenancy OCI: this can be found in the Tenancy’s page in OCI
  • Your OCI region
  • A private key file, its public key, and the related fingerprint.

The keys have been generated like this (I used a key without passphrase)


# Private key generation

$ mkdir ~/.oci
$ openssl genrsa -out ~/.oci/oci_api_key.pem 2048
$ chmod go-rwx ~/.oci/oci_api_key.pem

# Public key generation
$ openssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pem

# Fingerprint generation
$ openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem | openssl md5 -c

Once you did that, you will have to configure you user in OCI to add it the public key, the fingerprint given in the interface must match with the previous command:

OCI User 1

 

It’s important to keep your private key “private” (don’t send it to another people, or leave it without any protection on your laptop).

Now, your local environment is configured, we will need a dictionary structure in our python script to use the SDK.

This dictionary can be build manually and embedded in the code, then you will have to fill the required fields (Key file location is the private key location):

config = {
    "user": "ocid1.user.oc1..aaaaaaaamcel7xygkvhe....aaaaaaaaaaaaaaaaaaaaa" ,
    "key_file": "~/.oci/oci_api_key.pem",
    "fingerprint": "35:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa",
    "tenancy": "ocid1.tenancy.oc1..aaaaaaaahgagkf7xygkvhe....aaaaaaaaaaaaaaaaaaaaa",
    "region": "eu-frankfurt-1"
}

Or, you can configure a local “config” file in your ~/.oci/ directory and then load it in the code with the given python code below:


$ cat ~/.oci/config
[DEFAULT]
user=ocid1.user.oc1..aaaaaaaamcel7xygkvhe....aaaaaaaaaaaaaaaaaaaaa
fingerprint=35:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa
key_file=~/.oci/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..aaaaaaaahgagkf7xygkvhe....aaaaaaaaaaaaaaaaaaaaa
region=eu-frankfurt-1

Note : you can embed more than one user profile in this file. The only one required is the DEFAULT profile.


>>> import oci
>>> import pandas as pd
>>> config=oci.config.from_file()
>>> df=pd.DataFrame.from_dict(config, orient='index')
>>> df
                                                                       0
log_requests                                                       False
additional_user_agent
pass_phrase                                                         None
user                             ocid1.user.oc1..aaaaaaaamcel7xygkvhe...
fingerprint              35:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa
key_file                                          ~/.oci/oci_api_key.pem
tenancy                     ocid1.tenancy.oc1..aaaaaaaahgagkf7xygkvhe...
region                                                    eu-frankfurt-1

You can read the oci config file and select another profile by using this:


config = oci.config.from_file(profile_name="laurent")

Or use another file by using this parameter


config = oci.config.from_file(file_location="~/OCI_config.uat")

You can see there are more parameters in the dictionary, you can find the details by reading this: https://docs.cloud.oracle.com/iaas/Content/API/Concepts/sdkconfig.htm.

Creating an Object Storage bucket in OCI

Now our client is well configured to access OCI through a user and his keys.

it’s really easy to create a bucket. We have to request an ObjectStorageClient object and use it to create the bucket:

import oci
from oci.object_storage.models import CreateBucketDetails

compartment_id = config["tenancy"]
object_storage = oci.object_storage.ObjectStorageClient(config)

namespace = object_storage.get_namespace().data
bkt_name = "python-bucket"
object_name = "python_file"

print("Creating a new bucket {!r} in compartment {!r}".format(bkt_name, compartment_id))
request = CreateBucketDetails()
request.compartment_id = compartment_id
request.name = bkt_name
bucket = object_storage.create_bucket(namespace, request)

This will produce that kind of output:

Creating a new bucket 'python-bucket' in compartment 'ocid1.tenancy.oc1..aaaaaaaahgagkf7xygkvhe...'

And in the OCI web interface, our bucket appeared :

OCI Bucket 1

Put a file into the Object storage bucket

Now we have a bucket created in our compartment, it’s easy to put a file on it (I’ll put a binary file which is a PNG file).

To do that, the below code will be enough (considering variables have been initiated by previous code parts … see above)


with open("images/myimage.png", mode='rb') as file:
my_data = file.read()

obj = object_storage.put_object(
namespace,
bkt_name,
object_name,
my_data)

In the OCI console, inside the previously created bucket, the file has been created and is available:

OCI Bucket 2

As you can see, deploying resources on the OCI is easy and you can deploy your full infrastructure with a bunch of code.

Next investigations will be made soon, specially to deploy virtual machines, storage and databases of course.

That’s it for today 🙂

Advertisements

Oracle 12.2, new release … new editions or just a cloudy feature ?

Recently I have created an Oracle 12.2 database in the Oracle Cloud (Extreme Performance). I was able to test some of the new features.

And usually, when I test a new release of Oracle, I have a look into the ins_rdbms.mk file to see if there are some new options to link the Oracle kernel, and in this release I found some interesting stuff.

As I said before, my instance was in the Extreme Performance, it was normal that, when connected, I got this banner:

Connected to:
Oracle Database 12c EE Extreme Perf Release 12.2.0.1.0 - 64bit Production

And when I had a look to the V$INSTANCE view, there’s a column EDITION. This column is already here in 12.1 and is documented.

If you have a closer look to the documentation (12.1), you can see different editions:

  • CORE EE: CORE Enterprise Edition
  • CORE SE: CORE Standard Edition
  • EE: Enterprise Edition
  • PO: Personal Edition
  • SE: Standard Edition
  • XE: Express Edition

Ok for EE, PO, SE and XE … they are well known edition, but what about those “Core” Editions (Standard and Enterprise).

Note: on 1st December, 12.2 documentation mention only Core EE, EE, PO and XE (Might be a doc bug) 

I had a look to both (12.1 and 12.2) ins_rdbms.mk files (located in $ORACLE_HOME/lib folder). For On premises installations (11.2 & 12.1), no trace of these new editions, but they are available on Oracle Cloud Platform

  • Oracle 12.1.0.2
$ grep -i edi /u01/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/ins_rdbms.mk
edition_corestandard:
        $(SILENT)$(ECHO) "Deploying Oracle Database Core Standard Edition"
edition_coreenterprise:
        $(SILENT)$(ECHO) "Deploying Oracle Database Core Enterprise Edition"
edition_standard:
        $(SILENT)$(ECHO) "Deploying Oracle Database Standard Edition"
edition_enterprise:
        $(SILENT)$(ECHO) "Deploying Oracle Database Enterprise Edition"
edition_highperf:
        $(SILENT)$(ECHO) "Deploying Oracle Database Enterprise Edition High Performance"
edition_extremeperf:
        $(SILENT)$(ECHO) "Deploying Oracle Database Enterprise Edition Extreme Performance"

  • Oracle 12.2.0.1
$ grep -i edi /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/ins_rdbms.mk
edition_corestandard:
        $(SILENT)$(ECHO) "Deploying Oracle Database Core Standard Edition"
edition_coreenterprise:
        $(SILENT)$(ECHO) "Deploying Oracle Database Core Enterprise Edition"
edition_standard:
        $(SILENT)$(ECHO) "Deploying Oracle Database Standard Edition"
edition_enterprise:
        $(SILENT)$(ECHO) "Deploying Oracle Database Enterprise Edition"
edition_highperf:
        $(SILENT)$(ECHO) "Deploying Oracle Database Enterprise Edition High Performance"
edition_extremeperf:
        $(SILENT)$(ECHO) "Deploying Oracle Database Enterprise Edition Extreme Performance"

ohoohhh ;), let’s try to activate those rules and relink the kernel

$ make -f ins_rdbms.mk edition_coreenterprise ioracle
Deploying Oracle Database Core Enterprise Edition
mv -f /u01/app/oracle/product/12.2.0/dbhome_1/lib/libvsn12.a /u01/app/oracle/product/12.2.0/dbhome_1/lib/libvsn12_backup.a.dbl
cp /u01/app/oracle/product/12.2.0/dbhome_1/lib/libvsn12_cee.a.dbl /u01/app/oracle/product/12.2.0/dbhome_1/lib/libvsn12.a
chmod 755 /u01/app/oracle/product/12.2.0/dbhome_1/bin

 - Linking Oracle
rm -f /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/oracle
/u01/app/oracle/product/12.2.0/dbhome_1/bin/orald  -o /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/oracle -m64 -z noexecstack -Wl,--disable-new-dtags -L/u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/ -L/u01/app/oracle/product/12.2.0/dbhome_1/lib/ -L/u01/app/oracle/product/12.2.0/dbhome_1/lib/stubs/   
-Wl,-E /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/opimai.o /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/ssoraed.o /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv12 -Wl,--no-whole-archive /u01/app/oracle/product/12.2.0/dbhome_1/lib/nautab.o 
/u01/app/oracle/product/12.2.0/dbhome_1/lib/naeet.o /u01/app/oracle/product/12.2.0/dbhome_1/lib/naect.o /u01/app/oracle/product/12.2.0/dbhome_1/lib/naedhs.o /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/config.o  
-ldmext -lserver12 -lodm12 -lofs -lcell12 -lnnet12 -lskgxp12 -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lclient12  -lvsn12 -lcommon12 
-lgeneric12 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap12" ; fi` -lskjcx12 -lslax12 -lpls12  -lrt -lplp12 
-ldmext -lserver12 -lclient12  -lvsn12 -lcommon12 -lgeneric12 `if [ -f /u01/app/oracle/product/12.2.0/dbhome_1/lib/libavserver12.a ] ; then echo "-lavserver12" ; else echo "-lavstub12"; fi`
 `if [ -f /u01/app/oracle/product/12.2.0/dbhome_1/lib/libavclient12.a ] ; then echo "-lavclient12" ; fi` -lknlopt -lslax12 -lpls12  -lrt -lplp12 -ljavavm12 -lserver12  -lwwg  `cat /u01/app/oracle/product/12.2.0/dbhome_1/lib/ldflags` 
   -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lngsmshd12 -lnro12 `cat /u01/app/oracle/product/12.2.0/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lngsmshd12 -lnnzst12 -lzt12 -lztkg12 -lmm -lsnls12 -lnls12  -lcore12
 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lztkg12 `cat /u01/app/oracle/product/12.2.0/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lngsmshd12 -lnro12
 `cat /u01/app/oracle/product/12.2.0/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lngsmshd12 -lnnzst12 -lzt12 -lztkg12   -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12
 -lsnls12 -lnls12 -lcore12 -lnls12 `if /usr/bin/ar tv /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo12 -lserver12"; fi` 
-L/u01/app/oracle/product/12.2.0/dbhome_1/ctx/lib/ -lctxc12 -lctx12 -lzx12 -lgx12 -lctx12 -lzx12 -lgx12 -lordimt12 -lclsra12 -ldbcfg12 -lhasgen12 -lskgxn2 -lnnzst12 -lzt12 -lxml12 -lgeneric12 -locr12 -locrb12 -locrutl12 -lhasgen12
 -lskgxn2 -lnnzst12 -lzt12 -lxml12 -lgeneric12  -lgeneric12 -lorazip -loraz -llzopro5 -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls12 -lnls12  -lcore12 
-lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lsnls12 -lunls12  -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 
-lcore12 -lnls12 -lasmclnt12 -lcommon12 -lcore12  -laio -lons  -lfthread12   `cat /u01/app/oracle/product/12.2.0/dbhome_1/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/12.2.0/dbhome_1/lib -lm   
 `cat /u01/app/oracle/product/12.2.0/dbhome_1/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/12.2.0/dbhome_1/lib `test -x /usr/bin/hugeedit -a -r /usr/lib64/libhugetlbfs.so && 
test -r /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/shugetlbfs.o && echo -Wl,-zcommon-page-size=2097152 -Wl,-zmax-page-size=2097152 -lhugetlbfs`
test ! -f /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle || (\
           mv -f /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracleO &&\
           chmod 600 /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracleO )
mv /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/oracle /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle
chmod 6751 /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle

That seems to work, now let’s start the instance and let’s connect:

$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Mon Nov 28 16:41:28 2016

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition - Core 12.2.0.1.0 - 64bit Production

New banner … but are there any new features ?

If we compare EE Core edition and EE Extreme perf, it seems there are some difference in the field of enabled options:

  • EE Core Edition (12.2)
SQL> select parameter,value from v$option where value!='TRUE' order by 1;

PARAMETER                      VALUE
------------------------------ ------------------------------
ASM Proxy Instance             FALSE *
Active Data Guard              FALSE
Adaptive Execution Plans       FALSE
Advanced Analytics             FALSE
Automatic Storage Management   FALSE *
Cache Fusion Lock Accelerator  FALSE
Change Data Capture            FALSE
DICOM                          FALSE
Data Mining                    FALSE
Exadata Discovery              FALSE
Global Data Services           FALSE
I/O Server                     FALSE * 
Management Database            FALSE *
OLAP                           FALSE
Oracle Data Guard              FALSE
Oracle Database Vault          FALSE *
Oracle Label Security          FALSE *
Partitioning                   FALSE
Real Application Clusters      FALSE *
Real Application Security      FALSE
Real Application Testing       FALSE
Spatial                        FALSE
Unified Auditing               FALSE *
  • EE Extreme Perf Edition and High Perf Edition (12.2)
SQL> select parameter,value from v$option where value!='TRUE' order by 1;

PARAMETER                      VALUE
------------------------------ ------------------------------
ASM Proxy Instance             FALSE
Automatic Storage Management   FALSE
I/O Server                     FALSE
Management Database            FALSE
Oracle Database Vault          FALSE
Oracle Label Security          FALSE
Real Application Clusters      FALSE
Unified Auditing               FALSE

And if we have a look to “traditional” EE that we can choose from the Cloud Interface … (12.2)

SQL> select parameter,value from v$option where value!='TRUE' order by 1;

PARAMETER                      VALUE
------------------------------ ----------------------------------------
ASM Proxy Instance             FALSE
Automatic Storage Management   FALSE
I/O Server                     FALSE
Management Database            FALSE
Oracle Database Vault          FALSE
Oracle Label Security          FALSE
Real Application Clusters      FALSE
Unified Auditing               FALSE

OK, so Enterprise Core Edition and Enterprise Edition look differents in terms of available options. And like Franck Pachot pointed it out, the Core EE is like a traditional EE but with all paid options disabled and other EE features disabled (DataGuard for example).

 

On the field of licensing, these Core editions seem to be unavailable. If you buy Oracle License, on-premises or in the cloud, you only have the choice between SE, EE and EE High Perf and EE Extreme perf (in the cloud for HP and XP), no Core Editions.

If we go back to ins_rdbms.mk file, it’s possible to link these editions only in Oracle Cloud Service for 12.1 and 12.2, but Oracle Enterprise Edition 12.2 is impossible to link (because of missing libraries), but if Core Edition is a core based license model, Core Edition and Actual Enterprise Edition are licensed per core not per socket, no difference on this field:

$ make -f ins_rdbms.mk edition_enterprise ioracle
Deploying Oracle Database Enterprise Edition
mv -f /u01/app/oracle/product/12.2.0/dbhome_1/lib/libvsn12.a /u01/app/oracle/product/12.2.0/dbhome_1/lib/libvsn12_backup.a.dbl
cp /u01/app/oracle/product/12.2.0/dbhome_1/lib/libvsn12_ee.a.dbl /u01/app/oracle/product/12.2.0/dbhome_1/lib/libvsn12.a
cp: cannot stat `/u01/app/oracle/product/12.2.0/dbhome_1/lib/libvsn12_ee.a.dbl': No such file or directory
make: *** [edition_enterprise] Error 1

Oracle Standard Edition and Standard Core Edition are both able to be linked.

As a conclusion, the “Core” Edition could be :

  • A new licensing model for Cloud and On Premises platform and will be released later. As a consequence, a new licensing model will appear for Standard Edition, not per Socket, but per Core like EE (the Socket model would be abandoned ?)
  • A new licensing model for Cloud platform … only
  • A bug ?

The future will tell us 😉