Data … as usual

Recently, I had to updrade an EM 12cR1 platform to 12cR2. At the end of this procedure, you have to upgrade agents from 12.1.0.1 to 12.1.0.2. But at this customer site, the em agent located on the OMS was’nt installed correctly so I have to remove and reinstall it in the correct directory. Easy ??? of course, but an agent located on the OMS has some peculiarities that a traditional agent doesn’t have.

First, I needed to remove the agent. To do this, I used the MOS Note: How to Manually Remove an Agent From 12C Cloud Control [ID 1380846.1].

Next, I pushed an agent to the new location by using the “add host” function in the EM Console.

Be careful, deploy agent on the Fully qualified name of the server. If you don’t use the same name, internal targets won’t be discovered. You can find this name by querying the repository:

SQL> select p.value OMS_HOST_NAME from mgmt_oms_parameters p where p.name = 'HOST_NAME';
OMS_HOST_NAME
--------------------------------------------------------------------------------
cloudcontrol12c.localdomain

Here was the start of my problems because an agent installed on the OMS have some plugins you will not find in a traditional agent :

[oracle@cloudcontrol12c ~]$ emctl listplugins agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
oracle.sysman.beacon 12.1.0.2.0 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.beacon.agent.plugin_12.1.0.2.0  <<< Plugin for Oracle Beacon 
oracle.sysman.csa 12.1.0.2.0 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.csa.agent.plugin_12.1.0.2.0        <<< Plugin for Oracle Client System Analyzer
oracle.sysman.emas 12.1.0.3.0 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.emas.agent.plugin_12.1.0.3.0      <<< Plugin for Oracle Fusion Middleware
oracle.sysman.emrep 12.1.0.2.0 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.emrep.agent.plugin_12.1.0.2.0    <<< Plugin for Oracle Management Services & Repository
oracle.sysman.oh 12.1.0.2.0 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.oh.agent.plugin_12.1.0.2.0          <<< Plugin for Oracle Home
oracle.sysman.db 12.1.0.3.0 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.3.0          <<< Plugin for Oracle Database
oracle.em.soav 12.1.0.2.0 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/../../plugins/oracle.em.soav.agent.plugin_12.1.0.2.0              <<< Plugin for Audit Vault

But after installing the agent to its new destination, there’s a lack of plugins :

[oracle@cloudcontrol12c ~]$ emctl listplugins agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
oracle.sysman.oh 12.1.0.2.0  /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.oh.agent.plugin_12.1.0.2.0
Plugins mentioned above in blue can be easily installed in the agent by using the menu : Setup / Extensibility / Plug-ins. After this, plugins installed on the agent are:
[oracle@cloudcontrol12c ~]$ emctl listplugins agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
oracle.sysman.oh    12.1.0.2.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.oh.agent.plugin_12.1.0.2.0
oracle.sysman.db 12.1.0.3.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.3.0
oracle.sysman.emas 12.1.0.3.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.emas.agent.plugin_12.1.0.3.0
oracle.em.soav 12.1.0.2.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.em.soav.agent.plugin_12.1.0.2.0
oracle.sysman.beacon 12.1.0.2.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.beacon.agent.plugin_12.1.0.2.0

Note: I encountered some trouble by deploying all plugins, I recommend to deploy plugins one by one.

That seems better, but there are 2 plugins I couldn’t find in the plug-ins list (CSA and Management Repository and services). But, I can find them in the plugin list in the Self update interface:

The only way to publish them is to used the emcli tool with the plugin sweet name 😉 to deploy it on the agent:

[oracle@cloudcontrol12c ~]$ emcli deploy_plugin_on_agent -plugin="oracle.sysman.csa" -agent_names="cloudcontrol12c.localdomain:3872"
Agent side plug-in deployment is in progress
Use "emcli get_plugin_deployment_status -plugin_id=oracle.sysman.csa" to track plug-in deployment status.

[oracle@cloudcontrol12c ~]$ emcli deploy_plugin_on_agent -plugin="oracle.sysman.emrep" -agent_names="cloudcontrol12c.localdomain:3872"
Agent side plug-in deployment is in progress
Use "emcli get_plugin_deployment_status -plugin_id=oracle.sysman.emrep" to track plug-in deployment status.

Note: same remark, deploy plugins one by one.

After a while, both plugins are deployed on my agent:

[oracle@cloudcontrol12c ~]$ /opt/oracle/agent12c/agent_inst/bin/emctl listplugins agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
oracle.sysman.oh    12.1.0.2.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.oh.agent.plugin_12.1.0.2.0
oracle.sysman.db 12.1.0.3.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.3.0
oracle.sysman.emas 12.1.0.3.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.emas.agent.plugin_12.1.0.3.0
oracle.em.soav 12.1.0.2.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.em.soav.agent.plugin_12.1.0.2.0
oracle.sysman.beacon 12.1.0.2.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.beacon.agent.plugin_12.1.0.2.0
oracle.sysman.csa 12.1.0.2.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.csa.agent.plugin_12.1.0.2.0
oracle.sysman.emrep 12.1.0.2.0 /opt/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.emrep.agent.plugin_12.1.0.2.0

Now, I can discover all the targets. If internal targets (like EM Console, EM Repository etc.) are not automatically discovered. You can add them manually, or to use addinternaltargets command of the agent:

[oracle@cloudcontrol12c ~]$ /opt/oracle/agent12c/agent_inst/bin/emctl config agent addinternaltargets
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
[oracle@cloudcontrol12c ~]$ /opt/oracle/agent12c/agent_inst/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

On an Oracle Enterprise Manager 12c platform, agents represent a main part of the architecture. Sometimes, you need information about its status to diagnose a problem.

In this blog post, I will present commands based on emctl executable and tips you can use to get health and performance indicators.

• The first and well known command is “emctl status agent”. This command shows a summary of the agent status

In this summary, you can see the agent availability status, uploads and XML related files status, OMS address etc.

[oracle@oel agent_inst]$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.2.0
OMS Version : 12.1.0.2.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/product/agent12c/agent_inst
Agent Binaries : /u01/app/oracle/product/agent12c/core/12.1.0.2.0
Agent Process ID : 24845
Parent Process ID : 24799
Agent URL : https://oel.localdomain:3872/emd/main/
Repository URL : https://cloudcontrol12c.localdomain:4900/empbs/upload
Started at : 2012-12-16 13:23:55
Started by user : oracle
Last Reload : 2012-12-16 14:16:11
Last successful upload : 2012-12-16 14:33:04
Last attempted upload : 2012-12-16 14:33:04
Total Megabytes of XML files uploaded so far : 0.62
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 20.09%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2012-12-16 14:36:04
Last successful heartbeat to OMS : 2012-12-16 14:36:04
Next scheduled heartbeat to OMS : 2012-12-16 14:37:04

---------------------------------------------------------------
Agent is Running and Ready

• emctl status agent target TGT_NAME,TGT_TYPE

This command is useful to see status of a specific target (target type is based on EM12c internal name), and metric collected for this specific target.

[oracle@oel agent_inst]$ emctl status agent target orcl,oracle_database
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Target Name : orcl
Target Type : oracle_database
Current severity state
----------------------
Metric Column name Key State Timestamp
--------------------------------------------------------------------------------
DeferredTrans deftrans_count n/a CLEAR Sun Dec 16 13:46:44 CET 2012
DeferredTrans errortrans_count n/a CLEAR Sun Dec 16 13:46:44 CET 2012
Response State n/a CLEAR Sun Dec 16 13:46:26 CET 2012
Response Status n/a CLEAR Sun Dec 16 13:46:26 CET 2012
Temporary_File_Status STATUS 1 CLEAR Sun Dec 16 14:00:44 CET 2012
UserAudit username SYS_oel WARNING Sun Dec 16 14:32:46 CET 2012
UserLocks maxBlockedDBTime TM CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedDBTime TX CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedDBTime UL CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedSess TM CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedSess TX CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedSess UL CLEAR Sun Dec 16 13:52:08 CET 2012
archFull archUsedPercent NOARCHIVELOG CLEAR Sun Dec 16 13:58:01 CET 2012
audit_failed_logins failed_login_count 2012-12-16 12:58:30 GMT CLEAR Sun Dec 16 13:58:30 CET 2012
audit_failed_logins failed_login_count 2012-12-16 13:28:30 GMT CLEAR Sun Dec 16 14:28:30 CET 2012
dbjob_status broken n/a CLEAR Sun Dec 16 14:03:01 CET 2012
dbjob_status failed n/a CLEAR Sun Dec 16 14:03:01 CET 2012
dumpFull dumpAvail background CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpAvail core CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpAvail user CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpUsedPercent background CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpUsedPercent core CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpUsedPercent user CLEAR Sun Dec 16 13:52:51 CET 2012
wait_sess_cls dbtime_waitclass_pct Concurrency WARNING Sun Dec 16 14:21:05 CET 2012
wait_sess_cls dbtime_waitclass_pct Other CLEAR Sun Dec 16 14:02:04 CET 2012

---------------------------------------------------------------
Agent is Running and Ready

• emctl status agent scheduler

This command shows all past and future schedule for the agent. Schedules can be general (e.g. Ping OMS or SchedulerHeartBeat) or target specific (metric collection : oracle_database:orcl:tbspAllocation)

[oracle@oel agent_inst]$ emctl status agent scheduler
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Number of currently running scheduled items :: 0
Number of currently ready scheduled items :: 0
Number of scheduled items :: 134
Schedule Attempts Delay Percentage :: 0.000%
Running entries::
Ready entries::
Scheduled entries::
2012-12-16 14:32:26.901 : oracle_database:orcl:observer_11g
2012-12-16 14:32:27.297 : oracle_database:orcl:Response
2012-12-16 14:32:27.570 : oracle_emd:oel.localdomain:3872:z#TaskRuntimeMap
2012-12-16 14:32:28.542 : SchedulerHeartbeat
2012-12-16 14:32:32.981 : oracle_emd:oel.localdomain:3872:EMDStatusCollection
2012-12-16 14:32:39.979 : oracle_listener:LISTENER_oel.localdomain:Load
2012-12-16 14:32:45.888 : oracle_emd:oel.localdomain:3872:z#AgentStat_AgentHeapUsage
2012-12-16 14:32:46.485 : oracle_database:orcl:service_10i
2012-12-16 14:32:46.547 : oracle_database:orcl:UserAudit
2012-12-16 14:32:47.066 : HTTP Listener
2012-12-16 14:32:52.613 : oracle_emd:oel.localdomain:3872:z#UploadSystemStats
2012-12-16 14:32:59.369 : AgentSystemMonitorTask
2012-12-16 14:33:00.312 : oracle_database:orcl:rac_global_cache_10i
2012-12-16 14:33:01.830 : oracle_database:orcl:dbjob_status
2012-12-16 14:33:03.276 : oracle_emd:oel.localdomain:3872:z#ThreadPoolStatistics
2012-12-16 14:33:03.868 : Ping OMS
2012-12-16 14:33:04.300 : oracle_emd:oel.localdomain:3872:EMDUploadSystemMonitor

.../...

2012-12-16 14:47:03.241 : host:oel.localdomain:NetworkLinux
2012-12-16 14:47:19.120 : osm_instance:+ASM_oel.localdomain:Instance_Volumes_Summary
2012-12-16 14:47:21.555 : host:oel.localdomain:DiskActivityLinux
2012-12-16 14:53:59.251 : CollectionManager.PersistSchedule
2012-12-16 14:58:30.259 : oracle_database:orcl:audit_failed_logins
2012-12-16 15:06:24.048 : oracle_database:orcl:DatabaseVaultRealmConfigurationIssue_collection
2012-12-16 15:10:11.728 : oracle_database:orcl:segment_advisor_count
2012-12-16 15:11:06.604 : oracle_database:orcl:DatabaseVaultCommandRuleViolation_collection
2012-12-16 15:11:50.888 : oracle_database:orcl:DatabaseVaultCommandRuleConfigurationIssue_collection
2012-12-16 15:13:14.610 : oracle_database:orcl:DatabaseVaultRealmViolation_collection
2012-12-16 15:16:51.178 : oracle_listener:LISTENER_1526_oel.localdomain:TNSERRORS
2012-12-16 15:18:17.660 : oracle_database:orcl:DatabaseVaultPolicyChanges_collection
2012-12-16 15:20:28.584 : osm_instance:+ASM_oel.localdomain:ofs_performance_metrics
2012-12-16 15:23:16.786 : oracle_listener:LISTENER_oel.localdomain:TNSERRORS
2012-12-16 15:31:11.126 : MetricReporter.Task
2012-12-16 18:00:46.334 : oracle_database:orcl:haconfig1_collection
2012-12-16 18:02:10.092 : oracle_database:orcl:audit_failed_logins_historical
2012-12-16 21:47:39.678 : oracle_database:orcl:feature_usage_collection_item
2012-12-16 21:50:39.668 : osm_instance:+ASM_oel.localdomain:Disk_Path
2012-12-16 21:51:21.248 : oracle_database:orcl:exadataCollection
2012-12-16 21:53:43.531 : host:oel.localdomain:Swap_Area_StatusLinux
2012-12-16 22:00:41.994 : host:oel.localdomain:HostStorageSupport
2012-12-16 22:01:04.098 : oracle_database:orcl:invalid_objects_rollup
2012-12-16 22:15:12.757 : oracle_database:orcl:ocm_instrumentation
2012-12-16 22:15:27.932 : oracle_database:orcl:problemSegTbsp
2012-12-17 13:23:59.377 : PurgeADR
2012-12-17 13:43:26.423 : host:oel.localdomain:host_storage
2012-12-17 13:48:53.710 : oracle_database:orcl:ha_dg_target_summary
2012-12-17 13:49:58.180 : oracle_listener:LISTENER_1526_oel.localdomain:has_resources
2012-12-17 13:51:04.127 : oracle_database:orcl:oracle_storage
2012-12-17 13:51:44.509 : host:oel.localdomain:oracle_security
2012-12-17 13:52:09.276 : oracle_database:orcl:oracle_dbconfig
2012-12-17 13:52:28.613 : oracle_emd:oel.localdomain:3872:oracle_emd_config
2012-12-17 13:53:48.140 : oracle_home:agent12c1_5_oel:oracle_home_config
2012-12-17 13:56:38.819 : oracle_database:orcl:mgmt_database_listener_config
2012-12-17 13:59:21.165 : oracle_listener:LISTENER_oel.localdomain:has_resources
2012-12-17 13:59:53.042 : host:oel.localdomain:ll_host_config
2012-12-17 14:01:43.076 : osm_instance:+ASM_oel.localdomain:osm_osmconfig
2012-12-17 14:02:28.165 : oracle_home:Ora11g_gridinfrahome2_3_oel:oracle_home_config
2012-12-17 14:03:13.080 : host:oel.localdomain:DiscoverTargets
2012-12-17 14:03:14.031 : oracle_listener:LISTENER_oel.localdomain:mgmt_listener_config
2012-12-17 14:08:21.054 : oracle_database:orcl:has_resources
2012-12-17 14:10:19.581 : osm_instance:+ASM_oel.localdomain:osm_osminstconfig
2012-12-17 14:11:45.115 : oracle_listener:LISTENER_1526_oel.localdomain:mgmt_listener_config
2012-12-17 14:12:37.800 : osm_instance:+ASM_oel.localdomain:has_resources
2012-12-17 14:12:46.933 : oracle_database:orcl:mgmt_sql
2012-12-17 14:16:38.959 : oracle_home:OraDb11g_home2_4_oel:oracle_home_config
2012-12-18 21:42:55.497 : oracle_emd:oel.localdomain:3872:EMDCertExp
2012-12-18 21:48:46.106 : oracle_emd:oel.localdomain:3872:EMDUserLimits
2012-12-18 22:03:43.800 : oracle_emd:oel.localdomain:3872:EMDIdentity
2012-12-18 22:13:08.810 : oracle_database:orcl:tbspAllocation
2012-12-23 13:47:26.765 : osm_instance:+ASM_oel.localdomain:oracle_osm

---------------------------------------------------------------
Agent is Running and Ready

• emctl startschedule agent TGT_NAME,TGT_TYPE

This command is used to schedule a collection on a specified target.

[oracle@oel agent_inst]$ emctl startschedule agent orcl,oracle_database
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Start Schedule Target succeeded

• emctl startschedule agent -type TGT_TYPE

This command is the same than previous one, but it schedules collection for all target of type mentioned by TGT_TYPE (e.g. oracle_database or oracle_emd)

[oracle@oel agent_inst]$ emctl startschedule agent -type oracle_database
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Start Schedule TargetType succeeded

• emctl getmetric agent TGT_NAME,TGT_TYPE,METRIC

This command gets real time metric for a specific target.

[oracle@oel agent_inst]$ emctl getmetric agent oel.localdomain:3872,oracle_emd,EMDStatus
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Timestamp,Transfer time from Agent to OMS (msec),% of Collection Interval spent in Execution,CPU_Used,Java_Used_Pct,Collections scheduled for next hour,Ping response (mSec),UploadQueueSize
2012-12-16 14:22:32,164.328,0.047,0.122,37.2,1276,58.390,21603
2012-12-16 14:23:32,163.176,0.041,0.203,37.3,1275,58.817,27822
2012-12-16 14:24:32,158.634,0.041,0.126,30.9,1275,58.213,-5633
2012-12-16 14:25:32,158.634,0.042,0.084,34.6,1275,58.371,-4844
2012-12-16 14:26:32,158.634,0.042,0.083,38.1,1275,58.524,-4596
2012-12-16 14:27:32,158.634,0.042,0.072,40.4,1275,58.016,-4286
2012-12-16 14:28:32,158.634,0.042,0.113,44.3,1275,58.477,-2151
2012-12-16 14:29:32,158.634,0.042,0.107,32.2,1275,58.667,-128
2012-12-16 14:30:32,157.769,0.042,0.122,39.7,1275,58.806,4493
2012-12-16 14:31:32,157.769,0.038,0.405,42.0,1275,59.221,5611

• emctl status agent cpu

To be able to run this command, you have to set “topMetricReporter=true” in emd.properties file (located in $AGENT_HOME/sysman/config)

[oracle@oel agent_inst]$ emctl status agent cpu -depth 5
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Report generated at: /u01/app/oracle/product/agent12c/agent_inst/sysman/emd/topMetric/1355665421178_metricSummaryReport.html
---------------------------------------------------------------
Agent is Running and Ready

[oracle@oel agent_inst]$ emctl status agent cpu -full_cpu_report
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Report generated at: /u01/app/oracle/product/agent12c/agent_inst/sysman/emd/topMetric/1355665536091_metricSummaryReport.html
---------------------------------------------------------------
Agent is Running and Ready

This command will give you cpu statistics for every task and top metrics. It can be used to print full details (-full_cpu_report) or a top-n values (-depth N). The command generates an html based report :

• emctl status agent dbconnections

[oracle@oel agent_inst]$ emctl status agent dbconnections
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
DBConnection Cache contents as of: Sun Dec 16 14:46:52 CET 2012
------------------------------------------------------------
Connection Cache Name = (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oel.localdomain)(Port=1521))(CONNECT_DATA=(SERVICE_NAME=+ASM)(INSTANCE_NAME=+ASM)(UR=A)))
 :276b5b206e103eb650eec1cec74a7d1
MinCacheLimit = 0
MaxPoolSize = 10
InitialLimit = 0
ConnectionWaitTimeout = 300
ValidationConnection = true
InactivityTimeout = 300
AbandonedTimeout = 0
TimeToLiveTimeout = 300
PropertyCheckInterval = 300
Number of active connections = 0
Number of available connections = 1
Number of abandoned connections = 0
Average connection waittime (mSecs) = 1
Peak connection waittime (mSecs) = 3
Average useage time (mSecs) = 998
Pending requests = 0
Total connections returned = 18
Connection Last Established at 2012-12-16 14:45:22
Connection Last Relinqueshed at 2012-12-16 14:45:22
------------------------------------

• emctl pingOMS

This command shows every connection details established by the agent to targets (ASM or database).

[oracle@oel agent_inst]$ emctl pingOMS
 Oracle Enterprise Manager Cloud Control 12c Release 2
 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
 ---------------------------------------------------------------
 EMD pingOMS completed successfully

This commands tries to ping the OMS and checks the connectivity to the Oracle Management Server.

• emctl listplugins agent

This command shows plugins deployed on the agent: name, version and installation directory.

[oracle@oel agent_inst]$ emctl listplugins agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
oracle.sysman.oh 12.1.0.2.0 /u01/app/oracle/product/agent12c/plugins/oracle.sysman.oh.agent.plugin_12.1.0.2.0
oracle.sysman.db 12.1.0.2.0 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.2.0

• emctl dumpstate agent

[oracle@oel agent_inst]$ emctl dumpstate agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Dumpstate succeeded
[oracle@oel agent_inst]$ ls -ltr /u01/app/oracle/product/agent12c/agent_inst/sysman/emd/dumps/
total 8696
-rw-r----- 1 oracle oinstall 4440752 Dec 16 13:51 SystemDump_1355662309788.xml
-rw-r----- 1 oracle oinstall 4433309 Dec 16 14:32 SystemDump_1355664774563.xml

This command dumps agent state. The dump is a complex XML file generated in $AGENT_HOME/sysman/emd/dumps

• EM Agent browser

All information mentioned above can be obtained by using the EM Agent Browser. These information are available with a web browser through a specific URL. This URL is build on the agent URL (see emctl status agent) by adding the “browser” keyword between emd and main :

Agent URL         : https://oel.localdomain:3872/emd/main

Agent Browser URL :  https://oel.localdomain:3872/emd/browser/main

Access to this browser is authenticated by agent unix hostname owner (e.g. oracle in my case).

Below two screenshot of what you can see. Using this browser can be very useful if some metrics are not collected, or if your agent is in a pending status.

If like me, you are addicted to command line, this post will help you to realize useful operation in EM Cloud Control 12c using emcli tool rather than GUI.

You can use emcli when you need to implement batch scripts on your Oracle Management Server (OMS).

• Login to EM

$ emcli login -username=sysman
Enter password :
Login successful

OR

$ emcli login -username=sysman -password=XXXXXX
Login successful

• Synchronize emcli with the OMS

$ emcli sync
Synchronized successfully

• Logout EM

$ emcli logout
Logout successful

• List all the targets already promoted

$ emcli get_targets
Status  Status           Target Type           Target Name
 ID
-9      n/a              group                 OFM
1       Up               host                  xxxdb-prod3.mydomain.com
1       Up               host                  xxxdb-prod2.mydomain.com
1       Up               host                  xxxlgc-prod2.mydomain.com
1       Up               host                  xxxdb-prod1.mydomain.com
1       Up               j2ee_application      /EMGC_GCDomain/GCDomain/EMGC_OMS1/e
                                               mgc
1       Up               j2ee_application      /EMGC_GCDomain/GCDomain/EMGC_OMS1/O
                                               CMRepeater
1       Up               j2ee_application      /EMGC_GCDomain/GCDomain/EMGC_OMS1/e
                                               mpbs
-9      n/a              metadata_repository   /EMGC_GCDomain/GCDomain/EMGC_ADMINS
                                               ERVER/mds-sysman_mds
-9      n/a              metadata_repository   /EMGC_GCDomain/GCDomain/EMGC_ADMINS
                                               ERVER/mds-owsm
1       Up               oracle_apache         /EMGC_GCDomain/instance1/ohs1
1       Up               oracle_apm            /EMGC_GCDomain/GCDomain/EMGC_OMS1/o
                                               racle.security.apm(11.1.1.3.0)
1       Up               oracle_beacon         EM Management Beacon
1       Up               oracle_em_service     EM Console Service
1       Up               oracle_em_service     EM Jobs Service
1       Up               oracle_emd            xxxdb-prod1.mydomain.com:1830
1       Up               oracle_emd            xxxdb-prod3.mydomain.com:1830
1       Up               oracle_emd            xxxlgc-prod2.mydomain.com:3872
1       Up               oracle_emd            xxxdb-prod2.mydomain.com:1830
1       Up               oracle_emrep          Management Services and Repository
-9      n/a              oracle_home           agent12g1_9_xxxlgc-prod2
-9      n/a              oracle_home           WebLogicServer10_3_5_0_0_xxxlgc-pro
                                               d2
-9      n/a              oracle_home           oms12g1_3_xxxlgc-prod2
-9      n/a              oracle_home           webtier12g1_20_xxxlgc-prod2
-9      n/a              oracle_ias_farm       EMGC_GCDomain
1       Up               oracle_oms            xxxlgc-prod2.mydomain.com:4889_Managem
                                               ent_Service
1       Up               oracle_oms_console    xxxlgc-prod2.mydomain.com:4889_Managem
                                               ent_Service_CONSOLE
1       Up               oracle_oms_pbs        xxxlgc-prod2.mydomain.com:4889_Managem
                                               ent_Service_PBS
-9      n/a              weblogic_domain       /EMGC_GCDomain/GCDomain
1       Up               weblogic_j2eeserver   /EMGC_GCDomain/GCDomain/EMGC_OMS1
1       Up               weblogic_j2eeserver   /EMGC_GCDomain/GCDomain/EMGC_ADMINSERVER

• Delete a specific target

$ emcli delete_target -name="/wlstest_wlstest/wlstest/myServer_jvm" -type="java_virtual_machine"

• Delete an agent and its targets

$ emcli delete_target -name="xxxlgc-prod2.mydomain.com:3872" -type="oracle_emd" -delete_monitored_targets

• Follow a plugin deployment (on the OMS / on an agent)

-- Oracle Database plugin
$ emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db
-- Oracle Fusion Middleware plugin
$ emcli get_plugin_deployment_status -plugin_id=oracle.sysman.emas
-- My Oracle Support plugin
$ emcli get_plugin_deployment_status -plugin_id=oracle.sysman.mos

• Import an update (for example: a plugin update) into the software library.

$ emcli import_update -file="p14018177_112000_Generic.zip" -omslocal

• Deploy a plugin on the OMS

$ emcli deploy_plugin_on_server -plugin=oracle.sysman.db -sys_password=XXXXX

• Deploy a plugin on EM agent(s)

$ emcli deploy_plugin_on_agent -plugin="oracle.sysman.db"  -agent_names="xxxdb-prod1.mydomain.com:3872;xxxdb-prod2.mydomain.com:3872"

• List available agents in the library

$ emcli get_supported_platforms
Getting list of platforms ...
Check the logs at /home/oracle/agent.log
About to access self-update code path to retrieve the platforms list..
Getting Platforms list ...
-----------------------------------------------
Version = 12.1.0.1.0
 Platform = Linux x86-64
-----------------------------------------------
Version = 12.1.0.1.0
 Platform = Linux x86
-----------------------------------------------
Version = 12.1.0.1.0
 Platform = IBM AIX on POWER Systems (64-bit)
-----------------------------------------------
Version = 12.1.0.1.0
 Platform = Microsoft Windows (32-bit)
-----------------------------------------------
Version = 12.1.0.1.0
 Platform = Microsoft Windows (64-bit)
-----------------------------------------------
Platforms list displayed successfully

• Download an agent from the library (used for agentDeploy.sh script method)

$ emcli get_agentimage -destination=/home/oracle -platform="Microsoft Windows x64 (64-bit)" -version="12.1.0.1.0"
Platform:Microsoft Windows x64 (64-bit)
Destination:/home/oracle
 === Partition Detail ===
Space free : 11 GB
Space required : 1 GB
Check the logs at /home/oracle/get_agentimage_2012-06-11_15-18-09-PM.log
Setting property ORACLE_HOME to:/u02/Middleware/oms
calling pulloneoffs with arguments:/u02/Middleware/oms/u02/softLibrary/32/C0C9BF40C6A36668E0439D33010AE58512.1.0.1.0windows_x64
Check this logs for more information: /u02/Middleware/oms/sysman/prov/agentpush/logs

• Set monitoring credentials for a specific target (example given for an Oracle database instance)

$ emcli set_credential -target_type=oracle_database -target_name="prod1" -credential_set=DBCredsMonitoring -user=sysman -column="Role:SYSDBA;UserName:sys;password:XXXXX" -monitoring