Data … as usual

On an Oracle Enterprise Manager 12c platform, agents represent a main part of the architecture. Sometimes, you need information about its status to diagnose a problem.

In this blog post, I will present commands based on emctl executable and tips you can use to get health and performance indicators.

• The first and well known command is “emctl status agent”. This command shows a summary of the agent status

In this summary, you can see the agent availability status, uploads and XML related files status, OMS address etc.

[oracle@oel agent_inst]$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.2.0
OMS Version : 12.1.0.2.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/product/agent12c/agent_inst
Agent Binaries : /u01/app/oracle/product/agent12c/core/12.1.0.2.0
Agent Process ID : 24845
Parent Process ID : 24799
Agent URL : https://oel.localdomain:3872/emd/main/
Repository URL : https://cloudcontrol12c.localdomain:4900/empbs/upload
Started at : 2012-12-16 13:23:55
Started by user : oracle
Last Reload : 2012-12-16 14:16:11
Last successful upload : 2012-12-16 14:33:04
Last attempted upload : 2012-12-16 14:33:04
Total Megabytes of XML files uploaded so far : 0.62
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 20.09%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2012-12-16 14:36:04
Last successful heartbeat to OMS : 2012-12-16 14:36:04
Next scheduled heartbeat to OMS : 2012-12-16 14:37:04

---------------------------------------------------------------
Agent is Running and Ready

• emctl status agent target TGT_NAME,TGT_TYPE

This command is useful to see status of a specific target (target type is based on EM12c internal name), and metric collected for this specific target.

[oracle@oel agent_inst]$ emctl status agent target orcl,oracle_database
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Target Name : orcl
Target Type : oracle_database
Current severity state
----------------------
Metric Column name Key State Timestamp
--------------------------------------------------------------------------------
DeferredTrans deftrans_count n/a CLEAR Sun Dec 16 13:46:44 CET 2012
DeferredTrans errortrans_count n/a CLEAR Sun Dec 16 13:46:44 CET 2012
Response State n/a CLEAR Sun Dec 16 13:46:26 CET 2012
Response Status n/a CLEAR Sun Dec 16 13:46:26 CET 2012
Temporary_File_Status STATUS 1 CLEAR Sun Dec 16 14:00:44 CET 2012
UserAudit username SYS_oel WARNING Sun Dec 16 14:32:46 CET 2012
UserLocks maxBlockedDBTime TM CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedDBTime TX CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedDBTime UL CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedSess TM CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedSess TX CLEAR Sun Dec 16 13:52:08 CET 2012
UserLocks maxBlockedSess UL CLEAR Sun Dec 16 13:52:08 CET 2012
archFull archUsedPercent NOARCHIVELOG CLEAR Sun Dec 16 13:58:01 CET 2012
audit_failed_logins failed_login_count 2012-12-16 12:58:30 GMT CLEAR Sun Dec 16 13:58:30 CET 2012
audit_failed_logins failed_login_count 2012-12-16 13:28:30 GMT CLEAR Sun Dec 16 14:28:30 CET 2012
dbjob_status broken n/a CLEAR Sun Dec 16 14:03:01 CET 2012
dbjob_status failed n/a CLEAR Sun Dec 16 14:03:01 CET 2012
dumpFull dumpAvail background CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpAvail core CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpAvail user CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpUsedPercent background CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpUsedPercent core CLEAR Sun Dec 16 13:52:51 CET 2012
dumpFull dumpUsedPercent user CLEAR Sun Dec 16 13:52:51 CET 2012
wait_sess_cls dbtime_waitclass_pct Concurrency WARNING Sun Dec 16 14:21:05 CET 2012
wait_sess_cls dbtime_waitclass_pct Other CLEAR Sun Dec 16 14:02:04 CET 2012

---------------------------------------------------------------
Agent is Running and Ready

• emctl status agent scheduler

This command shows all past and future schedule for the agent. Schedules can be general (e.g. Ping OMS or SchedulerHeartBeat) or target specific (metric collection : oracle_database:orcl:tbspAllocation)

[oracle@oel agent_inst]$ emctl status agent scheduler
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Number of currently running scheduled items :: 0
Number of currently ready scheduled items :: 0
Number of scheduled items :: 134
Schedule Attempts Delay Percentage :: 0.000%
Running entries::
Ready entries::
Scheduled entries::
2012-12-16 14:32:26.901 : oracle_database:orcl:observer_11g
2012-12-16 14:32:27.297 : oracle_database:orcl:Response
2012-12-16 14:32:27.570 : oracle_emd:oel.localdomain:3872:z#TaskRuntimeMap
2012-12-16 14:32:28.542 : SchedulerHeartbeat
2012-12-16 14:32:32.981 : oracle_emd:oel.localdomain:3872:EMDStatusCollection
2012-12-16 14:32:39.979 : oracle_listener:LISTENER_oel.localdomain:Load
2012-12-16 14:32:45.888 : oracle_emd:oel.localdomain:3872:z#AgentStat_AgentHeapUsage
2012-12-16 14:32:46.485 : oracle_database:orcl:service_10i
2012-12-16 14:32:46.547 : oracle_database:orcl:UserAudit
2012-12-16 14:32:47.066 : HTTP Listener
2012-12-16 14:32:52.613 : oracle_emd:oel.localdomain:3872:z#UploadSystemStats
2012-12-16 14:32:59.369 : AgentSystemMonitorTask
2012-12-16 14:33:00.312 : oracle_database:orcl:rac_global_cache_10i
2012-12-16 14:33:01.830 : oracle_database:orcl:dbjob_status
2012-12-16 14:33:03.276 : oracle_emd:oel.localdomain:3872:z#ThreadPoolStatistics
2012-12-16 14:33:03.868 : Ping OMS
2012-12-16 14:33:04.300 : oracle_emd:oel.localdomain:3872:EMDUploadSystemMonitor

.../...

2012-12-16 14:47:03.241 : host:oel.localdomain:NetworkLinux
2012-12-16 14:47:19.120 : osm_instance:+ASM_oel.localdomain:Instance_Volumes_Summary
2012-12-16 14:47:21.555 : host:oel.localdomain:DiskActivityLinux
2012-12-16 14:53:59.251 : CollectionManager.PersistSchedule
2012-12-16 14:58:30.259 : oracle_database:orcl:audit_failed_logins
2012-12-16 15:06:24.048 : oracle_database:orcl:DatabaseVaultRealmConfigurationIssue_collection
2012-12-16 15:10:11.728 : oracle_database:orcl:segment_advisor_count
2012-12-16 15:11:06.604 : oracle_database:orcl:DatabaseVaultCommandRuleViolation_collection
2012-12-16 15:11:50.888 : oracle_database:orcl:DatabaseVaultCommandRuleConfigurationIssue_collection
2012-12-16 15:13:14.610 : oracle_database:orcl:DatabaseVaultRealmViolation_collection
2012-12-16 15:16:51.178 : oracle_listener:LISTENER_1526_oel.localdomain:TNSERRORS
2012-12-16 15:18:17.660 : oracle_database:orcl:DatabaseVaultPolicyChanges_collection
2012-12-16 15:20:28.584 : osm_instance:+ASM_oel.localdomain:ofs_performance_metrics
2012-12-16 15:23:16.786 : oracle_listener:LISTENER_oel.localdomain:TNSERRORS
2012-12-16 15:31:11.126 : MetricReporter.Task
2012-12-16 18:00:46.334 : oracle_database:orcl:haconfig1_collection
2012-12-16 18:02:10.092 : oracle_database:orcl:audit_failed_logins_historical
2012-12-16 21:47:39.678 : oracle_database:orcl:feature_usage_collection_item
2012-12-16 21:50:39.668 : osm_instance:+ASM_oel.localdomain:Disk_Path
2012-12-16 21:51:21.248 : oracle_database:orcl:exadataCollection
2012-12-16 21:53:43.531 : host:oel.localdomain:Swap_Area_StatusLinux
2012-12-16 22:00:41.994 : host:oel.localdomain:HostStorageSupport
2012-12-16 22:01:04.098 : oracle_database:orcl:invalid_objects_rollup
2012-12-16 22:15:12.757 : oracle_database:orcl:ocm_instrumentation
2012-12-16 22:15:27.932 : oracle_database:orcl:problemSegTbsp
2012-12-17 13:23:59.377 : PurgeADR
2012-12-17 13:43:26.423 : host:oel.localdomain:host_storage
2012-12-17 13:48:53.710 : oracle_database:orcl:ha_dg_target_summary
2012-12-17 13:49:58.180 : oracle_listener:LISTENER_1526_oel.localdomain:has_resources
2012-12-17 13:51:04.127 : oracle_database:orcl:oracle_storage
2012-12-17 13:51:44.509 : host:oel.localdomain:oracle_security
2012-12-17 13:52:09.276 : oracle_database:orcl:oracle_dbconfig
2012-12-17 13:52:28.613 : oracle_emd:oel.localdomain:3872:oracle_emd_config
2012-12-17 13:53:48.140 : oracle_home:agent12c1_5_oel:oracle_home_config
2012-12-17 13:56:38.819 : oracle_database:orcl:mgmt_database_listener_config
2012-12-17 13:59:21.165 : oracle_listener:LISTENER_oel.localdomain:has_resources
2012-12-17 13:59:53.042 : host:oel.localdomain:ll_host_config
2012-12-17 14:01:43.076 : osm_instance:+ASM_oel.localdomain:osm_osmconfig
2012-12-17 14:02:28.165 : oracle_home:Ora11g_gridinfrahome2_3_oel:oracle_home_config
2012-12-17 14:03:13.080 : host:oel.localdomain:DiscoverTargets
2012-12-17 14:03:14.031 : oracle_listener:LISTENER_oel.localdomain:mgmt_listener_config
2012-12-17 14:08:21.054 : oracle_database:orcl:has_resources
2012-12-17 14:10:19.581 : osm_instance:+ASM_oel.localdomain:osm_osminstconfig
2012-12-17 14:11:45.115 : oracle_listener:LISTENER_1526_oel.localdomain:mgmt_listener_config
2012-12-17 14:12:37.800 : osm_instance:+ASM_oel.localdomain:has_resources
2012-12-17 14:12:46.933 : oracle_database:orcl:mgmt_sql
2012-12-17 14:16:38.959 : oracle_home:OraDb11g_home2_4_oel:oracle_home_config
2012-12-18 21:42:55.497 : oracle_emd:oel.localdomain:3872:EMDCertExp
2012-12-18 21:48:46.106 : oracle_emd:oel.localdomain:3872:EMDUserLimits
2012-12-18 22:03:43.800 : oracle_emd:oel.localdomain:3872:EMDIdentity
2012-12-18 22:13:08.810 : oracle_database:orcl:tbspAllocation
2012-12-23 13:47:26.765 : osm_instance:+ASM_oel.localdomain:oracle_osm

---------------------------------------------------------------
Agent is Running and Ready

• emctl startschedule agent TGT_NAME,TGT_TYPE

This command is used to schedule a collection on a specified target.

[oracle@oel agent_inst]$ emctl startschedule agent orcl,oracle_database
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Start Schedule Target succeeded

• emctl startschedule agent -type TGT_TYPE

This command is the same than previous one, but it schedules collection for all target of type mentioned by TGT_TYPE (e.g. oracle_database or oracle_emd)

[oracle@oel agent_inst]$ emctl startschedule agent -type oracle_database
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Start Schedule TargetType succeeded

• emctl getmetric agent TGT_NAME,TGT_TYPE,METRIC

This command gets real time metric for a specific target.

[oracle@oel agent_inst]$ emctl getmetric agent oel.localdomain:3872,oracle_emd,EMDStatus
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Timestamp,Transfer time from Agent to OMS (msec),% of Collection Interval spent in Execution,CPU_Used,Java_Used_Pct,Collections scheduled for next hour,Ping response (mSec),UploadQueueSize
2012-12-16 14:22:32,164.328,0.047,0.122,37.2,1276,58.390,21603
2012-12-16 14:23:32,163.176,0.041,0.203,37.3,1275,58.817,27822
2012-12-16 14:24:32,158.634,0.041,0.126,30.9,1275,58.213,-5633
2012-12-16 14:25:32,158.634,0.042,0.084,34.6,1275,58.371,-4844
2012-12-16 14:26:32,158.634,0.042,0.083,38.1,1275,58.524,-4596
2012-12-16 14:27:32,158.634,0.042,0.072,40.4,1275,58.016,-4286
2012-12-16 14:28:32,158.634,0.042,0.113,44.3,1275,58.477,-2151
2012-12-16 14:29:32,158.634,0.042,0.107,32.2,1275,58.667,-128
2012-12-16 14:30:32,157.769,0.042,0.122,39.7,1275,58.806,4493
2012-12-16 14:31:32,157.769,0.038,0.405,42.0,1275,59.221,5611

• emctl status agent cpu

To be able to run this command, you have to set “topMetricReporter=true” in emd.properties file (located in $AGENT_HOME/sysman/config)

[oracle@oel agent_inst]$ emctl status agent cpu -depth 5
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Report generated at: /u01/app/oracle/product/agent12c/agent_inst/sysman/emd/topMetric/1355665421178_metricSummaryReport.html
---------------------------------------------------------------
Agent is Running and Ready

[oracle@oel agent_inst]$ emctl status agent cpu -full_cpu_report
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Report generated at: /u01/app/oracle/product/agent12c/agent_inst/sysman/emd/topMetric/1355665536091_metricSummaryReport.html
---------------------------------------------------------------
Agent is Running and Ready

This command will give you cpu statistics for every task and top metrics. It can be used to print full details (-full_cpu_report) or a top-n values (-depth N). The command generates an html based report :

• emctl status agent dbconnections

[oracle@oel agent_inst]$ emctl status agent dbconnections
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
DBConnection Cache contents as of: Sun Dec 16 14:46:52 CET 2012
------------------------------------------------------------
Connection Cache Name = (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oel.localdomain)(Port=1521))(CONNECT_DATA=(SERVICE_NAME=+ASM)(INSTANCE_NAME=+ASM)(UR=A)))
 :276b5b206e103eb650eec1cec74a7d1
MinCacheLimit = 0
MaxPoolSize = 10
InitialLimit = 0
ConnectionWaitTimeout = 300
ValidationConnection = true
InactivityTimeout = 300
AbandonedTimeout = 0
TimeToLiveTimeout = 300
PropertyCheckInterval = 300
Number of active connections = 0
Number of available connections = 1
Number of abandoned connections = 0
Average connection waittime (mSecs) = 1
Peak connection waittime (mSecs) = 3
Average useage time (mSecs) = 998
Pending requests = 0
Total connections returned = 18
Connection Last Established at 2012-12-16 14:45:22
Connection Last Relinqueshed at 2012-12-16 14:45:22
------------------------------------

• emctl pingOMS

This command shows every connection details established by the agent to targets (ASM or database).

[oracle@oel agent_inst]$ emctl pingOMS
 Oracle Enterprise Manager Cloud Control 12c Release 2
 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
 ---------------------------------------------------------------
 EMD pingOMS completed successfully

This commands tries to ping the OMS and checks the connectivity to the Oracle Management Server.

• emctl listplugins agent

This command shows plugins deployed on the agent: name, version and installation directory.

[oracle@oel agent_inst]$ emctl listplugins agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
oracle.sysman.oh 12.1.0.2.0 /u01/app/oracle/product/agent12c/plugins/oracle.sysman.oh.agent.plugin_12.1.0.2.0
oracle.sysman.db 12.1.0.2.0 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.2.0

• emctl dumpstate agent

[oracle@oel agent_inst]$ emctl dumpstate agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Dumpstate succeeded
[oracle@oel agent_inst]$ ls -ltr /u01/app/oracle/product/agent12c/agent_inst/sysman/emd/dumps/
total 8696
-rw-r----- 1 oracle oinstall 4440752 Dec 16 13:51 SystemDump_1355662309788.xml
-rw-r----- 1 oracle oinstall 4433309 Dec 16 14:32 SystemDump_1355664774563.xml

This command dumps agent state. The dump is a complex XML file generated in $AGENT_HOME/sysman/emd/dumps

• EM Agent browser

All information mentioned above can be obtained by using the EM Agent Browser. These information are available with a web browser through a specific URL. This URL is build on the agent URL (see emctl status agent) by adding the “browser” keyword between emd and main :

Agent URL         : https://oel.localdomain:3872/emd/main

Agent Browser URL :  https://oel.localdomain:3872/emd/browser/main

Access to this browser is authenticated by agent unix hostname owner (e.g. oracle in my case).

Below two screenshot of what you can see. Using this browser can be very useful if some metrics are not collected, or if your agent is in a pending status.