Wednesday, February 8, 2017

Monitoring Mechanism for Hyperion EPM V11.1.2.4

General

I was happy when we successfully implemented BACKUP mechanism for Hyperion EPM V11.1.2.4 system for the first client which was Logistic company from France. The further discussion popped up few more queries like "How many FTE's for monitoring and maintaining the Hyperion System?", "What kind of automation for monitoring tasks?"bla bla.
This expedited my thought process and finally we proposed and implemented adeptly a proper Monitoring mechanism for this client. And yes, the process followed for our successive clients from various domains - specifically - Oil and Gas company from Dubai, Courier Services Leader from Norway and Donuts chain from USA.
As the famous astronomer and author Carl Sagan has said:
Absence of evidence is not evidence of absence.

So might be the case that right now, we do not have an evidence that there is no issue with our Hyperion EPM V11.1.2.4 but there are some hidden issues present and we are ignoring due to lack of evidence.

So, here I am, trying to outline the few monitoring and maintenance activities to properly maintain and support Hyperion EPM V11.1.2.4 from an IT prospective.

EPM V11.1.2.4 Backup Steps

Hyperion Services Monitoring

All the services responsible for Hyperion EPM V11.1.2.4 application should be monitored regularly.
If any service goes down, that will generate an alert for Hyperion Admin group via email.

Hyperion Services Monitoring
Server
All Hyperion servers
WebSvr1, WebSvr2, AppSvr1, AppSvr2, EssSvr
Services to be monitored
Hyperion Services on all above servers
Action 
Critical alert call – tickets for normal - if service goes down

Services to be monitored:
WebSvr1 and WebSvr2
Name                                                                                                                          Startup
Oracle Hyperion CALC Manager Java Web Application                                            Manual
Oracle Hyperion Financial Management – Web Tier                                                   Manual
Oracle Hyperion Financial Reporting Java Web Application                                      Manual
Oracle Hyperion Foundation Services Managed Server                                              Manual
Oracle Hyperion Reporting and analysis Framework Java Web Application              Manual
Oracle Hyperion Reporting and analysis Framework                                                  Manual
Oracle Process Manager (ohsinstance 1649849633)                                                   Manual

AppSvr1 and AppSvr2
Name                                                                                                                          Startup
Oracle Hyperion FDM Enterprise edition Java Web Application                               Manual
Oracle Hyperion Financial Management – Java Server                                              Manual
Oracle Hyperion Planning Java Web Application                                                       Manual
Oracle Hyperion Provider Services Java Web Application                                         Manual
Oracle Hyperion RMI Registry                                                                                    Manual
Oracle Hyperion Strategic Finance Java Web Application                                          Manual
Oracle Hyperion Strategic Finance Server                                                                   Manual

EssSvr
Name                                                                                                                          Startup
Oracle Process Manager                                                                                             Manual
Oracle Hyperion Essbase Studio Server                                                                     Manual
Oracle Hyperion Administration Services Java Web Application                              Manual

Hyperion Logs Monitoring

Hyperion log analysis report

Log Analysis Report
Server

All Hyperion servers
WebSvr1, WebSvr2, AppSvr1, AppSvr2, EssSvr
Frequency
Daily
No of Backups before purging
30
Action
Send email to Hyperion admin with report as attachment only if there is error in report

Log Analysis utility is provided with V11.1.2.3 onwards.
But this is standalone utility provided in patch - - and can be used with previous versions as well.
PFA introductory demo of this utility.

Command = loganalysis.bat -system -d e:\Oracle\Middleware -o total_report
We should include this command to run once every day and send report to Hyperion if there is any error in report.

Periodic Cleanup of logs

Periodic cleanup of logs
Server

All Hyperion servers
WebSvr1, WebSvr2, AppSvr1, AppSvr2, EssSvr

Frequency
Weekly
Action
Send email to Hyperion admin with report of logs modified after deletion

It’s important to keep the system clean. The Hyperion logs can get huge if left alone. Large log files can seriously impact performance as well. Periodic log cleanup is needed
Check logging.xml setting for each module of hyperion on ALL servers and modify to delete files after the size goes beyond 1GB

Sample logging.xml
<?xml version="1.0" encoding="UTF-8"?>
-<logging_configuration>
-<log_handlers>
-<log_handler class="oracle.core.ojdl.logging.ODLHandlerFactory" name="hfminterop-handler">
<property name="path" value="${logging.folder}/InteropJava.log"/>
<property name="maxFileSize" value="50000000"/>
<property name="maxLogSize" value="1000000000"/>
<property name="deleteFiles" value="true"/>
<property name="useSourceClassAndMethod" value="false"/>
<property name="useRealThreadId" value="true"/>
</log_handler>
</log_handlers>
-<loggers>
-<logger name="" useParentHandlers="false" level="INCIDENT_ERROR:1">
<handler name="hfminterop-handler"/>
</logger>
</loggers>
</logging_configuration>

Periodic Cleanup of application artifacts

Periodic cleanup of logs
Server

All Hyperion servers
WebSvr1, WebSvr2, AppSvr1, AppSvr2, EssSvr
Frequency
Weekly
Action
Send email to Hyperion admin with report of logs modified after deletion

It’s important to keep the system clean. Over the years, we accumulate stale an unused reports, applications, data, etc. It’s especially important to ensure you remove and de-provision any users that are no longer valid. I recommend a quarterly cleaning. Stale security, applications, and data can impact performance and make upgrades messier.

Patches/Upgrades

Patches are released all the time. It important to keep updated on what’s out there. Just make sure you do not just simply install patches simply because they are out there. You should apply only those that exactly match issues and the products/configuration you have.
 
Patches/Upgrades
Server

All Hyperion servers
WebSvr1, WebSvr2, AppSvr1, AppSvr2, EssSvr
Frequency
Semester
No of Backups before purging
Complete
Action
Ticket for Admin for analyzing new Hyperion patches and applying in system


Nightly jobs

Nightly jobs
Server

All Hyperion servers
WebSvr1, WebSvr2, AppSvr1, AppSvr2, EssSvr
Frequency
Daily
Action
Send email to Hyperion admin with report as attachment
In almost every case there are nightly data loads, calculations, backups, consolidations, etc that run and these jobs need to run on time. We must be alerted if jobs fail. Monitoring, automating, and resolving issues with nightly jobs can be time consuming depending on the complexity and frequency.

Manual Health Monitoring of application

Manual Health Monitoring
Server

All Hyperion servers
WebSvr1, WebSvr2, AppSvr1, AppSvr2, EssSvr
Frequency
Daily
Action
Send email to Hyperion admin with report as attachment

An IT staff should be prepared to periodically check the health of the system by fully logging in, checking logs, services, process, memory utilization and other system resources, etc. Of course this can be done programmatically using standard industry monitoring tools as well.

Periodic IT-Finance Meetings

IT-Finance Meetings
Server
NA
Frequency
Monthly
Action
Send MOM to all participants with action plan
I recommend a monthly meeting between the business users of the system and IT. It’s important to talk about general issues, service levels, root cause analysis of historical problems, performance, and what’s on the horizon for changes coming up.
Seem like a lot? It is. Is it a full time job? Depends. Either way, day-to-day maintenance activities like these can be a killer on today’s IT administrators that are supporting many enterprise systems at a time, not to mention implementing new projects constantly.
One option to reduce cost and risk but to ensure these activities are always being done is to leverage our Managed Applications group.  It is an affordable and flexible way to meet whatever service needs you have.

No comments:

Post a Comment