When evaluating Application Performance Management, what aspect do you think is the most important to look for?

Speed to get data into the platform is one of our most important metrics. We NEED to know what is going on right now, not 3-4minutes ago.

1.Ability to Corelate 

2. Machine learning/AI based thresholds 

3. Ease of configuration ( in bulk)

Hi,
Full disclosure I am the COO at Correlsense.
2 years ago I wrote a post just about that - "Why APM project fails" - I think it can guide you through the process of the most important aspects of APM tools.

Take a look - feel free to leave a comment:
www.correlsense.com
Elad Katav

There are so many monitoring systems in every company that provide alerts. This is what's called - alert fatigue phenomena. Moreover, most alerts are handled manually which means it takes a long time and cost a lot of money to resolve an event. I think companies should evaluate the remediation part of monitoring systems... What is there to instantly and automatically resolve problems that come up instead of just alerting.

Tracing ability like record level, latency and capabilities to inform good predictions in advance, history storage, pricing, support etc..,

Full stack end-to-end monitoring including frontend and backend server profiling, real user monitoring, synthetic monitoring and root cause deep dive analysis. Ease of use and intuitive UX. 

In order to evaluate/benchmark APM solutions, We can based on the 5 dimension provided by Gartner:
1. End-user experience monitoring: the capture of data about how end-to-end application
availability, latency, execution correctness and quality appeared to the end user
2. Runtime application architecture discovery, modeling and display: the discovery of the
various software and hardware components involved in application execution, and the array of
possible paths across which those components could communicate that, together, enable that
involvement
3. User-defined transaction profiling: the tracing of events as they occur among the components
or objects as they move across the paths discovered in the second dimension, generated in
response to a user's attempt to cause the application to execute what the user regards as a
logical unit of work
4. Component deep-dive monitoring in an application context: the fine-grained monitoring of
resources consumed by and events occurring within the components discovered in the second
dimension
5. Analytics: the marshalling of a variety of techniques (including behavior learning engines,
complex-event processing (CEP) platforms, log analysis and multidimensional database
analysis) to discover meaningful and actionable patterns in the typically large datasets
generated by the first four dimensions of APM

In other side, we tried to benchmark internally some APM solutions based on the following evaluation groups:
Monitoring capabilities
Technologies and framework support
Central PMDB (Performance Management DataBase)
Integration
Service modeling and monitoring
Performance analysis and diagnostics
Alerts/event Management
Dashboard and Visualization
Setup and configuration
User experience
and we got interresting results

Most vendors have similar transaction monitoring capabilities so I look at the End user experience monitoring features to differentiate. Not only RUM (mobile and web) but also Active Monitoring through synthetics.

I Check about a Customer Experience Tool, I think the organization have to improve your client experience and if you don't have a tool to view the Improve Points this is so hard to do!!

Visibility into the application transaction and all the magic behind the curtain

Monitoring end user experience (including impact of external partners). It's the price of admission

Being able to see every transaction and tracking of the performance of what the end user sees.

Know and meet de NFR's. Find correlations and have superb SLA management and reporting.

Hi As per exp.. customer always look how much faster application its open, how many users are accessed in a faster way...how are then got an errors....application page load....end user request load on application ....how the server resource utilized the application....is there any more resource are need to run the application

In my opinion, an APM solution should demonstrate the following characteristics:
Ease of use. The solution should be easy to use from installation, configuration and maintenance/support.
Fulfill the business need. There are so many solutions/vendors out in the marketplace that claim to have the RIGHT solution for you. After carefully evaluating some of the solutions, you find that they do not satisfy your business need and promises to have the additional functionality in their future roadmap. Why wait?
Integration. An APM solution should have the ability integrate with various other tools/systems without disrupting your current environment.
Cost effective. In most cases, "you get what you pay for". Free isn't always the best solution for your business need, as well as paying a substantial price tag for a solution and still does not give you the value you need.

Easy deployment and integration. E2E monitoring on a component level. Automatic CI discovery and mapping. Support of ITIL processes - incident, configuration, service level management...

Application Performance Management is a shared concern. APM tools should be used, easily, by all people involved in the build and run process, to share a common view of which part of the application/system is used and how it impacts the whole performance of the service according to the user/customer point of view.

My scope for a monitoring system is the capability to monitor:
- Servers
- Switches
- Firewalls
- Workstations
- FC switches
- Storage (SAN, TapeLibrary,...)

These are the main components where the monitoring system has to get as much data as possible out of the systems.
Here is the HA of the monitoring system very important.

We have tested CACTI and Nagios and SCOM but we are the most happy with OpManager of ManageEngine. because the correlation with the other components is so easy and logic we prefer and recommend the solutions of ManageEngine.

In my view following aspects should be consider:

- Server Monitoring capabilities
- Ease for user i.e. easily configurable and end-user experience
- Alerts and events management
- Dashboards available so you can visualize the stats
- Analyse performance and diagnose problem
- Must be under budget

We mostly recommend Nagios and Cacti as tools to our client.

These posts hit my main points on the nose.
Bob_J " 1. End-to-end transaction visibility 2. Scalability 3. Low/no over-head 4. Ease of use "

reviewer142791 - "Being able to see EVERY transaction. No sampling or just high water marks. If you don't have information on all of your data, you can't effectively benchmark and trend performance over time."

I would also look for transaction topology mapping. I picture paints a 1000 words in expressing urgency.

When you want to evaluate an APM Solution in my option the most important aspect is
1) What are my organization needs and if that solution partialy complies to the needs.
If we know for what we need the APM Solution we go for the other things:
2) Monitor End-to-end transactions
3) Integration with monitoring tools
Usualy after implementing the APM Solution the client wants an unified solution that agregates the data from the APM and from the monitoring tool.
4) Synthetic transactions
5) Cost-benefit
Every organisation has it's own needs. Sometimes the client doesn't choose the solution they really need because of the price, and in time they discover that their choise is not fullfilling it's purpose and they need to buy other solution.
6) Event correlation
It is ideal to correlate the events from a server with the events from the APM. The correlation helps the administrators to solve the issue ASAP when you have all the data.
7) Easy to use
If the solution is easy to use, it gives the administrator the time to develope new rules and it give him the ability to learn the solution faster.
8) Ability to analyse application code for flaws
If you see errors in the APM from your applications, usually the application owner is asking what it will be the cost for troubleshooting and for error remediation. If the APM is able to provide the flaws, than the cost of repair is much lower than expected.

1. The APM solution getting implemented should not have an additional overhead on the system performance in terms of server resources. ( Very Important aspect)
2. The APM solution should be compatible with the technology /software you are trying to use.

1. End-to-end transaction visibility
2. Scalability
3. Low/no over-head
4. Ease of use

Well, the most important: the solution covers your application stack and your ecosystem, and this happens at the implementation and in the roadmap of your company. Then you should decide having in count: easy of use, data integrated from multiple sources, transactional view and in-depth root cause analysis available. All platforms of this type performs well and are based in very similar concepts. After a pilot or PoC you will have more light in this field.

Ease of implementation, scalability, breadth of monitoring from L3-L7 of OSI model and low overhead for agents.

Easy to implement, use; Automatic rootcause, proactive and helps in optimal use of applications

Number of systems monitored and the amount of traffic they transverse through you LAN and WAN

Dashboard information to show summary, but easy drill-down access to more detailed information; easy configuration and user adaptation.

end user experience, network visibility, ease of use

1. Low Overhead
2. Ability to use it in production, test and development
3. No Sampling or just turning it on when there is something suspicious going on -this is to late to find the root cause
4. Ability to follow exactly end to end transactions from the click/touch to the database
5. Real User Monitoring for Web, Mobile Apps and Rich Clients
6. Ability to give APM data to different audience in a convenient way. eg. CEO can view high level data on a tablet, Developer can deep dive on stored data a long a bug report, operators can have their personalized view of the system on the ops screens

It depends on multiple factors, some of them are Real time monitoring, scalability, easy of use and licensing cost.

1. Presentation of key performance data
2. Compactness and minimal dependencies
3. Customization
4. Platform independent

One suite which would cover
1. End-to-end business/audit transaction flow irrespective of technology stack/Integrated components,
2. NFRs/Infrastructure/capacity metrics and their management as well
3. lightweight,
4. Easy configurable/manageable.
5. Plugin availability for integration for customization.
6. Of Course budgetary/cost saving in long run.

By default APM tool/suite should provide evidence/root cause of performance bottleneck and notify reaching threshold of configured KPIs.

1. How the tool affects application load is most important. An APM tool is useless if it slows down the production environment.
2. Ease of installation.
3. Ability to drill down into the application stack for root cause analysis.
4. Alerts on performance and errors.
5. Ability to customize the data stream.

1> Monitoring application performance with the help of APM introscope.
2> Being able to see EVERY transaction. No sampling or just high water marks. If you don't have information on all of your data, you can't effectively benchmark and trend performance over time

End user metrics including Network latency, load time and application Launches.

Being able to see EVERY transaction. No sampling or just high water marks. If you don't have information on all of your data, you can't effectively benchmark and trend performance over time.

Nothing can be more important than end user visibility.
If it wasn't for the end user, what metrics would have any value since we talk about Application Performance Management?
The judges of the performance is the end users - period. It can never be the tools!
In a APM solution, there needs to be a direct correlation of what a user does to what resources that particular action demands from the infrastructure. Of course the time spent in those resources is also valuable.

Check resources availability.

To know NFR's

For one, the APM tool needs to be stable and be able to notify me when it is unable to complete the configured monitoring. After that, it is important to be able to monitor log message occurrences for the software that my company produces.

You should list till what level the monitoring is required.

Starting from
1. Application layer
2. Database layer
3. Code Level
4. Network & Infrastructure Level

This will help you select the right bundle of the tools to achieve your requirements.

As I believe you cannot manage what you cannot measure, I give the most importance to data quality. No sampling, no gaps, just the real thing!

I am not an actual user of these products. I am a consultant that offers comparative research support for clients with vested, potential investment or consumer interest in them or at times, for the developers themselves.

Measure every users transactions and trace them through to back end for immediate root cause analysis.

Knowing business needs and capabilities, and map them to the best tool.
Getting the full picture, quickly find problem and getting pointed the right way

Manageability, stability, and ease of use

Query / Transaction response for expected load conditions.

Two factors affect application performance management. First user interaction (workstation, data flow, access, response rates) and second backend systems (replication, db updates).

When it comes to performance, I monitor how long it took to refresh the screen for the application.
Scalability, low impact of the performance etc

End User Experience along with its operation time,slow operations and the application performance.
For the web based applications,I think we should look the TCP and HTTP errors which are basic parameters.
For SSL Based applications,it is TCP and SSL errors.

Clear full stack tracing... starting at the end user all the way to a minor external service call somewhere in a microservice sometime in the entire transaction. ..

Price
Ease of Installation
Customization
Reporting

Ease of use, Predefined customizable reporting capability, high visibility, high capability of raw data and reporting import export

How quickly you can ID bottlenecks while providing the performance that the customer demands.

Knowing what you wil do with results. Match them with technical and functional requirements and think about what information you would need if you need to troubleshoot the applictaion chain. Then look for a tool that supports this.

cost per port, scalabity for Tier1 operations, ease of deployment and use

low impact in the overall performance and ease to identify bottleneck

1. Identify problems and pinpoint root-cause.
2. Understand your application or website’s readiness before go-live.
3. Shorten application release cycles.
4. Understand and optimize your end-users’ journey.
5. Understand outside forces affecting your application and/or website.
6. Simplify deployment and management of your APM platform.
as www.researchinaction.de

scalability and flexibility come first, in order for an Organization to -proactively- runs its business and -effectively- adapt to this fast growing business environment. Then comes all the other obvious APM capabilities and benefits customers expect, some listed above.

technologies (big data, apps, mobile, cloud, etc) have been quickly evolving... an organization can only run its IT Operations and be more proactive if it relies on APM that are built on 3year old (or less) technology stack, so the APM is flexible, scalable and can adapt to any business and technological changes, in an ever increasing big data, mobile, cloud and highly integrated environment.

most other APMs that are built on 5-15year old technology stack may have more "analytics" pre-configured, but it takes weeks, months to add more capabilities to them and make them scalable and flexible to have these legacy systems keep up the pace with this ever/quickly changing world.

take a look at:
Germain APM (Application Performance Management)
Germain CRT (Code Review Tool)
on www.germainsoftware.com

Enable Complex Distributed Application Architecture Discovery.
Enable Business Transaction and Internal Call Discovery, bottleneck and latency discovery.
Enable Business and IT Customisable Dashboard build.
Enable Alerting based on Pattern / Baselining.

A Way To Be Pro-Active whether used in QA Environnement, or Efficient on Production issues analysis.

A Must Have In Today, RealTime, Distributed, Multi-tiers Applications

Visibility into the end user experience.

24/7 transparency along the whole process chain with user experience as the key performance indicator

Check resources availability.

TIme to value and ease of use

Ease of use, price, and scalability

Easy of diagnoses and analysis. Getting to the actual root cause quickly.

Price, scalability and easy of use

Ease of Integration ... There are almost 7 different tolls currently in play and each group is in his own world...We want to add capability not add double the cost

User response with the ability to highlight any component that is negatively impacting this response from the user through the network back to the server, database server, and storage.

Transaction detail and End-User experience metrics.

Ability to discover flow transaction, capture all business transaction. Easy to use and to deploy. Also have customize dashboard, as my user comes from different role (management, it operation, application owner, business development, etc)

Ease of use

1) Capabilities
2) Synthetic transactions
3) Open Source approach
4) Integration
5) Cost-benefit
6) End-user Support
7) Dashboard
8) Event correlation
9) Easy, user-friendly configuration and setup
10) No invasive monitoring

Monitoring application performance with the help of APM
Visibility to see All transaction. Not a sampling or just high water marks.
The ability to see what the End user is experiencing is most important.
Problem solving not finger pointing

Self learning from the topology.

End user visibility

Thanx Ariel for starting interesting APM ongoing forum, lot of valuable info/inputs from our peer's couldn't stop liking those valuable comments. I would personally suggest ease of APM solution which supports multiple diversified Apps ,DB's,O/S all in one shot for complete system including Upstream/Down stream systems and produces transaction level reports which can be easily drilled down and can be understood by Non technical management teams also.

- each transaction must be monitored and analyzed
- the solution has to correlate traffic generated by a user with its true EURT measure
- any kind of metrics has to be suppored by its pcap, so the system has to be able to extract automatically the pcap related to the analyzed detail
- great reporting and dashboarding

All functionality being equal, I want ease of deployment and low management overhead

User Visualization and interpretation.

Level of monitoring for Deep dive Analysis
Transaction level monitoring

Depth of analysis and ease of use

Most efficency in performance and root causes for issues

Cost, flexibility and restitution quality.

Low TCO Real and Synthetic User Experience Monitoring

Application component deep dive

The ability to monitor everything across the stack.

Approval to run in the government space.

Action base monitoring

In depth coverage of the Stack

Ease of use is probably the biggest item to consider for APM. If no one will use it then it's just wasting money.

Ease of use and limited/no performance impact form the toolset itself.

VM’s, scalability, easy of use, accessibility

1. Doing what it has promised, helping root cause analysis
2. End-2-end transaction monitoring (end user+data center) without sampling
3. Discovery of application topology
4. Not affecting apps that are monitoring

Easy to deploy, accurate results on transactions.

Finding the bottlenecks and Drill down the bottlenecks to increase the performance of the application.

The excellent end user experience regarding on transaction and also conversion rate.

Ease of use, self discovery capability and ability to interoperate with AANPM

Automation and Visibility

Accessibility - Manageability - Stability - Availability

Manageability of the tool

Software lifecycle

Price, scalability and easy of use

Dealing with VM's