Coming October 25: PeerSpot Awards will be announced! Learn more
2014-02-27T19:49:00Z
PeerSpot user
IT Analyst at a government with 501-1,000 employees
  • 42
  • 895

BI Tool Evaluation Criteria Rating Matrix -- anyone have one they've used in making a tool selection?

Does anyone have a BI (Business Intelligence)/BA tool Evaluation Criteria Rating Matrix or RFP they used in making a tool selection?


Several users submitted their BI matrix to share with the community. You can download them here.

If you have a matrix or RFP to share with our community of tech buyers, please email it to tools@itcentralstation.com

33
PeerSpot user
33 Answers
PeerSpot user
Lead Technical with 1,001-5,000 employees
Vendor
2015-12-19T17:28:44Z
19 December 15

You can get a comprehensive comparison from Panorama website, we have used this, it is very informative

http://www.panorama.com/bi-comparison-tool/

PeerSpot user
Developer at a tech services company
Consultant
2015-12-15T13:04:14Z
15 December 15

First of all the Gartner - well structured in-depth analysis, forecasting

Jhornber - PeerSpot reviewer
Director, BI & Analytics at a leisure / travel company with 10,001+ employees
Real User
2014-04-01T17:52:43Z
01 April 14

KLBaxter: Partly because we were focused mostly on the dashboarding side of the equation, and also because we already utilize a combination of SSRS and SAP Business Objects for canned and ad hoc reporting respectively, with no plans to replace those products in the near term. Had we been looking to find a single solution to cover most, or all our reporting and analysis needs, I'd likely have explored some additional vendors, especially because canned & distributed reporting is not one of Qlikview's strong suits. However, as I'm sure you and many others here have also come to realize, finding one vendor that is a leader in all things BI is a very lofty ambition. The phrase "Jack of all trades, master of none" comes to mind. ;)

PeerSpot user
Information Project Analyst with 501-1,000 employees
Vendor
2014-04-01T08:17:48Z
01 April 14

Jhornber - was there a reason Cognos wasn't evaluated? All my initial scoping keeps leading me to QlikView but I'm determined to give the others a fair trial!

Jhornber - PeerSpot reviewer
Director, BI & Analytics at a leisure / travel company with 10,001+ employees
Real User
2014-03-31T22:04:18Z
31 March 14

Ultimately Qlikview was selected, as it seemed to best meet our particular needs.

Ariel Lindenfeld - PeerSpot reviewer
Director of Content at PeerSpot (formerly IT Central Station)
Community Manager
2014-03-30T13:36:11Z
30 March 14

Thanks for sharing Jhornber! Which BI tool did you end up selecting?

Find out what your peers are saying about Microsoft, Tableau, SAP and others in BI (Business Intelligence) Tools. Updated: September 2022.
633,572 professionals have used our research since 2012.
Jhornber - PeerSpot reviewer
Director, BI & Analytics at a leisure / travel company with 10,001+ employees
Real User
2014-03-30T13:28:20Z
30 March 14

I've posted my BI tools selection criteria here:
http://www.itcentralstation.com/articles/i-evaluated-business-objects-tibco-spotfire-tableau-qlikview-sisense-prism-and-pentaho

Let me know if you’d like to know anything else!

Ariel Lindenfeld - PeerSpot reviewer
Director of Content at PeerSpot (formerly IT Central Station)
Community Manager
2014-03-27T18:03:01Z
27 March 14

Jhornber - Can you share your criteria here with the community? I'm sure everyone will benefit from reading it.

PeerSpot user
Team Leader, Mfg Systems Pulp at a manufacturing company with 501-1,000 employees
Vendor
2014-03-27T16:41:34Z
27 March 14

Yes, I would be interested in seeing your criteria.

Joanne.morgan@canforpulp.com

PeerSpot user
Project Manager at a university with 1,001-5,000 employees
Vendor
2014-03-27T16:12:37Z
27 March 14

Jhornber, it will be nice to share it.

PeerSpot user
Application Manager at a tech services company
Consultant
2014-03-27T16:11:16Z
27 March 14

Please share your thoughts...

Jhornber - PeerSpot reviewer
Director, BI & Analytics at a leisure / travel company with 10,001+ employees
Real User
2014-03-27T15:52:20Z
27 March 14

I just completed an evaluation of (and ultimately a final selection from) several vendors and identified about a dozen high level evaluation criteria (using Gartner capabilities as a starting point, and adding those specific to our focus). We were primarily concerned with an efficient, competent dashboarding tool with a secondary focus on ad hoc analysis. If you're interested in seeing my criteria, let me know. Happy to share if it helps.

PeerSpot user
Information System Manager with 501-1,000 employees
Vendor
2014-03-27T11:40:53Z
27 March 14

We have noted that one single BI does not meet the requirements of large organisation. I would suggest some changes in the template shared.

PeerSpot user
CEO with 1,001-5,000 employees
Consultant
2014-03-27T11:09:51Z
27 March 14

Brevity is rewarded - the operative word is "report" and not analysis visual or otherwise

PeerSpot user
Sales at TIBCO Spotfire
Vendor
2014-03-27T08:19:50Z
27 March 14

Hi, before responding to this question let me be open and declare a bias towards certain types of BI tools because I work for Tibco Spotfire, a data discovery and visualisation vendor.

Leaving product aside therefore lets establish some principals before starting down the road of a matrix - firstly, what needs to be in that matrix?
What do the business (and technical) users want to be able to do? There is a wide variety of capabilities across the vendors and building a matrix on what every vendor does is going to be a time consuming exercise to both build it and then administer it, especially if it is not all relevant to what the business needs.

For instance, if the key requirement is to burst out a pixel perfect report every Monday morning to 300 recipients then tools like Qlikview, Tableau, and Spotfire are unlikely to be your best bet.

If on the other hand you want to empower users to explore data themselves (in order to minimise the IT overhead), mash-up data from disparate systems, follow a train of thought to identify new threats and opportunities, and collaborate across the enterprise with colleagues on the insights you are discovering, and quickly prototype new dashboards - then the new generation of data discovery tools is likely to be more useful. That said, according to Gartner, this is where all of the traditional BI vendors are trying to take their products. They just haven't got there yet. So, timescales are important.

The fact is that extensive matrices are a tick box exercise, perhaps okay for a first cut, but largely a waste of time. All vendors (or their marketing departments) will try to put as many ticks in the product boxes as possible. Just because a vendor says their product can do something does not mean they do it very well and there are only two ways to find out the truth of it - buy it and build it, or talk to someone who has.
If you are clear about the capabilities you need (and I emphasise 'capability' not product 'functionality') your short list comes together pretty quickly. At that point, you start on the detail, and you will find your matrix more focused and easier to manage and you will be able to get to a conclusion a lot more quickly.

If you do pull down a generic matrix, use it as a guideline to getting to something specific that will meet the needs of your business, not someone else's a few years ago.

PeerSpot user
IT Analyst at a government with 501-1,000 employees
Vendor
2014-03-25T09:00:26Z
25 March 14

Thanks for all the helpful comments. Please send me the full matrix from example 1. Thanks, Steve

PeerSpot user
Architect at Microsoft
Real User
2014-03-24T22:19:04Z
24 March 14

Starting with customer acceptance criteria before going to a Evaluation scorecard would be my advise. Ask them what they want and go have a look on how they work. If you choose the "best" tool but the rampup for learning how to use it is too big, customer want simply reject. Usual they know what they want, you just Need to translate it into a Feature & Performance set.

Disclaimer: I work for Microsoft

PeerSpot user
Consultant at a tech services company
Consultant
2014-03-24T14:30:14Z
24 March 14

Quite an interesting discussion and eye opener.

Rusty Isaac - PeerSpot reviewer
Vice President at PeerSpot
Vendor
2014-03-23T16:21:33Z
23 March 14

Very useful, thanks.

Ariel Lindenfeld - PeerSpot reviewer
Director of Content at PeerSpot (formerly IT Central Station)
Community Manager
2014-03-23T14:58:03Z
23 March 14

Very interesting discussion! We've added several examples of BI matrices submitted by the community. See them here: http://www.itcentralstation.com/articles/bi-evaluation-criteria-matrix

Email matrix@itcentralstation to get your complete copy.

Vendor
2014-03-23T08:26:23Z
23 March 14

I would recommend Gartner's BI Magic Quadrant see http://www.gartner.com/technology/reprints.do?id=1-1QLGACN&ct=140210&st=sb . Hope this helps.

Vendor
2014-03-23T08:21:51Z
23 March 14

The problem with tool selection criteria is the way they are applied and the lack of any environment-specific filters. Check out Gartner and then go with what they see as the future.
I personally advocate Cognos for technically competent organizations. Government agencies do not fit this criteria.

Almost without exception, organizations(especially government orgs) do not have the expertise to implement BI tools properly. Microsoft had a terrible false start with Performance Point Server a few years back. They actually had people lined up to buy their BI tools and had to back out of the market until they got their tools matured. BI is not trivial.

To wring performance from BI tools requires a few people with very solid data modeling abilities and exceptional data/db skills. These people are hard to find and hard to keep.
Another thing that causes havoc when buying BI tools is any organization's structure. If the org has an ETL group, are they going to do the BI ETL or is the BI group going to do that also? Will your organization let the BI group build report-specific databases? Who will maintain the reporting databases; The enterprise or the BI group or both?

BI was crippled at a large aerospace company because the organization considered all databases to be equal. This meant that an enterprise warehouse counted one and the tiny, by comparison, BI report-specific cubes and stars were also counted as one each. The company bought Cognos without the ETL/DB building tool "Decision Stream/Data Manager" and suffered because of it. Finally after many years of failure or slow progress, it was acknowledged that BI requires its own databases to gather and stage the data. You cannot ask a meta-data modeling tool such as "Framework Manager" to do everything and you cannot ignore ETL. A BI environment will have dozens or hundreds of stars and cubes for specific purposes and report requirements.

I work at a government agency right now and would rate them a 1 on a scale of 1 to 10 for technical ability. There are sharp, up-to-date people but they are the exception. There are maybe 5 out of 100 with current skill sets.

Vendor
2014-03-23T08:14:48Z
23 March 14

I have one I used, but frankly I found that the "can you do this" feature-specific type matrix doesn't get to the real crux of why one tool is better than another.

A good deal of tool selection has to be based on what will fit with your company: are you aggregating data from multiple sources? If your database organized such that building views will be fairly easy? Are you migrating an existing reporting system for which you already have all the SQL and you don't want to re-invent the wheel? Do you have the resources to spend 6 months getting an "end-to-end" tool implemented? (And there are some good ones out there).

What about in-memory technology and performance? Caching? How does the cache get refreshed? Can I use my native SQL database, or do I have to build a star schema?

I find feature lists don't uncover these kind of implementation/design criteria.

PeerSpot user
Vice President at JPMorgan Chase
Vendor
2014-03-23T07:46:01Z
23 March 14

I don't have access to an evaluation matrix. I worked for a very large company that had many BI applications at my disposal. There are a kit of considerations to be made. The most important considerations surround cost and functionality. After the budget is identified and the right applications are identified the next steps would be to gauge the support available from the vendor and then identify internal resources available to help in that effort. Some application licenses are more expensive but they also provide great support.

it_user6534 - PeerSpot reviewer
BI Expert at a retailer with 501-1,000 employees
Vendor
2014-03-23T07:43:39Z
23 March 14

I would suggest checking in with Cindy Howson at the BI Scorecard. She typically has the tool area covered nicely. She could have just the info you are looking for.

Ts Nurul Haszeli - PeerSpot reviewer
Technology Consultant (Data, BI, IT) at Freelance
Real User
Top 10
2014-03-23T07:40:09Z
23 March 14

Yes. I can help out. You can send me a message.

PeerSpot user
User with 1,001-5,000 employees
Vendor
2014-03-20T20:36:47Z
20 March 14

When it comes to BI, unless the business sees the need, it will forever say no!

When it does come around to saying yes, to make sure the win is certain, and the picture is rosy clear, I suggest the following approach:

Clearly define your requirements - (every step of the way):

In a rather common sense approach, satisfy the following questions/aspects:

1. What's my target?
How can the business benefit? Is it just for a specific unit/department, or the entire business? Can I take a modular approach - add on over time?

2. Functionality:
Who are my target audience - endusers, SMEs, hybrid (mix of both), executive management?
What kind of training will they need?
How many users - now and in the future? - plan for growth
Is the tool for internale use only, or both internal and external? Will it be accessed outside the firewall?
Do executives or salespeople need mobile BI capabilities?
Will dashboards, data visualization tools, etc. be implemented as well?
How important is real-time BI for operational workers, like customer service staff or tech support?
What kinds of ad hoc queries must the tool be able to handle?
Browser-based? Specific Application?

3. Licensing:
What pricing/licensing options are available?
How are support and maintenance contracts set up?
How many users will be querying the system - concurrent users, named users?
What kind of licensing scheme?

4. Off the shelf vs. customization:
What can the product do?
How does it meet our requirements?

5. Data makeup and placement:
Will you need to build a new data warehouse?
Heterogeneous environment... or separate data marts?
Centralized data, or all over the place?
Can you identify you data sources?
What is the technical environment that this BI software will need to integrate into (source systems, databases, preferred hardware platforms, etc.)?
How will your BI software integrate into our current environment?

6. Data volume and growth:
What are my data volumes today? What will they be in the near and far future?
What are your software update cycles like? How easy is it to add on new functions/features?


7. Training:
How will BI users and administrators be trained?
All-inclusive training class?
Or will you train only administrators and have the administrators teach the others?
8. Testing:
Do you want to test the product in your environment? Can we get a proof of concept?
Do you want a Proof of concept -- in your environment with your data? Or use Vendor's test environment?

9. Time frame for implementation?
Corporate deadline?
How long will implementation take? Will you need consultants or vendor services? In-house team?
Who's going to be on my BI team?
Who's driving the process (evaluation & purchase) - IT? Or by the business? Or both?

If you've made it this far, you've answered the biggest question - Do we need BI?

Now you can finish with the last 2 questions:

1. Should we write a request for proposal (RFP)?

2. What's my budget?

3. Who can we talk to to compare LOE (level of effort)? Find someone in the same field/industry.

PeerSpot user
BI Expert at a financial services firm with 501-1,000 employees
Vendor
2014-03-20T20:23:09Z
20 March 14

A review of the gartner magic quadrant is a good starting point. There also might be something for use on the TDWI website

PeerSpot user
Project Manager at a healthcare company with 1,001-5,000 employees
Vendor
2014-03-20T20:20:24Z
20 March 14

Gartner probably does the best job of all the rating organizations, but Klas also has a pretty good reputation for healthcare products.

PeerSpot user
BI Expert at a non-tech company with 51-200 employees
Vendor
2014-03-20T20:14:28Z
20 March 14

This is a pretty comprehensive overview with a Magic Quadrant chart.

http://www.gartner.com/technology/reprints.do?id=1-1QLGACN&ct=140210&st=sb

Good luck.

Vendor
2014-03-20T20:13:09Z
20 March 14

Gartner BI-MQ-2014 can help

PeerSpot user
BI Expert with 51-200 employees
Vendor
2014-03-20T20:09:25Z
20 March 14

The best and most known evaluation Matrix is Gartner´s Matrix. It also explains all the criteria and the strengths of the most important BI tools in the market

http://www.gartner.com/technology/reprints.do?id=1-1QLGACN&ct=140210&st=sb

PeerSpot user
CEO with 1,001-5,000 employees
Consultant
2014-03-20T20:05:03Z
20 March 14

Yes - but it addresses only data visualization and analysis vendors

Related Questions
reviewer1772406 - PeerSpot reviewer
User at Unionbank Philippines
Feb 02, 2022
Hi experts, I work at a large financial organization. I have been wondering what types of problems (=use cases) can be solved/optimized using the Edge AI technology in industries such as Banking and  Finance, Power and Land and Agriculture?  I appreciate your thoughts and suggestions.
See 1 answer
Djalma Gomes, Pmp, Mba - PeerSpot reviewer
Managing Partner at Data Pine
02 February 22
AI has already been used for decades in banking, finance and agriculture.  Credit and loan analysis use a lot of logistic regression to know who to loan money to. Finance uses AI for regression analysis of forecasts and optimum stock portfolios, agriculture to predict production and supplies needs based on the amount of rain forecasted. What we have is the evolution of tools, methods and techniques. We no longer need cubes. Although Hadoop is meant for safe storage at low cost and not for AI purposes, tools have evolved and new tools arose providing the speed required for AI analysis. We have evolved in terms of data governance with the use of DMBoK (Data Management Body of Knowledge), visual planning frameworks like the Machine Learning Canvas, Self Service BI approaches stimulating collaboration amongst users.  We now have tools, practices, frameworks and enough manpower that makes the job more challenging. We have been using AI in those sectors for quite a while, but now we have the momentum to reach a different level of data thinking organizations.
PeerSpot user
Program Manager at a aerospace/defense firm with 1,001-5,000 employees
May 26, 2014
We're looking for recommendations for embedding reporting for MySQL. We are looking for a solution to embed reports in our web app. Our database will be deployed on the cloud in a Linux environment and the reports need to be triggered on demand. We have been looking into BI solutions, although our use case is not BI. We want to easily create reports that can show both data and charts and inclu...
2 out of 13 answers
PeerSpot user
Manager of Data Analytics at Viewpoint Construction Software
03 April 14
We recently implemented an embedded BI solution in our collaboration software using LogiAnalytics Info and were very happy with the results. The solution provides users the ability to create customized dashboards and the underlying data source is a web service using JSON. The Architect wrote a couple of plug-ins to support our token based security and re-pivot of the data, they took about a week to complete. Also, we used CSS to re-skin brand LogiAnalytics as part of the application. LogiAnalytics licensing model is flexible and we choose server based licensing since we have a large user base with unpredictable usage patterns. Our solution includes various data visualizations and grids tied to our underlying search API. LogiAnalytics doesn’t provide pixel perfect reports but can handle most other duties.
PeerSpot user
IT - Consulting at a tech services company
03 April 14
I suggest you to use a report Server such as MICROSOFT (SSIRS), Jasper Report Server or any other related Technology. You'll have to first create a connection to your MySQL db and design those reports. So the user will log in into this server and call reports they have access to.
Download Free Report
Download our free BI (Business Intelligence) Tools Report and find out what your peers are saying about Microsoft, Tableau, SAP, and more! Updated: September 2022.
DOWNLOAD NOW
633,572 professionals have used our research since 2012.