WhereScape RED Reviews

Enterprise Data Warehouse and enterprise data staging.  Automation and documentation of all data warehousing processes.

It did not pass usability, reliability, sustainability or performance tests so there was no potential benefit other than minor documentation advantages over the current automated documentation solution.

Documentation has excellent potential as it provides the capability to generate data lineage and a complete set of rich documentation and output in multiple formats.

The tool supports multiple target update methods.

Support for multiple design methodologies including Data Vault.

In my opinion, the tool requires a much different architectural approach to be effective. It has fundamental architectural flaws that prevent it from being a practical solution for even moderate volumes of data.

The largest issue is that it violates the most basic database data-movement physics through its reliance upon the infamous RBAR or "row by agonizing row" cursor operations within Tsql. In contrast, efficient database apply operations to whole sets of data rather than row by row achieved through Sql joins and set based queries. Note that this is a common problem with code generators as they end up trading scalability for flexibility. Ironically this also makes them unable to take advantage of the much touted "push down" operations that is one of their big selling points. This tool avoids the traditional lookup ETL lookup cache component issue (sometimes referred to as blocking transactions) - that much is true.  However it ends up doing something similar but just within the database and with no easy way to override it.  Experienced ETL developers know not to do any blocking transactions or RBAR if they can possibly help it. And with traditional ETL tools you either take advantage of built-in push-down capabilities or pre-join the data in views or data sources to avoid this kind issue.

Other constraints include...

- Repository only supports a single target database

- No support for change data capture or delta detection - that must be custom coded

- Must be used to stage data or it requires a redundant copy of staged data.  This ended up more than double our storage

- Relies 100% upon target database server with no ability to scale ETL to separate server

- Relies 100% on the target database engine to do all the work (along with everything else the database must do like reporting)

- Cannot acquire data from sources other than flat files. It relies upon the use of linked servers (which many shops don't allow) or calling an ETL tool such as SSIS in the background for data movement. In that case you're still relying on an ETL tool for the ELT portion so then you're actually using two different tools - Wherescape plus a 3rd party ETL tool instead of just one tool.

Specific list of areas for improvement:

* Performance (this is the area that needs the most improvement)

* Efficient storage utilization

* Logging

* Count validations (data movements don't validate counts which is 101 basics to prevent data leaks)

* Scheduler reliability and transparency

* Ability to separate data acquisition/staging processing from target schema population

* Providing automatic record count validations

* Support for change data capture

* Support for in-line transformational procedure language that extends beyond database procedures

* Documentation lineage accuracy and completeness

* Support for sharing metadata repository across multiple targets

* Improvements in staging architecture requiring fewer copies of source data

* Supporting transparency into post-gens

* Providing encapsulated pre and post events for custom code that doesn't get wiped out when you re-generate supported code from the tool.

* Better support for task orchestration in a top-down way that can be easily understood.

* Support for direct parameter passing from job schedulers.

The scheduler often hangs with nothing to inform the user why it isn't working. Sometimes this is simply due to a developer starting a scheduler from their PC which integrates with the repository but isn't obvious when looking at the shared scheduler. Other issues seemed to be only resolved by re-generating the process from scratch and hope for a different result (which often there is), especially if the scheduler hangs. It is very easy to choose an incorrect option when generating processes only to result in something that just hangs with no error message. Then you're knee deep in database procedural language trying to debug a tool's generated code.

The sheer resources that it consumes easily overwhelms a server and causes the database to freeze due to exceeding resource constraints. The tool generates an unexpected and seemingly unpredictable amount of staging data (both in and out of the database depending on data movement option chosen) consuming very large amounts of disk space with redundant copies of data. This tool will not promote friendly relations with your system administrators.

The product was deemed unscalable in terms of data storage and performance.

Customer Service:

Very good

Technical Support:

Very good, although this was a POC so I can't speak for production support after the product is purchased.

Currently use a traditional ETL tool that meets our requirements. The test scenario involved around 250 GB of source data with a resulting 50 GB (compressed) star schema target. The ETL tool can talk to all our sources including multiple cloud sources using SOAP and REST web services and this tool is just a database procedure generator with no adapter capability. It instead relies on ETL tools to do that work so it doesn't eliminate ETL tools ending up having to support not one tool but two tools - both in licensing, support and expertise which was expected to cause significant increase in total support costs.

The tool relies on ODBC connections requiring complications in security in an AD environment over using OLEDB.

It requires a developer to either access the server directly or setup file shares. It was often confusing when doing development from a PC but working with a scheduler on a server and knowing what files need to be available from the PC verses from the server. The tool requires a great deal of attention to disk usage and location especially when using native database load utilities as it unloads all the data first into temporary files during the load which can easily consume your servers available space in unpredictable ways. Getting the environment functioning just for the training class was unexpectedly difficult.

The metadata repository is unique to a single target database and cannot be shared with any other target which prevents use beyond a silo datamart. This means data lineage is a silo repository for each target.

Software vendor was the lead implementor.

The ROI was expected to be very negative to the point of failure.

Factor in the price of specialized consulting who know this product. They're hard to find and expensive.

The existing traditional mainstream ETL tool and the previous one have been functioning without problems but the concept of a self-documenting tool that provides a common framework and shared metadata repository was a compelling alternative. This tool was already in use in our shop in a limited way so it begged the question on whether it could serve an expanded role.

Advice: Before choosing any tool it is best to do your homework and compare features to competitors and test the tool in a real life volume scenario. Software is never what its advertised from any vendor. Don't take other's word for it, especially if they have a vested interest in you using the tool. I know of testimonials about this tool and others that are based on wishful thinking and not reality. Caveat Emptor my friends. So the real question is how far off is it and does it have show stoppers that will prevent it's use in your environment? Fail to take this precaution at your and your organization's peril.

The compelling selling point for this and similar tools as "datamart in a box" providers is that they are pre-built frameworks and documentation. The opportunity is that staff with little data warehousing experience can get up to speed easily and not have to worry about building logging, orchestration and error frameworks or even know much about data warehouse design.

Unfortunately this tool's basic architecture has severe flaws which make it likely impractical for most real world marts or data warehouses. It has some significant cons in the following areas:

- Metadata support specific (cannot share across multiple) target database

- Unreliable lineage

- Lacks capability to interface with application API's

- Lacks out-of-box CDC control capabilities

- Poor/outdated GUI (looks like early 1990s)

- Lack of available resources in the market place who can operate this tool

- Lack of customer base and information available

- Primitive logging and error handling

- Forced duplication of staging data multiple times

- Inability to separate source data capture from targets (tightly couples sources with targets)

- Stores all objects in single target database causing number of issues such as backup/recovery, fragmentation, limited organization options for objects

Automated creation of documentation, quick time to market, ease of use, code standardization and support for Data Vault 2.0.

We can focus on the data itself (data quality, MDM, metrics) instead of spending time building code to move and transform the data. During the POC myself and one WS resource were able to deliver a project in 3 days that had been in development by other resources for 6 months and still hadn't been delivered.

We were more productive from the moment we started using RED but that X-factor has only improved as we have become more comfortable with the toolset and as RED adds new features with each successive release.

Since WS supports Data Vault 2.0, I would like to see automated support for hash keys on the appropriate Data Vault tables. WS does have a table trigger that can be manually added to the WS repository that will accomplishes this but it isn’t built in to the IDE. I would like to see multi-database (creating/storing tables in different databases based on type, e.g. staging, data mart, etc.). Similarly to the hash keys, it can be accomplished (I'm using multiple databases) it just requires making a manual entry in a RED repository table for each database. Again, having it officially built in to the IDE would be great. The only other thing I would note is that the IDE isn’t as intuitive as I would like. Like most tools, after several weeks of use it makes a lot more sense…but for new users there will be a moderate ramp up time.

About 1 year.

No.

The releases have been of surprisingly high quality compared to other vendor software I have used. We had the opportunity to test out a customer preview (i.e. beta) and were surprised to find no stability issues nor bugs in our testing. The features that were exposed in the customer preview were stable enough we felt comfortable moving forward with upgrading our Dev environments to begin development work using those features.

I have not run in to any issues with scalability at this point. Although I don't expect code-generation tools to write optimal code. I am surprised to see WS generate fairly good SQL code. I am very skilled in writing SQL and I was very impressed with what was generated.

Previously, I felt that I might need to write custom procedures to handle large table loads (i.e. 500+GB in a table), however, I no longer believe that to be true. I haven't reached those sizes in this environment (yet), however, I believe columnar indices and other standard SQL features would be able to cover any performance concerns since RED is producing well written SQL.

Good. Searching on the customer forums is very well done and has helped me not only solve issues I was running in to but also discover other features I wasn't aware of.

Good.

1. Information Builders iWay Data Migrator

2. SQL Server Integration Services

3. Hand coded SQL procedures

Data Migrator is just another ETL tool and the UI is not intuitive. Multi-threading is the only advantage Data Migrator has over SSIS and hand-coded procedures, however the cost isn't justified.

WhereScape RED handles the versioning, building of the objects (SSIS packages, SQL procedures, etc.) and provides an advanced, multi-threaded scheduler that makes it easy to manage object dependencies and objects that can be loaded in parallel.

A REALLY COOL feature is to create a track-back diagram from some destination object, e.g. FactSales. That opens a tab with a pretty diagram of the flow of data from the source system all the way through to the fact table. Tell RED to create a job from the diagram and *boom*, you have a schedule with all tasks and dependencies created to make sure that FactTable is loaded appropriately.

I felt the setup was fairly straightforward. I've seen better and I've seen worse. As with most things, it's just wrapping your head around the architecture and methodology WS has implemented.

In-house.

I believe the break-even on the investment is less than 1-year at this point.

Because WS builds native SQL code you don't have to worry about vendor lock-in as with other solutions. If you decide to walk away from WS you still have a working data warehouse environment. The only thing you would need to do is schedule the load procedures with another scheduling tool. Licensing negotiations are pretty straightforward as the development IDE is just a seat license model. They are very flexible in how they approach licensing.

I have used SSIS, Information Builders' Data Migrator and home grown SQL Server stored procedures. If I was to order these solutions in terms of ROI they would be:

1. WhereScape RED

2. SQL Server stored procedures

3. SSIS

4. Data Migrator

No solution is perfect and I could probably find nits here and there, however, I find WhereScape RED to be exactly what I was looking for. Roughly 80% of a data analyst's job is preparing data and 20% is doing the analysis. I purchased WhereScape RED to flip that equation on its head.

• The automatic generation of all scripts allows for a very agile development process.
• Wherescape also generates very thorough documentation automatically.

The development speed was greatly improved by using WhereScape. It saves time in virtually every aspect of building a Data Warehouse. It provides a great overview of everything you've created which makes trouble-shooting much easier as well.

There are some newer features that haven't been included in the training materials yet so they're a little outdated (although the help documentation is very good).

There are also some more complex data transformations that aren't supported and so need to be done in multiple steps.

We are two months into an enterprise-wide data warehousing project using Wherescape as the primary tool.

No deployment issues noticed.

No.

No.

In our experience they are happy to help and quick to reply with development solutions whenever needed.

No, this was a ground-up Data Warehouse build.

The initial setup is very easy and we were able to begin development almost immediately.

We are a data analytics consultancy that is implementing WhereScape for one of our clients.

Two months into the project we are currently 50% under budget due to how quickly it's possible to develop large-scale solutions in WhereScape. We were able to train a team with no prior WhereScape experience and begin development in much less time than if we built a standard data-warehouse, while writing procedures and creating our own documentation.

WhereScape pays for itself very quickly.

We considered many alternatives including only using SSIS and SQL Server to build the entire warehouse.

Automation, Automation, Automation....

Automation of Data Warehouse Tasks - Traditional ETL tools are a one-size-fits-all solution. They can often be like a programming language--you can do anything you could ever imagine, but you have to manually create each and every instruction. This leads to a total lack of agility because you have to manually create each and every step. WhereScape, on the other hand, is meant for the specific task of building data warehouses. The product takes on much of the painful, time-consuming, manual and repetitive tasks, allowing you to focus on meeting business requirements.

Automation of Documentation - This is perhaps the best feature of the product. Based on your design, it creates two sets of documentation--one for your technical teams and one for end-users--which show the data model, metadata, and just about anything else you’d want to know about the model.

Impact Analysis - During the lifetime of a data warehouse, there is regularly a need to enhance and extend the warehouse, but it can often be difficult to understand how changes will impact other things downstream. In WhereScape, you can simply click on an object and perform an impact analysis to see how changes to that object will impact other objects. You can also do the reverse of this, looking at an object and “tracking back” to see what objects are used to build it.

Code Runs on the Database - When you build objects in WhereScape (through a point-and-click GUI), the product automatically generates target-specific code. So, for instance, if your target platform is SQL Server, it will generate T-SQL stored procedures for updating your warehouse objects. When jobs are run, this code gets executed on the target database. This essentially means that the ETL is serverless. You do not need a massively sized ETL server to handle the ETL code because it’s simply code being handled by the database, which in my experience, is much more efficient.

We are able to build data models much faster than previously and much faster than using a traditional ETL tool. It allows us to focus on the most important aspect of a data warehouse--business requirements--instead of focusing on the low-level details of the development platform.

As with any product, there are things that can be improved, but most are fairly minor, including things like greater flexibility in ordering job tasks. I’d also like to see increased support for NoSQL data sources.

We have been using the product for about 5 months.

None yet. Because there is no server component, other than a lightweight job scheduler, it’s really just the installation of the software and some basic initial setup. Couldn’t be easier.

No.

No.

Very good customer service. Kind, helpful, and knowledgeable.

This is another area where WhereScape shines. They are not a huge organization, so you get very prompt and personalized care. My support tickets are always answered very quickly and they listen to my suggestions. And, perhaps most importantly, their support team knows the product inside and out. You will not deal with inexperienced first-level support techs.

They also made experts available to help us quickly answer any questions about best practices, how to accomplish specific tasks, etc.

Our purchase of WhereScape was part of a larger re-architecture of our data warehouse. We chose to move away from other products for a variety of reasons, but our choice of WhereScape was largely based on its ability to speed up the development process through automation.

As noted previously, the software generates database code for the target platform and runs that code on the database server itself, so it’s very lightweight. The only components are 1) Client Software and 2) A Lightweight Scheduler. The installation of these two components is very quick and easy.

In-house, although we did get some help from their professional services group, largely for advice on best practices, how to handle complex data transformations, etc. The team is very knowledgeable and seems to have answers to just about every complex question we could think of.

Probably a bit too early to report on this but we expect a good ROI.

The product is much less expensive than traditional ETL tools. Licensing is very simple to understand.

Yes, we looked at various traditional ETL packages as well as other Data Warehouse automation tools. We chose WhereScape over traditional ETL for reasons noted previously and chose WhereScape over other automation tools because it appeared to be the most mature of these solutions.

As other reviewers have noted, data warehouse automation requires you to change your thinking and perspective somewhat. If you are looking at this product as a simple replacement for your existing ETL and try to use it exactly the way you used ETL, then you will not fully realize the functional benefits (the cost benefits will still be there though). You need to think more in terms of agile data warehouse design. For those new to this concept, I recommend a book called “Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema” by Lawrence Corr and Jim Stagnitto. WhereScape fits very nicely into the concepts presented in the book.

• Metadata-based architecture
• Ability to prototype solutions quickly
• Automated documentation
• Great performance

The biggest win for my organization are the great improvements we have seen in load / refresh processing time over our legacy ETL tool. I was skeptical of the ELT architecture going into our Proof of Concept with WhereScape. I had believed that a more traditional ETL architecture of dedicated ETL servers and applications doing the transformations in memory, then writing to the database, would be faster. It turned out that the ELT approach of using the muscle of the RDBMS was significantly faster than our legacy ETL tool. We are now 2.5 years into using WhereScape and all warehouse code we have converted from our old tool to WhereScape has performed faster; anywhere from 20-80% reduction in processing time. This was huge for us, as increased demand for the warehouse and increased upstream operational system batch processing has really shrunk our batch window. We now have room in our batch window to add more content to the warehouse. In addition to improved performance, we no longer need dedicated ETL servers in our warehouse environment.

The documentation WhereScape provides has also been great. We have always struggled with maintaining documentation on our ETL. It is very nice to get source-to-target mappings, table diagrams, and dependency diagrams, which are up to date, with the click of a button.

Finally, the use of RED has resulted in a much more standardized, easier to support warehousing environment, from what we had prior.

I would love to see a GUI interface for defining dependencies between build processes. RED provides a spreadsheet like interface for defining the dependencies between builds. Once the dependencies are defined, RED can produce a nice dependency diagram to give a visualization of the dependency tree. It would be easier to define complex dependency relationships if the dependency diagram were interactive. Our legacy ETL tool provided this GUI dependency definition via a drag and drop diagram which was very useful. The solution provided by WhereScape does work, and the dependency diagrams generated are helpful. It would just be nice to have the ability to define dependencies via a diagram, since dependencies relationships are much easier to understand via a diagram.

I have used it for 2.5 years.

We have not encountered any deployment issues.

We have not encountered any stability issues.

We have not run into any scalability issues. As mentioned elsewhere, we have seen significant improvements in our “ETL” processing times as we have converted our ETL logic and processing from our legacy ETL tool to WhereScape RED. Here is a little information about our warehousing environment:

• Oracle RDBMS
• 4 ETL developers
• 15 subject area dimensional data marts
• 905 tables (production presentation tables only)
• 1,160,393,098 records (production presentation tables as of 1/31/2017)
• 1.2 Terabytes of used database space (production only)

Customer service is great. Our interactions with WhereScape staff have all been very positive.

Technical support is great. Two-and-a-half years in, we have not encountered a question we could not answer via a simple search on the Customer Support Portal. There has not been a need to open a support ticket yet.

We previously had a traditional ETL tool, which was being sunset by the vendor.

WhereScape RED has a very straightforward, easy-to-understand architecture, so setup was easy.

The learning curve is relatively mild, so the implementation team was the existing ETL team of four developers. The team did go through a couple-day WhereScape RED onsite training session. We also worked with WhereScape’s professional services on a short engagement to evaluate our legacy ETL environment and come up with an approach for conversion of the legacy code to WhereScape.

ROI is hard to measure, but faster time to market for new subject areas, and increased development with existing staff would be two areas we have seen improvements due to our purchase of RED.

An additional influencing factor in our decision to purchase RED is their licensing model. The other products we evaluated were all CPU-based licensing. The license quotes came in very high for the vendor-estimated CPU needs for our existing data warehouse; this did not include future expansion as our warehouse environment should grow as we add more subject areas.

WhereScape RED licensing is based on developer seat. This is great for us. We purchased a four-seat license and we do not need to worry about architecting our warehouse around CPU licensing restrictions. We can size our Warehousing databases to meet the performance needs of our batch processing and user queries. We can also build and support a lot with four developers.

• Oracle Data Integrator ODI
• Informatica Power Center
• IBM InfoSphere DataStage

WhereScape RED has been a great fit for my organization.

The most valuable feature of WhereScape RED is its metadata-driven approach to data warehouse ETL development, although its scheduling and administration capabilities are also a huge plus. Our Teradata data warehouse codebase is over 2000 stored procedures and counting. RED makes administering, enhancing and maintaining this codebase as painless as can be reasonably expected.

Previously, our data warehouse codebase was homegrown, hand-written stored procedures that made scheduling, administration and tracing problems extremely time-consuming and sub-optimal. Now with RED we have much better control over our codebase and processes.

Self-Documentation capabilities while good, are an area we would like to see continued improvement.

6 years.

No issues with deployment have been encountered.

No issues with stability have been encountered.

No issues with scalability have been encountered, in fact we continue to do more and more with RED.

Excellent.

Very good. We have even requested troubleshooting or help with tricky problems and the RED engineering team has consistently been available to consult and even collaborate with us on solutions.

Previously, we did a great deal of MS SQL Server stored procedure and SSIS package development. We knew we did not want to manage the development of an enterprise class data warehouse and ETL codebase with handwritten procedures and SSIS packages.

Yes, the installation and configuration was very straight-forward, although we engaged several WhereScape RED service consultants to ensure that it was. I highly recommend leveraging their service offerings to smooth/expedite initial setup.

We used an in-house team, but engaged 2-3 WhereScape service consultants during first year of the initiative.

ROI has been huge. We have very large healthcare data warehouse and the amount of ETL code we have produced and continue to maintain would not be possible without a tool as cost effective as RED.

WhereScape was very fair during pricing and licensing. We were an early adopter of the tool in the US almost 6 years ago, and WhereScape offered very attractive pricing compared to comparable/competitive offerings from other vendors.

Yes, we evaluated Ab Initio and Informatica as well, but chose RED for its ease of use and the portability it offered. Several members of our team had a wealth of Ab Initio experience, and yet still recommended selection of WhereScape RED for our project.

I highly recommend engaging with WhereScape service consultants during adoption of RED. Learn the best practices they promote, and live by them to get the best results.

• Good fit with agile development including requirements discovery and data discovery
• Embedding of standards and design patterns into the tool yields substantial quality and consistency gains
• Schema generation
• Source-to-target mapping automation
• Code/script generation
• Test automation
• Versioning at multiple levels from objects to entire data warehouse
• Scheduling / runtime management functions

• Agility
• Speed of deployment
• Quick response to changes in requirements and changes in source data
• Skilled people no longer burdened with repetitive low-value tasks

• Data discovery would be more powerful with machine learning features
• Big data functions could include some of the features of the advanced self-service data preparation tools

Excellent

Excellent.

Reviewed several other solutions and chose WhereScape because:

(1) It is built by practitioners, for practitioners

(2) It is a "white box" solution. You can see (and even modify) everything that is generated. It doesn't lock you in like the "black box" proprietary products.

Don't confuse data warehouse automation with ETL automation. DWA automates the entire data warehousing life cycle. Change your approach (think agile), change your processes (think end-to-end ... source data to business impact), and change your perspective (think cyclic, not linear). Without these changes you miss out on many of the benefits of automation.