2020-04-18T11:55:00Z
Bruce Tan - PeerSpot reviewer
Director at SLS Bearings (S) Pte. Ltd.
  • 8
  • 226

Is a data warehouse the best option to consolidate data into one location?

I am a director at a business enhancement solutions company that provides mechanical systems and distributes high-quality bearings and related products.

I am planning for b2b e-commerce. I have operations in Singapore, Malaysia, Indonesia, China, and Mongolia. Each country uses a different ERP. Singapore HQ uses JDE (one world), that was last implemented 20 years ago.

I intend to consolidate the data to one 'location' for the integration of data into e-commerce or e-marketplace. Is a data warehouse the best way to go? 

Thanks! I appreciate your help. 

9
PeerSpot user
9 Answers
DR
President at a computer software company with 11-50 employees
Real User
Top 20
2021-08-05T13:39:59Z
Aug 5, 2021

For most clients, we suggest a data pipeline that includes an inexpensive storage step along the way such as AWS S3 or AZURE Block Store. 


Once the data is landed at rest it can then be loaded into any EDW or CDW (Snowflake) as needed based on your consumer persona needs. 


In addition, depending on timeliness and other requirements, you can retrieve the data directly from S3 storage for example using Athena without having to move it to any other platform. 


Of course, like with anything else the devil is in the details. Each platform/approach should consider how you intend to costume and utilize the underlying data so that performance and costs are projected into your decision process.

Search for a product comparison in Data Warehouse
RW
CEO at WInterCorp LLC
User
2021-08-05T18:01:16Z
Aug 5, 2021

The answer is yes.   


If this platform is supporting e-commerce, distribution or manufacturing then a critical component of the data warehouse may be the ability to support tactical or operational queries on data that is updated throughout the day. This considerably narrows the choices.    


I teach a course on how to select a cloud data warehouse platform for modern requirements. You can watch last December's version online at https://tdwi.org/Events/Confer...  also, I am speaking on the subject at a TDWI event next week https://tdwi.org/events/virtua...  the course (updated) will be offered live at a TDWI conference in November (virtual + live in Orlando).   


Or book me for a private conversation at https://tinyurl.com/bookrw30

BL
Certified Adjunct Faculty, School of Engineering and Computing at a university with 1,001-5,000 employees
Real User
Top 5
2021-08-05T19:12:54Z
Aug 5, 2021

A definition of a data warehouse was been for years anchored in retaining data that is no longer changing. A B2B e-commerce environment includes many dynamics of data while being volatile, with factors that influence the frequency of queries and updates. eCommerce is anchored in dynamic operational data for applications that support both internal and external visitors. 


The ultimate purpose of a traditional enterprise data warehouse (guided by a "top-down" methodology) is a that and EDW imports operational records after achieving a state of being complete.  


I would call your target environment a data repository instead. Even so, there may be many elements from your original source databases that are not needed and are no longer relevant to being integrated together into a single location. 


I would proceed only after answering questions about: 


1. What outcomes you are seeking and who will be the users who believe that all data needs to be integrated into a single location? and 


2. Consequences on software that is now processing source systems databases that have different purposes and maybe to persist. 

BL
Certified Adjunct Faculty, School of Engineering and Computing at a university with 1,001-5,000 employees
Real User
Top 5
Aug 10, 2021

@Jay Allen 
Jay, Look again at sources for your own understanding. My understanding of top down data warehouse for an enterprise has an origin that has been trusted for decades. "Subject-oriented, non-volatile, integrated..." (crediting Bill Inmon). As a practitioner, I do admit that in my years a data administrator, I have encouraged a bottom up approach which imports recently historical data from operational sources into data mart(s) designed to be fact based with descriptive dimensions ... (credit to Ralph Kimball).  
With regard to B2B, partners in data exchange can link to either to operational database sources which are dynamic, or to data stores into which no-longer-volatile source data have been imported. -- Barbara

PeerSpot user
Jay Allen - PeerSpot reviewer
Information Systems and Database Architect at Relational Models LLC
User
2021-08-05T17:08:54Z
Aug 5, 2021

Your question is one about architecture, but I see some answers about the technology. 


Choosing technologies is just as critical, but to your question, @David Rossi ​has it right. It sounds like a straightforward DW requirement. With the multiple source systems and potential data integrity issues, it also seems like a top-down DW requirement. Your operations seem like they may be complex. So you may be facing requirements that are a bigger question than the one DW answers. 


Also, take a look at Corporate Information Factory (CIF) Architecture and its evolution to see if this more specific architecture offshoot of DW addresses your particular complexities.


https://www.sciencedirect.com/...

Carlo Arioli - PeerSpot reviewer
Senior Solution Marketing Manager at Salesforce
Real User
2021-08-05T17:03:37Z
Aug 5, 2021

IMHB ... yes and no, i.e. you should really clarify first which use case you have in mind in sharing the data - especially if it comes to analytics, versus sharing data for integrated processes towards customers (i.e. yous service desk vs your eCommerce vs...)


The "yes" piece is that an analytical centralized DWH is for sure a must for the architecture to have the ability of advanced reporting, cross-analysis, etc. In doing so, I'm biased from my past, but as many mentioned, a good analytical modern data warehouse (better if columnar) scalable with the ability to connect to lower-cost storage (S3, Azure blob...) options, leaving the data there, would be terrific. Loading there all, with historical data is for sure good. Someone mentioned Snowflake, I was a Vertica champion... your call. 


The "no" part is you might still need a more transactional integration layer to connect your different applications from legacy CRM, service desk, e-commerce, marketing. Depending on what you have here you have few different options which would then synch and load data in the analytical data warehouse.

Shahrukh Dastur - PeerSpot reviewer
General Manager-Business Analytics and Decison Systems at a retailer with 10,001+ employees
Real User
2021-08-04T10:16:49Z
Aug 4, 2021

The answer is a YES. 


We have almost the same scenario with various domains of business using different systems from instore to eCommerce. And finally, JDE is used for accounting. 


We designed a central data warehouse with a strong transformation engine for the past 10 years and everything works like a magic. Any new integrations, systems, countries and even new business lines consolidate in the central data warehouse, making it a cake walk for reporting and analytics. Also, the data warehouse has become a single version of truth providing tremendous power to business to use the data for financial and business integrations and monetizations.  


Just a piece of advice, when you plan for the business data warehouse, be it on cloud or premise, go for a less technical complex architecture one. This will allow you to focus on business and speedy integrations. More tech teams and more tech products for integration may look interesting and futuristic, but take care and do a proper analysis before you take the next step. All the best.  

Learn what your peers think about Oracle Exadata. Get advice and tips from experienced pros sharing their opinions. Updated: November 2022.
656,862 professionals have used our research since 2012.
Okay Akdeniz - PeerSpot reviewer
Business Application Group Manager at OBASE
Real User
Top 5Leaderboard
2021-08-04T13:51:19Z
Aug 4, 2021

Nope, the Data warehouse approach is analyze and decision platform for business. 


You need a central data mart platform to provide data to 3rd party connectors. Sometimes you need near-realtime data. Data Warehouse platforms commonly present T-1 data. Also, in the future, lots of 3rd party companies want to access this data and you don't want to provide access to your analytics platform.

DR
President at a computer software company with 11-50 employees
Real User
Top 20
2020-04-23T14:58:38Z
Apr 23, 2020

The idea of an EDW or enterprise data warehouse is the right path. That said, it's all about choosing the appropriate set of technology platforms and tools. Bringing together data from disparate platforms (you mentioned multiple ERP implementations) is not a trivial task and the choices you make for your EDW platform and the related data management tool will greatly affect your timeline, costs, and maintenance. Best of luck let me know if there is anything else we can do. Dave

RamakrishnanRamanathaiah - PeerSpot reviewer
Vice President at a tech services company with 1,001-5,000 employees
Real User
2020-04-22T22:31:20Z
Apr 22, 2020

It depends on whom do you ask the question. I personally say, Yes, Absolutely required. If you wanted to combine the data from multiple systems. For sure it is definitely required.

Regards
Ram

Related Questions
Netanya Carmi - PeerSpot reviewer
Content Manager at PeerSpot (formerly IT Central Station)
May 17, 2022
Why would you choose that one?
See 1 answer
Dovid Gelber - PeerSpot reviewer
Tech blogger
May 17, 2022
When I compared various data warehouse tools and solutions, I found Snowflake’s software as a service (SaaS) platform and Oracle Exadata to be the most effective data warehouse solutions currently available on the market. One of the things that I initially noticed about Snowflake’s software as a service (SaaS) platform was how it made my operations more efficient by enabling me to search for and find relevant data in a more efficient way than had previously been possible. Snowflake allows me to create a custom storage unit for all of my critical data. Part of this customization includes the ability to make any and all data searchable. As soon as I started to use it, it began to show its value. All of the data that I have stored in Snowflake’s virtual warehouse becomes easy for me to locate. Instead of spending long periods of time seeking the particular piece of data that I need, all I have to do is to type in a search term. This will immediately call up the information that I want to find. This aspect of the solution makes it immensely valuable. It enables me to save time that I can then devote to other critical tasks. A major advantage that Snowflake offers me is that it gives me the ability to perform a number of different functions with a single solution. It is a highly flexible solution and allows me to store organized processed data, centrally store raw data that has yet to be processed, process data through data engineering, examine data using data science, develop data applications, and securely share and take in real-time or shared data. It empowers me to make my data really and truly my own. I am able to make full use of my data and shape it in whatever way my needs dictate. Two aspects of the Oracle Exadata Database Machine that I really appreciate are its scalability and its ease of use. This solution makes it so that I can expand my digital warehouses virtually limitlessly. If I needed to, the Oracle Exadata Database Machine would enable me to scale my data warehouses to hold 31 petabytes of data. I can easily meet my data needs without having to worry about running out space for my data. Additionally, every component of this tool, including its database servers, storage servers, and network, are all ready to use straight out of the box. Everything about this solution is pre-configured, pre-tuned and pre-tested before we ever received it from Oracle. All of the components work in perfect harmony without outside intervention. This means that I don’t have to struggle to deploy it or do very much to get all of the features to work in tandem. This solution also offers me the ability to easily move workloads from a data center to the cloud. I am able to migrate my data without having to worry about availability, scaling, or performance. My workloads lose nothing in the transfer. Additionally, the same database options that are available to me on my physical systems are available on the cloud. This allows me to continue using the solution the same way that I had been using it up to this point. Ultimately, either of these two solutions will empower you to take full control of every stage of your data and its lifecycle from its initial storage to its final use.
Evgeny Belenky - PeerSpot reviewer
Director of Community at PeerSpot (formerly IT Central Station)
Sep 9, 2022
What are the relations between them? What are their use cases?
2 out of 7 answers
Ariful Mondal - PeerSpot reviewer
Consulting Practice Partner - Data, Analytics & AI at FH
Oct 10, 2021
Hi @Evgeny Belenky ​ - great question.  Here is the best answer crafted by Talend  Data Lake Data Warehouse Data Structure Raw Processed Purpose of Data Not Yet Determined Currently In Use Users Data Scientists Business Professionals Accessibility Highly accessible and quick to update More complicated and costly to make change Please read more here https://www.talend.com/resourc...
RW
CEO at WInterCorp LLC
Oct 11, 2021
Many of the comparisons of data lake and data warehouse that you see (such as the one below from Talend) are based on an out-of-date or dumbed-down idea of the data warehouse.   The more advanced data warehouse engines: - support a wide range of data types and formats - can access external data (e.g., in object storage) that has never been ingested - support data scientists as well as business users (e.g., with an ability to run Python, R, SAS routines and data science libraries on data in place in parallel in the data warehouse) - support operational query on live, rapidly changing data While also providing capabilities and services never provided on data lakes or their cloud-based equivalents.  Data warehouses, properly operated and housing data that is properly curated, are much more efficient, cost-effective and performant for data that is intensively shared and widely used. Data lakes are good repositories for data that is more lightly or locally used and does not merit the level of curation usually desired in a data warehouse.
Related Articles
Netanya Carmi - PeerSpot reviewer
Content Manager at PeerSpot (formerly IT Central Station)
Apr 26, 2022
PeerSpot’s crowdsourced user review platform helps technology decision-makers around the world to better connect with peers and other independent experts who provide advice without vendor bias. Our users have ranked these solutions according to their valuable features, and discuss which features they like most and why. You can read user reviews for the Top 5 Data Warehouse Tools to help you d...
Moderator
Ricky Santos - PeerSpot reviewer
System Administrator at ON Semiconductor Phils. Inc.
Real User
Top 10
Related Categories
Related Articles
Netanya Carmi - PeerSpot reviewer
Content Manager at PeerSpot (formerly IT Central Station)
Apr 26, 2022
Top 5 Data Warehouse Tools 2022
PeerSpot’s crowdsourced user review platform helps technology decision-makers around the world to...
Download Free Report
Download our free Oracle Exadata Report and get advice and tips from experienced pros sharing their opinions. Updated: November 2022.
DOWNLOAD NOW
656,862 professionals have used our research since 2012.