2017-06-28T06:23:00Z
CS
Business Analyst at TekSlate.com
  • 1
  • 968

Python with Numpy/Scipy vs. Pure C++ for Big Data Analysis

Doing Python on exceptionally small tasks makes me admire the dynamically typed nature of this language (no want for assertion code to hold track of types), which regularly makes for a faster and less painful development system alongside the manner. but, I experience that during a whole lot larger tasks this may virtually be a hindrance, because the code might run slower than say, its equal in C++. however then again, the use of Numpy and/or Scipy with Python may additionally get your code to run just as fast as a native C++ software (in which the code in C++ could sometimes take longer to broaden).

I publish this query after reading Justin Peel's touch upon the thread "Is Python faster and lighter than C++?" wherein he states: "also, folks who communicate of Python  being gradual for critical range crunching haven't used the Numpy and Scipy modules. Python  https://goo.gl/Zx5ehp is in reality taking off in medical computing these days. Of course, the velocity comes from the use of modules written in C or libraries written in Fortran, however it really is the beauty of a scripting language in my view." Or as S. Lott writes on the identical thread concerning Python: "...because it manages memory for me, I do not ought to do any reminiscence management, saving hours of chasing down middle leaks." I also inspected a Python/Numpy/C++ associated overall performance query on "Benchmarking (python vs. c++ the use of BLAS) and (numpy)" wherein J.F. Sebastian writes "...there is no difference between C++ and numpy on my machine."

both of those threads were given me to wondering whether or not there is any actual advantage conferred to knowing C++ for a Python programmerhttps://goo.gl/FNhVHm that makes use of Numpy/Scipy for producing software program to analyze 'large facts' in which overall performance is manifestly of top notch significance (but additionally code readability and improvement speed are a need to)?

2
PeerSpot user
2 Answers
it_user158343 - PeerSpot reviewer
Software Architect at a tech consulting company with 51-200 employees
Real User
2017-07-13T12:57:18Z
Jul 13, 2017

Consultant working with Microsoft development tools and Database tools as a software architect for more than 20 years.

Search for a product comparison in Data Integration Tools
it_user158343 - PeerSpot reviewer
Software Architect at a tech consulting company with 51-200 employees
Real User
2017-07-13T12:51:09Z
Jul 13, 2017

From my own experience, your question does not admit a "one size fits all" kind of answer, in fact, the answer should start with the (annoying): it depends.

I will start with describing how I work with the kinds of tools that you mention:

When I have to solve a new problem I start exploring the problem and prototyping the solution with Python and the usual toolset (NumPy, Pandas, SciPy, MathPlotLib, the works).

If the problem is an ongoing concern, instead of a being a "just one shot" nuisance, once I have a stable prototype, but need a more robust and performant solution for the long run, I start a new project to design such a solution with .NET (typically, with C#), either on Windows or on Linux, depending on the circunstances of the problem.

So if you face similar scenarios, that is, you need to have robust and performant solutions for the long run on many of your problems, my suggestion is to start getting friendly with a compiled language like C++ or C#, or some other, and follow an approach in the line of what I do.

Kind regards, GEN

Find out what your peers are saying about Ab Initio, Talend, Microsoft and others in Data Integration Tools. Updated: January 2023.
672,411 professionals have used our research since 2012.
Related Questions
Miriam Tover - PeerSpot reviewer
Service Delivery Manager at PeerSpot
Nov 8, 2022
Hi, We all know it's really hard to get good pricing and cost information. Please share what you can so you can help your peers.
See 1 answer
RV
Tech lead at a tech services company with 51-200 employees
Nov 8, 2022
Co>Operating System's pricing is on the expensive end since it tends to be used by big enterprises.
it_user434868 - PeerSpot reviewer
Senior Director of Delivery at a tech services company with 51-200 employees
Apr 25, 2022
How do you or your organization use this solution? Please share with us so that your peers can learn from your experiences. Thank you!
See 1 answer
AM
Senior Solutions Architect at IDS Comercial
Apr 25, 2022
My primary use of this solution is in the banking sector to process financial movements and generate reports. I also use it in the risk area of banking to detect thefts and risky behaviors.
Related Articles
Anna Odrynska - PeerSpot reviewer
Chief Strategy Officer at Alpha Serve
Oct 11, 2022
Business owners cannot rely on their own market assessments and strategic assumptions to make informed tactical decisions. Rapid technology development and tough competition mean that thorough data analysis is vital for success. In this article, we explain how business analysts, IT teams, and company managers can benefit from Microsoft Power BI integration with the help of BI connectors. Why i...
RG
EPM/BI certified Consultant, Oracle ACE and TeraCorp Consulting CEO at TeraCorp Consulting
Mar 22, 2021
Hey guys, how are you?  In this post, I’ll talk about my preferred tips and tricks that I use in ODI (any version). I’m always using these pieces of code for pretty much everything I do in ODI especially because it makes the code more elegant, dynamic, and easier to change later. First, let’s talk about my all-time preferred, Loops. Normally we learn to loop in ODI using a count variable and ...
Related Categories
Related Articles
Anna Odrynska - PeerSpot reviewer
Chief Strategy Officer at Alpha Serve
Oct 11, 2022
Power BI Integration: Why Should You Integrate Power BI and Your Software?
Business owners cannot rely on their own market assessments and strategic assumptions to make inf...
RG
EPM/BI certified Consultant, Oracle ACE and TeraCorp Consulting CEO at TeraCorp Consulting
Mar 22, 2021
How to use ODI to generate Dynamic code for you
Hey guys, how are you?  In this post, I’ll talk about my preferred tips and tricks that I use in...
Download Free Report
Download our free Data Integration Tools Report and find out what your peers are saying about Ab Initio, Talend, Microsoft, and more! Updated: January 2023.
DOWNLOAD NOW
672,411 professionals have used our research since 2012.