Apache JMeter Reviews, Competitors and Pricing

SridharVangapandu

Sr Manager - Quality Engineering at a computer software company with 501-1,000 employees

Dec 31, 2022

Download

Great performance testing for applications with a valuable load testing feature

Pros and Cons

"We find the load testing feature valuable."

"The reporting section of the solution can be better."

What is our primary use case?

Our primary use case for the solution is performance testing for applications, and we deploy it on the cloud and on-premises.

What is most valuable?

We find the load testing feature valuable.

What needs improvement?

The reporting section of the solution can be better. Additionally, more plugins can be included in the next release.

For how long have I used the solution?

We have been using the solution for approximately 15 years and are currently using version 5.3.

Buyer's Guide

Apache JMeter

April 2024

Free Report: Apache JMeter Reviews and More

Learn what your peers think about Apache JMeter. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.

DOWNLOAD NOW

768,857 professionals have used our research since 2012.

What do I think about the stability of the solution?

The solution is stable.

What do I think about the scalability of the solution?

The solution is scalable, but it all depends on the application. However, it is usually scalable by up to 2500 users. However, if you want to use it for any number of users, you can use JMeter as long as your machine supports it.

How are customer service and support?

We do not have experience with customer service and support because everything is stable.

Which solution did I use previously and why did I switch?

We previously used LoadRunner.

How was the initial setup?

The initial setup is straightforward. I rate it a nine out of ten.

What's my experience with pricing, setup cost, and licensing?

The tool is open source, so we do not pay for licensing.

What other advice do I have?

I rate the solution a nine out of ten. I advise new users to enhance their understanding of the solution via Youtube.

Disclosure: I am a real user, and this review is based on my own experience and opinions.

reviewer891849

Founder and Principal Consultant at a tech services company with 11-50 employees

Feb 24, 2021

Download

Priced well, stable, and integrates with other tools, but the installation is not intuitive

Pros and Cons

"I like the fact that JMeter integrates well with other tools."

"The installation needs some work. It could be simplified."

What is our primary use case?

I was experimenting with Apache JMeter. I was doing a proof of concept for myself to see if it fit my needs. I wanted to compare it with LoadRunner to see how it compares in terms of recording, ease of use, and functionality.

What is most valuable?

I like the fact that JMeter integrates well with other tools. For instance, we are also looking at OctoPerf and, I know for OctoPerf, you can import JMeter scripts, whereas, for LoadRunner, you cannot import LoadRunner scripts into OctoPerf.

I like the graphing; it is good.

Once I got past the learning curve, it was okay.

There were a lot of features that translated well from LoadRunner. I have LoadRunner experience, so once I applied my base knowledge to JMeter, then it was okay. The terminology was a little different, but the functionality is similar. For instance, They are called "extractors," in JMeter and called "correlation" in LoadRunner.

Once I learned the terminology and how to navigate through JMeter, then it was easier.

I still think LoadRunner is a better tool, but for some purposes, JMeter would suit the need.

What needs improvement?

The installation needs some work. It could be simplified.

When compared with LoadRunner, LoadRunner is a more mature product.

For how long have I used the solution?

I used it for a couple of weeks.

What do I think about the stability of the solution?

It's pretty stable, I don't have any complaints.

What do I think about the scalability of the solution?

We didn't do a large load test. I didn't explore the scalability.

I was the only person using this solution, but I ran 100 user load tests.

How are customer service and technical support?

I did not need to use technical support.

Which solution did I use previously and why did I switch?

Previously, we were testing LoadRunner.

How was the initial setup?

I reviewed some YouTube videos and went through them. It wasn't necessarily intuitive.

There was some configuration, and file that had to be set up.

It wasn't like LoadRunner where they have an automatic installer and wizard that does everything for you.

It wasn't as easy, but anyone who is technical can work through it.

What's my experience with pricing, setup cost, and licensing?

When comparing the price with LoadRunner, and if the cost is an issue then JMeter is a better choice. LoadRunner is very expensive.

Which other solutions did I evaluate?

I like LoadRunner better, although I might be biased because I do have LoadRunner experience. I am a little more familiar with it. I was leaning towards LoadRunner.

What other advice do I have?

I would advise others to look at YouTube videos for the setup. Other than that, it was pretty straightforward. I was pretty much satisfied with it. It's a good tool, and it sounds like it's been around for a while.

I would rate Apache JMeter a seven out of ten.

Which deployment model are you using for this solution?

On-premises

Disclosure: I am a real user, and this review is based on my own experience and opinions.

Buyer's Guide

Apache JMeter

April 2024

Free Report: Apache JMeter Reviews and More

Learn what your peers think about Apache JMeter. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.

DOWNLOAD NOW

768,857 professionals have used our research since 2012.

reviewer1179222

Executive Director/Consultant at a tech vendor with 1,001-5,000 employees

Oct 8, 2019

Download

Cost-effective and has good community support

Pros and Cons

"The most valuable feature for us is the available information on the forums and to be able to discuss and get answers from the people that are involved in using this tool."

"There is some work to be done with the integration."

What is our primary use case?

In general, the primary use case of this solution is for performance testing and load testing.

Typically it is for load tests. They have to focus on three metrics including throughput, response times, and the utilization of the resources including the CPUs.

We are delivering some bank houses with performance tests, through JMeter.

The deployment model varies depending on the client. Some have a private line.

What is most valuable?

From the customers' perspective, the most valuable feature is the price.

The most valuable feature for us is the available information on the forums and to be able to discuss and get answers from the people that are involved in using this tool.

What needs improvement?

They have to find a way to prepare the script or to prepare a detailed analysis. We have to collect all the information on each of the services we have to call. Based on this they have to collect in the phase of preparation for the performance test and then we can run our audit. It is easy to prepare a script to run a performance test.

You can't rely on the support. It's something that is not fully working.

The scalability of this solution needs some improvement.

There is some work to be done with the integration.

For how long have I used the solution?

I have been using this solution for over five years.

What do I think about the stability of the solution?

This solution is stable, but it does have one limitation with the number of throughputs. For example, if I want to generate four, five, or two thousand people working concurrently then they have to prepare our setup for ten PC servers.

What do I think about the scalability of the solution?

Normally they can run a load for five thousand people, customers, or users working concurrently. It is scalable but needs some improvement.

How are customer service and technical support?

The technical support on the site is not fully functioning.

How was the initial setup?

The initial setup is easy and straightforward.

You can have it running within thirty minutes.

What about the implementation team?

I had always implemented this solution myself, but It's been a couple of years since I did it last.

Now it is done through our company technicians. I just coordinate activity-performance tests for our customers.

What's my experience with pricing, setup cost, and licensing?

This is an open-source solution, and there are no fees.

What other advice do I have?

The requirements for most of our customers is to provide them with cheap solutions, and most prefer not to pay for software. This is the right solution for them based on some of the open-source tools that are available.

There are some commercial solutions that provide better integration to the solutions that must be tested when running loads.

I would rate this solution an eight out ten.

Disclosure: I am a real user, and this review is based on my own experience and opinions.

reviewer1191885

Director with 201-500 employees

Jan 27, 2022

Download

Fast, user-friendly open-source testing tool

Pros and Cons

"It is very quick and user-friendly."

"If the solution was GUI based, I believe that it would be more versatile."

What is our primary use case?

I implement this solution for my clients.

What is most valuable?

It is very quick and user-friendly.

What needs improvement?

If the solution was GUI based, I believe that it would be more versatile.

For how long have I used the solution?

I have been using this solution anywhere from six to eight years.

What do I think about the stability of the solution?

The solution is stable, I would definitely recommend it to those who are considering it.

What do I think about the scalability of the solution?

The solution is scalable.

What's my experience with pricing, setup cost, and licensing?

No licensing is required as it is a free, open-source tool.

Which deployment model are you using for this solution?

Hybrid Cloud

If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?

Amazon Web Services (AWS)

Disclosure: I am a real user, and this review is based on my own experience and opinions.

it_user403875

Consultant at a tech services company with 10,001+ employees

Mar 27, 2016

Download

It's very easy to download and setup the entire infrastructure.

Valuable Features:

Extremely light weight.

Very easy to download and setup the entire infrastructure (Controller +LGs)

Ease of scripting

Host of plugins which further boost its effectiveness

Availability Non-GUI mode which consumes even lesser resources than already lightweight GUI version.

Improvements to My Organization:

Robust scripts which could handle build changes

Easy LG(Master & Slaves) configuration setup

Very lightweight and low usage of resources

Room for Improvement:

GUI tends to freeze and shutdown under more load

Ctrl+z( Undo doesn't work) so gets very inconvenient at times

Cannot do a mass replace( Ctrl+H) on the GUI ( can do it if script is opened via notepad++)

Re-iteration problems during errors - Usually we have login in once only controller, Action in Loop controller and logout in once only controller. Now if an error comes when the user simply logouts due to some unhandled error then the script expects that your session would continue but if that error has made you logout then your next iterations won't go through and vice versa. Suppose you put everything in loop controller and error doesn't log you out then the script start to login when you are already logged in and tends to fail.

Deployment Issues:

We've had no issues with deployment.

Stability Issues:

See above Room for Improvement.

Scalability Issues:

We've had no issues with scalability.

Disclosure: I am a real user, and this review is based on my own experience and opinions.

it_user4032CTO with 11-50 employees

Report as inappropriate

Feb 2, 2017

Hello Kapil,
Thank you for your review.

Here are my comments on what you write:
- GUI tends to freeze and shutdown under more load => Load testing with GUI is a known ANTI-PATTERN. GUI is to create/record/correlate, NON GUI is for Load Testing. Since 3.1 a WARNING appears on startup.

- Ctrl+z( Undo doesn't work) so gets very inconvenient at times => True, we'll try to find a solution to this

- Cannot do a mass replace( Ctrl+H) on the GUI ( can do it if script is opened via notepad++) => With upcoming 3.2, you'll be happy to see this feature

- I don't understand the last item, so if you're willing to give more details maybe we can help on this, see jmeter.apache.org

Thank you
Regards
Philippe M. from Apache JMeter Team

See all 2 comments

it_user281952

Sr. Performance Test Engineer at a university with 501-1,000 employees

Jul 30, 2015

Download

It can record, run, and create reports, but it hangs up on large loads.

Valuable Features:

I like the ease of using JMeter. It can be downloaded and can be used to record and run load tests. JMeter can record, run and create reports easily without the need to have licenses, complex installations and paid support. It can be used to performance test services, web applications, database queries, LDAP, etc.

Improvements to My Organization:

We use JMeter for performance testing. We are able to know which module is lagging behind in performance, whether the software or hardware upgrades have improved the performance or not.

Room for Improvement:

I think JMeter hangs up on large loads. I have seen JMeter becoming unresponsive in the middle of a test. It is because of garbage collection in JMeter. We use best practices to mute all listeners, use non GUI mode, etc., but even with many best practices JMeter pauses in between load tests.

Cost and Licensing Advice:

It is a open source tool.

Other Advice:

My advice would be to have load generator machines. Do not run JMeter (or any load testing tool) from a desktop. Do not have just one JMeter installation to generate load. Use several JMeter instances to distribute load. If possible, use JMeter in Client Server mode.

Disclosure: I am a real user, and this review is based on my own experience and opinions.

it_user4032CTO with 11-50 employees

Report as inappropriate

Feb 2, 2017

Hello Rahul,
If in 2017 using JMeter 3.1 you still experience "hangs up on large loads." as you wrote here then it would be really nice to report this issue to JMeter team:
- jmeter.apache.org

From my experience with it, I am able since JMeter 2.11 at least to load test with a single JVM on a 8 Core machine with 6g of memory to load test without any issue up to 2000/2500 Threads with a correctly developed test plan. And whenever I switch to distributed load testing run load test on big E-Commerce website with Millions of users per day.

So I am very interested to know what kind of issue you face so that we can either help you to fix your test plan or fix the issue in JMeter if it is there.

Thank you
Philippe M. from Apache JMeter Team and big user of Apache JMeter

it_user69342

Consultant at a consultancy with 501-1,000 employees

Jul 24, 2015

3 Reasons for combining JMeter and WebDriver

The Apache Foundation states the following about JMeter:

The Apache JMeter™ desktop application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

So in other words, it’s a tool you can use to generate functional load on an application or a platform. This also immediately describes what it is good at: generating load. Yes if implemented well you not only generate dumb-load but also hit the functional application layers with the tool. But the basic function of JMeter is aimed at generating load and measuring the (server) response times during this load.

What does JMeter NOT do?

Despite being capable of generating functional load, JMeter does not render pages nor is it very well equipped to execute embedded JavaScript (it is simply not equipped to do that actually), therefore JMeter will not tell you anything (out of the box) about the render times of pages. Especially not about render times when the server is heavily overloaded by your scripts!

What is WebDriver good at?

SeleniumHQ gives a wonderful description of what Selenium (or nowadays WebDriver) does:
Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that.

In short, what WebDriver does is just about anything that happens within your browser. It does render pages, it does execute JavaScript, it retrieves the pages as if an actual human was clicking on a website. So for fully functional automated testing (or checking to stick with the more correct terminology) WebDriver is perfect.

What does WebDriver NOT do?

Well, it is not quite good at generating load. Since WebDriver basically requires a browser (yes, it is possible to run it headless of course) it is very difficult to generate multiple (virtual) users. That would require a bunch of browsers to start up, when talking about 10 users that may seem feasible, however when talking about generating real load (say several 1000’s concurrent users) a bunch of browsers becomes a lot more difficult to arrange.

Why combine them then?

The logical question then indeed is, why would you combine them? Below I have set out 3 clear reasons why combining JMeter and WebDriver scripts can be an excellent idea.

Impact of server-side load on render-times;
When the load on a server increases, the response times of various parts of a web application may increase as well. These increased response times can have implications on the render time of the web application. For example: a web application heavy with Ajax requests is put under load, the server response times increase, this may result in all Ajax-requests becoming slower, therefore making the website extremely unfriendly to the end-user. When you just run a JMeter script, this will hardly be noticed, and if you do notice it, you cannot express the impact it will have on the user. You can merely speculate about it.
Impact of server-side load on functional behavior;
Given that the server is experiencing increased load and therefore the business-logic of the application is working hard to handle all requests effectively, it can be safe to say the underlying database may also be stressed and therefore responding slower than expected. Slower response times of both application-logic and database requests can result in buggy behaviour of the application. For example incomplete data returned, or worse, a time-out on data or application-logic. How does the application deal with that? How are these errors reported to the end-user? Will the application still function normally within the browser when certain aspects of the application platform are malfunctioning? The best answer to this is by testing the functionality thoroughly while the application is under load. An easy way to test this repeatedly and consistently is by automating these functional tests, for example using (part of) the automated regression test while the servers are under increased load.
Advantage of screenshots of fully rendered pages and possible errors with the application under load;
As a result of the two points mentioned above, it may be extremely useful for both developers/system engineers and your customer to see errors on the pages affected by the increased server load, such as stylesheets not loading or not loading properly, JavaScript not loading, images missing etc. Screenshots (or screen captures in movie format) will help make clear to the customer what the problem is and more importantly how big the impact on the end-user will be.

I have listed 3 reasons why combining JMeter and WebDriver can be a good idea. I’d love to hear your suggestions of more reasons to want to combine the two.

In a follow-up post I will go into more detail on ways to achieve an effective combination of JMeter and WebDriver running along side each other, well timed and generating consistent logging and results.

Disclosure: I am a real user, and this review is based on my own experience and opinions.

it_user69342Consultant at a consultancy with 501-1,000 employees

Report as inappropriate

Apr 12, 2015

Hey, Peter-Thomas,

The easiest way I can suggest it to just start small: ensure you add timers to your existing webdriver scripts so you know what your application does under normal circumstances.
Once you have the functional side covered in a way that you know what you need to know, start building up a load generator in Jmeter and start running the two alongside one another manually. When you have that working properly and are getting some useful numbers drop the whole thing as a script into your CI environment for running after a full regressiontest.

See all 4 comments

it_user69342

Consultant at a consultancy with 501-1,000 employees

Feb 25, 2015

JMeter is great as a stand-alone load generator, however when you need a distributed load you'll require high server use

Over the past year I have spent a lot of time in close collaboration with Apache JMeter. I used JMeter as a standalone platform, in collaboration with several tools out there and with my own servers. Over the next few posts I will elaborate a bit more on what I have learned from working with tools like BlazeMeter, Loadosophia as well as working with your own remote servers and Jmeter. On top of that I will share some things I consider good practices to use when working with JMeter and building elaborate test scenarios in it.

JMeter and remote servers

Meter is great as a stand-alone load generator, however the moment you need to generate a substantial load or a distributed load, you will require the use of more servers. There are several possibilities to add external servers to JMeter. You can use a load generation platform such as BlazeMeter and Flood.io or you can choose to use your own machines (Disclaimer: I have only worked with Flood and BlazeMeter as services, so I only mention these two. There may very well be other services out there I am not (yet) aware of).

Why use your own remote servers?

Although there are several commercial options to run JMeter tests from a cloud-platform, it might be useful to have your own servers running. Your own machines, for example Virtual Private Servers, will give you at least the following advantages, please bear in mind these are the main reasons for me to run private servers. It may very well be that you have different reasons for doing the same (I’d be happy to hear them btw!):

Availability within firewalls: Not all applications are available outside of firewalls. For quite a lot of companies it is not an option to allow external addresses, especially cloud services, within their firewalled environment. When this is the case in your organisation it can be very useful to have your own load generator hosted inside the firewall.
Geographical location: not each and every country has platforms available for load generation. Although platforms like Blazemeter have a lot of different locations available, it may very well be that your particular country is not available through a service provider. However getting your own, hosted, machines within a country is generally not too difficult to do. Keep in mind though whenever you want to use something like a Virtual Private Server to verify with the hosting company whether they allow load generation from within their network!
Control: a fairly straightforward reason is in order for you to have full, unlimited control over the load generating servers. If you have your own (virtual) servers running you can easily adjust settings, properties and other things in order to make it fully match your (customers) needs.

Why use a service?

Even though there are quite some reasons for using your own machines, the advantages of using a service like BlazeMeter or Flood.io are plenty, I will just highlight the few that for me have tipped the scales several times in favor of a service.

Maintenance: Setting up your own servers is time consuming and therefore not cheap. These servers need maintenance even though you quite likely do not constantly need them. A Service is just there. You buy a subscription, be it with a time constraint (monthly) or a load constraint (max amount of concurrent users) or something like that, but the service is simply at your fingertips. The moment you need it, you have it.
Multiple locations: the JMeter based services generally make use of the Amazon AWS cloud, thus giving you, the user, an immense amount of servers at your disposal. With this comes a huge worldwide distribution, which can be very advantageous for big world-wide used applications.
Support: both mentioned platforms have a very solid support base for their customers. If you have questions, if your tests seem to not run properly or you simply have issues getting something done, there generally is a good support platform available. You can use forums or the actual helpdesk, but either way, there is a good, solid, commercial support-base to help resolve your issues.

In the end, I choose per assignment what best fits my needs. Sometimes I use the VPS, sometimes I use a service. It is important you at least think about what best suits your needs for the test you are about to build and execute.

Disclosure: I am a real user, and this review is based on my own experience and opinions.

it_user278154Works

Report as inappropriate

Jul 24, 2015

Hi Martin,

Thanks for your input, very useful to know. I am working as a consultant on a relatively new client site, and am preparing to head up the team to write an RFP proposal for a new performance testing tool to be used across the company.

Getting all the technical / protocol details for the various systems is proving to be more challening than it should - they seem to be all Web based (although only support old versions of IE like 8 and 9... I know). I am very tempted to strongly recommend JMeter as the tool (probably with BlazeMeter or Flood as the cloud based service), but I am concerned that a couple of months down the line some legacy system will come out of the woodwork that JMeter would not be able to support.

Hope that all makes sense. Would be interested in any other feedback that you had.

Cheers,

James

See all 5 comments