Arize AI Reviews

I have been using Arize AI for around two to two and a half years. We created an agent using the Google ADK, which is the Agent Development Kit, and we use Arize AI for the observability and monitoring or evaluations of that GenAI agent that we have created.

When we deployed our agent in the agent engine, we needed to send all the logs and spans, or all the conversation to Arize AI, which is an AI observability tool. When I ask a question to my agent, for example, "What were the sales in the last year?", it sends this question and the answer to Arize AI in the logs and all the tools or functions that my agent has called. We can track the model behavior over time, monitor how our agent is working, and identify any anomalies. For each conversation, it sends all the logs and traces to Arize AI. We have a dashboard in Arize AI where we can check each conversation. If I asked a question, I can see how that question is being answered by the agent, what functions it has invoked, what tools it has invoked, and what sub-agents the main agent has invoked. We can see everything, every step in Arize AI with detailed information, such as the input for a function, the output for the function, and that this function took one millisecond. We can see the whole logs in Arize AI.

We have some evals as well. Testing of an AI agent is a major concern in the market. We have the evals in Arize AI itself. We can have our own evals or evaluations. We can write that if this is the input, this should be the output. It matches semantically to whether the output is correct or not. One more use case is hallucination detection. One of the major problems with Arize AI and agents is that they hallucinate over time or when the RAG is too huge, they start hallucinating. Arize AI is useful to check whether our agent is hallucinating or not.

The major feature is observability. We can see how our agent is behaving over time. We can monitor the agent and we can have alerts as well. If the latency is going up to a threshold greater than any limit, it generates the alert. If any unexpected agent behavior is there, then it can also have custom alerts. We can have our own monitors in the dashboard in Arize AI. Apart from this, we can see the whole breakdown of the entire flow. This was a user prompt, this is a document that it has got from the RAG itself, this is the model response, these are the tools that it has called. A whole workflow of an agent conversation is visible in Arize AI. One more feature is hallucination detection. We can check whether our agent is hallucinating or not. These are some of the major features.

One of the major improvements is that prior to using Arize AI, our agent was hallucinating and we were not aware of when it hallucinates or we had a problem in debugging. We did not see the whole flow or which tool is calling, what is the input for this tool, and what is the output for this tool. After using Arize AI, we got the alerts whether there is some discrepancy or if it starts hallucinating itself.

The time savings are significant. When an issue comes, prior to this, we needed to go in the console and check for each of the traces and find those, and those traces were not in detail. It saves around 40% of our time while doing root cause analysis of an issue.

Observability and the detailed breakdown of the whole flow are what I rely on the most.

There are some more features that I have not used, but I have read about those. RAG evals and monitoring show how our RAG is behaving, what is the RAG accuracy, and what is the context coverage of the RAGs. These are some other features that I have not used, but I have read about those.

I think everything is there to be true. I do not think there is a scope for improvement in Arize AI. Everything is there.

It has a steep learning curve. It takes time to see how Arize works. It is not a very basic thing where anyone can go and start doing it because it takes time. There is a steep learning curve for Arize AI. Because there are so many things in the model or in an agent, it takes time. It is not very easy to use, it takes time. It has a lot of advantages, but it takes time to learn how Arize works.

As I mentioned earlier, it has a steep learning curve. It takes time to learn Arize AI, it takes time to configure, it takes time to create dashboards and monitors, and it takes time to understand the UI and determine what can I find where. It takes time to do all of that. It has a steep learning curve.

I have been working in the current field for around eight years.

I do not think so. When I ask the agent, it automatically sends all the logs in an asynchronous way to Arize AI. There was no downtime or latency that I felt at that time.

It was able to handle larger data sets. We provided a very large data set for the evals and it was able to do everything. It was able to process the evals and everything. I am satisfied with the scalability of Arize AI.

They were quite helpful. Arize AI provides the Python SDK that we have used and it is quite helpful and very easy to configure as well.

I was facing a firewall issue because it was an on-premise deployment. I approached them and they were quite helpful. They responded the same day and solved my issue. I was missing a small thing, so they suggested using a specific link. They provided me a documentation URL and it worked.

It was smooth.

Go ahead and use that. It provides a lot of observability capabilities that will help a lot while creating any agent or training any model. It is very useful. I would rate this product a nine out of ten.

We have been using Arize AI for more than three years.

We use Arize AI for observability and monitoring of our number of machine learning models which are being deployed in our system.

We are using Arize AI for monitoring OCR plus document extraction quality. HireRight processes IDs, payslips, bank statements, education certificates, and other documents, where the models extract names, dates, employment periods, university names, and other details. We utilize the model we have created for extraction accuracy drift, identifying and monitoring OCR quality degradation, getting field level confidence, monitoring hallucinated values, assessing model regressions, and recognizing vendor-specific failure patterns.

We use Arize AI for a variety of our use cases mainly to detect model drift and track key metrics such as precision, recall, and F1 score to determine whether the model is behaving in the right manner or not.

One of our models for the multimodal verification solution experienced drift, and we promptly saw the trends in Arize AI, which allowed us to tweak and fine-tune our model based on new information available, thus helping in reporting false positives and saving us from penalties.

Arize AI offers one of the most complete observability solutions for enterprises, providing model drift detection, embedding drift analysis, hallucination monitoring, trace analytics, latency and token monitoring, root cause analysis, and agent execution tracing. It has adopted one of the open-source frameworks, facilitating open telemetry alignment, easy traceability, and prompt inspection, while its visualization layer is quite intuitive, especially trace trees, agent execution graphs, and embedding clusters, which really helps.

The visualization layer is one of the best features because it gives an overall understanding of how the models are behaving without getting into the details. We can see the trends in the charts, especially the agent graph capability to trace back which agent went wrong, providing a high-level view of its performance and key strengths.

Arize AI has strong enterprise credibility, with a focus on compliance and governance for large-scale monitoring, and I have generally seen many regulated industries using Arize AI, which I believe is on the right path.

Arize AI has positively impacted HireRight, particularly because, being a regulated industry, it is vital that our models are working correctly, as any drift or false results can lead to significant penalties. It has helped us monitor key metrics, understand accuracy drift, and assess field level confidence, providing explainability, tracing decision lineage, audit logs, model output retention, and bias monitoring, which helps us get more out of the process. It aids in identifying which types of documents are failing, regions creating maximum exceptions, which models are triggering the most human reviews, and what confidence threshold we should set while tuning those models, making it invaluable for our daily operations.

The evaluation workflow lacks depth in comparison to competitors, which generally rely on traditional ML frameworks. Arize AI is stronger in observability but weaker in experimentation, simulation, CI/CD gating, and benchmark management. Competitors such as BrainTrust and Maxim AI focus much more on evaluation-first workflows. If these aspects are addressed, Arize AI, which already has enterprise credibility, could capture a larger market share. Additionally, the setup can sometimes be too complex for smaller teams, particularly regarding telemetry ingestion, making it feel heavy compared to solutions such as Helicone, Langfuse, or LangSmith. Creating a starter or limited functionality dashboard for those teams could help Arize AI penetrate that market segment.

Improvements can be made concerning the cost factor and the evaluation workflows to make them competitive with other options, which would further strengthen Arize AI's market share.

Pricing can sometimes be on the higher side, particularly if we are tracing telemetry or logs. The setup cost is generally a one-time expense; we have acquired a couple of licenses specifically for the AI/ML team to monitor our in-house AI/ML models because teams find it useful. Debugging AI failures manually can be very expensive, especially when hallucinations arise as they directly affect our customers. While it helps, the costs can escalate due to unknown error factors and the challenge of containing them.

Arize AI satisfies most of our use cases, but there are times when costs can escalate, especially with the extensive traces explored and large embeddings. If a mechanism can be found to contain these costs, it would be a perfect product. Otherwise, considering enterprise credibility and a strong governance model, it meets most of our needs.

Arize AI is stable.

Scalability is high; we manage different models without any hiccups, and the downtime is very low.

Customer support is at par; they are quick and effective in addressing the pain points our team raises regarding functionality or feature extraction. I would rate the customer support as nine.

We did not switch from a different solution; we found that Arize AI had the best reviews regarding compliance and experience in enterprise-grade offerings, so we directly purchased it to address our monitoring challenges that were previously manual, expensive, and time-consuming.

We have definitely seen a return on investment with Arize AI. It has saved us a lot in penalties, as we identified models drifting due to changes in ingestion and data format. Our timely actions, aided by Arize AI, have allowed us to report results with over 99% accuracy, proving it quite useful.

The setup cost is generally a one-time expense; we have acquired a couple of licenses specifically for the AI/ML team to monitor our in-house AI/ML models because teams find it useful.

We evaluated LangSmith and Helicone but chose Arize AI because of its enterprise-grade offerings.

My advice for others considering Arize AI is if you need an enterprise-grade solution with strong compliance requirements, go for Arize AI without hesitation. It provides reliable results and saves a lot of time. Arize AI is a good tool, and I believe that with improvements on cost and evaluation framework, it can be the go-to tool in this AI-native world. I give this product a rating of eight.

My main use case for Arize AI involves exploring alternative solutions for Langfuse and LLM platforms. I was exploring several products in the market for model evaluation and prompt testing.

A specific example of how I used Arize AI in one of my projects is that we conduct evaluation and test different prompts because the business idea involves business developers developing the business logic while product owners can test the prompt template from the playground.

For Arize AI, my team also uses logging, which is typical usage for most such platforms.

Arize AI offers standard features, some of which are solid. The features I consider particularly useful for my work include the prompt template, exploring with the playground, and evaluators as the next components we are touching.

Arize AI has positively impacted my organization because we were already familiar with such platforms before, including LLM and Langfuse. At the beginning, we were also testing LangSmith. Arize AI, with its major features similar to those platforms, is a good alternative.

Arize AI can add more functions. I see it has monitors, evaluators, and prompt test datasets, which are good. However, I feel that other platforms can provide even more comprehensive feature sets.

I would like Arize AI to have more features, for example, some platforms can provide end-to-end capabilities, including drag and drop for testing the flow and attaching the knowledge base. I do not see those features in Arize AI. However, this is fine if it focuses on just the evaluation or the prompt testing.

I started using Arize AI around last month.

My advice to others looking into using Arize AI is that if you are seeking to improve your agentic application quality or if you want to separate the workflow between your product owner, QA, and the developers, then Arize AI is a good choice. You can give it a try.

Regarding Arize AI's AI capabilities, I think we are not in government security. The accuracy and reliability of output regarding Arize AI's AI capabilities is not the job of Arize AI or such similar platforms. The accuracy comes from the prompt template provided by a user along with the model quality, which is provided by OpenAI or Claude.

I found this interview interesting, but I feel that some of the questions may not be suitable for these products, such as response accuracy and security. They do not even have a guardrail feature. How can we evaluate security and governance? Some of the questions may not be applicable for this instance, which is something to consider. I would rate this product a 7 out of 10.

My main use case for Arize AI is to create LLM software. Recently, we were looking for an AI agent to automate all the tasks that we were doing manually, such as creating a proper system where we can import data from a software, send direct emails to the system, and get responses to manage all operations. We did not want to hire a team for all that manual work. We preferred building an AI agent, so we used Arize AI and created that automation software to automate all our tasks and save more of our time.

I can see that Arize AI is used for LLM tracing. We can use that functionality. Suppose we are creating an agent, we can set up manual processes into this system. Suppose it will be operating on Instagram, it will be doing billing, or it will be providing tech support, or it will be giving knowledge to the system. A user can click on billing, then they can proceed with billing, and if they want customer support, then they can access customer support. All these things are properly managed by an agent nowadays. Arize AI is successful in that capacity.

I can say that the best features Arize AI offers is that I do not need to use multiple software solutions. Suppose I do not need to connect with third-party apps; it is a complete AI team. It is not just one software; it is a complete AI team. I can do anything available from this one software. I need not merge any third-party software. I need not integrate it. All the things that I want to do as an agent, a basic AI agent, I can access Arize AI and create an agent. I can trace from there, evaluate from there, experiment, give a prompt, monitor, and give annotation. All the things are possible.

The feature I use most often and find the most valuable in my daily work is that the prompt playground is more of a benefit for me. We can give a prompt, set the functions, and see how users interact with it. All these things, and we can target our language from the features. We can send messages also. We can see auto-generated prompts. We can view them from here. We can run two prompts at a time. We can run multiple prompts at a time. I think it is quite useful.

In the prompt playground, I can see we can do most of the things. We can translate the prompt from one thing to another. We can use any of ChatGPT. We can use any model from the AI, such as GPT, and we can use any parameters. It is not limited to one software. We can change software also. We can use AI bots also from here. I think that is quite useful.

Arize AI has positively impacted my organization by reducing most of our manual work. We have shifted to complete automation from this. Working hours are reduced and we are more focused. There is less chance of mistakes. We are more focused toward accuracy and can focus more on our work.

I think we can improve its interface. The interface is a little boring. We can make it cool and engaging.

I have been using Arize AI for around four to five months.

We can say that we hired three members for customer support and built an AI. Those three members were costing us around 60,000, and we spent that amount on this AI, so I think that was good. That is something we reduced.

If others are looking to build an AI agent and reduce headaches from the company and focus more on accuracy while reducing the politics of the company, I advise them to go for AI software, reduce manual workload, and shift to automated tasks so that you can focus more on your work rather than the politics happening in the company nowadays. Arize AI is quite useful and it is great. My review rating for this product is 10.

Arize AI is used for LLM observability, tracing requests, debugging bad responses, and monitoring model quality over time. Traditional ML models also benefit from Arize AI's drift monitoring. It was particularly helpful when a support bot provided inaccurate technical documentation due to hallucinating results. Arize AI allowed the team to pinpoint the issue with the retrieval strategy and improve response accuracy.

Another significant use was in the retrieval-based support chatbot where Arize AI helped trace the source of irrelevant answers, saving the team considerable guesswork.

Arize AI's evaluation tools are essential for running automated regression tests against core prompts when updating models or system instructions. This involves setting up a golden dataset for expected outputs and measuring performance in terms of relevance, toxicity, and hallucination rates. This ensures early detection of regressions and consistent model behavior as scaling occurs.

The most useful feature of Arize AI is its tracing feature, allowing for the inspection of every step in an LLM workflow, which is incredibly valuable. The evaluation tools are also significant for testing output quality. Additionally, OpenTelemetry support is crucial for flexibility, enabling handling of projects using LangChain and custom APIs.

Arize AI has made leadership more comfortable with introducing AI features by providing better visibility into failures and reducing unexpected issues in production. Debugging production issues is reportedly thirty to forty percent faster, and inefficient workflows have been identified, reducing wasted LLM calls by approximately fifteen percent, thus improving overall efficiency.

More end-to-end architecture examples would be beneficial as current technical documentation is solid, but more practical examples are desired. LLM monitoring dashboard customization could be improved, as logs were exported to external dashboards for deeper analysis. Additionally, pricing and onboarding could be improved to be smoother as traffic increases.

I have been using Arize AI for approximately seven months.

Arize AI is generally stable, with no major outages experienced, only occasional delays when processing larger datasets.

Arize AI scales well as it can handle high request volumes without major issues, making it suitable for larger production teams.

Customer support from Arize AI was helpful when addressing integration questions, with responses that were not instant but usually useful.

Before Arize AI, CloudWatch logs, DataDog, and custom dashboards were used. Those tools managed infrastructure issues but were less effective for debugging LLM behavior.

The setup for Arize AI was quick, with basic tracing operational in a day.

The biggest return on investment with Arize AI is faster debugging, leading to fewer production issues and saving engineering time, rather than direct infrastructure costs.

Setup was quick, with pricing manageable early on. However, as traffic increased, usage needed to be monitored more closely.

LangFuse and LangSmith were considered, but Arize AI was chosen for its stronger observability capabilities at scale, especially for both ML and LLM monitoring.

More end-to-end architecture examples would be beneficial. Arize AI becomes increasingly valuable as AI systems get more complex. For simple prototypes, it may feel excessive, but it is very useful for production AI applications.

My advice for others considering Arize AI is to invest in observability early when building AI applications in production to avoid user-reported issues later.

Arize AI is a solid product overall. I would rate this review an eight out of ten.

My main use case for Arize AI is building a people intelligence agent, specifically in the human performance and human resource management field. Arize AI helps us verify whether those agents are giving good, safe, accurate, and useful answers to customers. This encompasses more than a single use case.

The best features Arize AI offers are that it evaluates responses against simple quality rules. In the field of generative AI, LLMs can hallucinate, and AI can be biased, so we need a proper evaluation framework in place. Arize AI helps in creating those safeguards and boundaries when developing enterprise AI.

I find the evaluation framework in Arize AI to be much better compared to any other tools or manual methods I may have tried. The manual method is tedious, inaccurate, and not scalable. We used to perform sanity checks before releasing code to production, but there is a human limit to how much you can check. We need automation in the quality testing of AI responses, and Arize AI is one of the best tools available to do this.

Arize AI has positively impacted my organization as the answers are more accurate and agent quality has improved dramatically. We can now debug much more easily, and if there is any bug, biased report, biased answer, or AI agent hallucinating, we can debug it very clearly and pinpoint bugs.

I have noticed faster debugging and significantly improved quality of responses because we can now debug and solve issues easily. Faster debugging led to agent quality improvement and an improved customer NPS score.

I think Arize AI can be improved as we are moving towards a more agentic framework where one agent orchestrates multiple agents. While Arize AI is very good when you have multiple agents, it falls short if orchestration is happening between agents in a hierarchy. I would not say it is an issue but rather a futuristic vision, as right now it is quite accurate and is solving the current need.

I have started using Arize AI in the last six months.

I would not add anything else about the features. Regarding Arize AI's AI capabilities, I think its governance and security are very good. Regarding Arize AI's AI capabilities, I think its accuracy and reliability of output are highly reliable and highly accurate. The advice I would give to others looking into using Arize AI is that it is one of the best tools. When building an enterprise or responsible AI framework to deploy at a larger scale, you need a validation framework. Arize AI is solving a problem that exists in the current world, so I think it is definitely a good product with really good product-market fit, and it is needed. I would rate this product a 9 out of 10.