Gen Ai Architect at a tech vendor with 10,001+ employees
Real User
Top 20
Jul 29, 2026
Our client is completely using AWS products. For AI gateways, we are using Portkey, which is the product we have implemented, and we are leveraging all AWS services for internal application development. Applications can interact with different multiple LLM models. We are restricting direct access from the application to LLMs. Instead, we are introducing Portkey as an AI gateway in between. We send the application's requests to the respective LLMs through Portkey, and we have enabled token usage, caching mechanisms, and guardrails at the AI gateway level. All applications must send requests via Portkey to leverage LLM capabilities.
Portkey has many features for deploying an application to production, including an AI gateway, observability, guardrails, governance, and prompt management. We primarily use the AI Gateway, observability, and the guardrails. We have an application that uses LLM models such as Gemini, OpenAI models, or Claude models. We need to interact with these models, which we have deployed elsewhere, and we use Portkey as the AI/LLM gateway to check observability and monitor how our model is behaving. We also have custom fine-tuned trained models, and we want to check the observability or test these models using Portkey. In this scenario, when any person wants to call the LLM or use the AI, they do not need the API key for that model. They can directly call Portkey and it will provide the response. The API keys for all the models are not exposed to the developers or the engineers. They are stored in the vault and can only be accessed by Portkey.
Portkey serves as a centralized AI gateway for our organization. At Clydo, we are building multiple AI-powered features, including internal copilots, catalog intelligence, customer support assistance, and agentic workflows. This gives us a single layer for model routing, prompt management, and API key management. A primary example of our main use case is our internal AI assistant used by our operations and customer support team. Every user query goes through Portkey before reaching the language model. Portkey handles authentication and routes requests to appropriate models. One practical scenario involves our engineering team wanting to evaluate a newer model for better reasoning at a lower cost. Instead of changing code across multiple services, we update the routing configuration in Portkey and compare the responses and performance. Another benefit is observability. When users report that AI responses are slower or inconsistent, we use Portkey's logs to see the request latency. Overall, Portkey has become the control layer for our LLM infrastructure. It allows our engineers to focus on building AI features while providing us with centralized visibility, governance, and flexibility. The main use case is standardizing how AI is consumed across the organization. As more teams start building AI-powered features, we do not want every service to implement its own integration with retry logic or rate limiting, which becomes very difficult to manage. Portkey gives us a common layer that every AI application uses, whether it is an internal or customer-facing chatbot or any agentic flows. This makes the architecture much cleaner and easier to manage. It gives us the flexibility to experiment. We can compare different models and run A/B evaluations for specific workloads. From an engineering perspective, Portkey has evolved from being just an API gateway to becoming part of our platform. It is always a layer that provides consistency, observability, and governance across all our LLM-based applications.
Our team is searching for an alternative to LiteLLM, which is another monitoring management platform, and we found that Portkey is an alternative solution that we decided to try out. We want to conduct prompt template testing with Portkey by splitting our business logic along with the prompt, so we need to test that against different models and perform evaluation accordingly. This is one major thing we are trying. The product owner can separately conduct their test for the prompt template on the playground. There could be some advanced usage with Portkey such as setting up the MCP and guard rail. We are thinking about having that part as well, but so far we haven't touched it yet.
Portkey offers a centralized AI gateway that simplifies architecture and reduces maintenance by providing a unified interface for model interactions, with features ensuring detailed observability and flexibility in model operations.Portkey serves as a centralized hub for AI management, allowing organizations to streamline AI operations through a singular interface that accommodates multiple language models. It ensures efficient resource management with robust observability features for...
Our client is completely using AWS products. For AI gateways, we are using Portkey, which is the product we have implemented, and we are leveraging all AWS services for internal application development. Applications can interact with different multiple LLM models. We are restricting direct access from the application to LLMs. Instead, we are introducing Portkey as an AI gateway in between. We send the application's requests to the respective LLMs through Portkey, and we have enabled token usage, caching mechanisms, and guardrails at the AI gateway level. All applications must send requests via Portkey to leverage LLM capabilities.
Portkey has many features for deploying an application to production, including an AI gateway, observability, guardrails, governance, and prompt management. We primarily use the AI Gateway, observability, and the guardrails. We have an application that uses LLM models such as Gemini, OpenAI models, or Claude models. We need to interact with these models, which we have deployed elsewhere, and we use Portkey as the AI/LLM gateway to check observability and monitor how our model is behaving. We also have custom fine-tuned trained models, and we want to check the observability or test these models using Portkey. In this scenario, when any person wants to call the LLM or use the AI, they do not need the API key for that model. They can directly call Portkey and it will provide the response. The API keys for all the models are not exposed to the developers or the engineers. They are stored in the vault and can only be accessed by Portkey.
Portkey serves as a centralized AI gateway for our organization. At Clydo, we are building multiple AI-powered features, including internal copilots, catalog intelligence, customer support assistance, and agentic workflows. This gives us a single layer for model routing, prompt management, and API key management. A primary example of our main use case is our internal AI assistant used by our operations and customer support team. Every user query goes through Portkey before reaching the language model. Portkey handles authentication and routes requests to appropriate models. One practical scenario involves our engineering team wanting to evaluate a newer model for better reasoning at a lower cost. Instead of changing code across multiple services, we update the routing configuration in Portkey and compare the responses and performance. Another benefit is observability. When users report that AI responses are slower or inconsistent, we use Portkey's logs to see the request latency. Overall, Portkey has become the control layer for our LLM infrastructure. It allows our engineers to focus on building AI features while providing us with centralized visibility, governance, and flexibility. The main use case is standardizing how AI is consumed across the organization. As more teams start building AI-powered features, we do not want every service to implement its own integration with retry logic or rate limiting, which becomes very difficult to manage. Portkey gives us a common layer that every AI application uses, whether it is an internal or customer-facing chatbot or any agentic flows. This makes the architecture much cleaner and easier to manage. It gives us the flexibility to experiment. We can compare different models and run A/B evaluations for specific workloads. From an engineering perspective, Portkey has evolved from being just an API gateway to becoming part of our platform. It is always a layer that provides consistency, observability, and governance across all our LLM-based applications.
Our team is searching for an alternative to LiteLLM, which is another monitoring management platform, and we found that Portkey is an alternative solution that we decided to try out. We want to conduct prompt template testing with Portkey by splitting our business logic along with the prompt, so we need to test that against different models and perform evaluation accordingly. This is one major thing we are trying. The product owner can separately conduct their test for the prompt template on the playground. There could be some advanced usage with Portkey such as setting up the MCP and guard rail. We are thinking about having that part as well, but so far we haven't touched it yet.