Technology Stack
Overview
PromptMule is a managed caching service that delivers lightning-fast responses while reducing costs. It intelligently stores and retrieves AI model outputs, allowing applications to rapidly respond to user queries without expensive re-computation by employing advanced techniques like semantic caching, vector embeddings, and intelligent query matching, thus empowering developers and businesses to create high-performance, cost-effective, secure and easily scalable AI-powered applications that users can trust.
PromptMule was founded by two longtime friends with a shared passion for technology and a combined experience of over 20 years in product management and engineering. Having worked together at one of the world’s largest security companies, they recognized the immense potential of generative AI and set out to create a solution that would revolutionize the way AI applications create trust, transparency, and traceability.
Category
AI, Cloud Backend
Client
Promptmule
Tags
- AWS
- Serverless
- Pinecone
- OpenAI
Product Features
Ease of integration
Low-Latency API Cache
Cost Savings
Enhanced Security
User & App Metrics
Flexible Cache Access
01 Embedding Generator
Transforms text into numerical embeddings for similarity search.
The system can identify the most semantically relevant results, improving the accuracy and relevance of response
02 Cache Storage
Securely stores and retrieves model outputs for fast response.
03 Vector Store
Enables efficient similarity search to find relevant cached data.
Helps in various applications, including personalized recommendations, context-aware search results, and improved dialogue management in conversational agents.
04 LLM Adapter
Integrates with various language models through unified APIs.
Provides ease of integration with new LLM providers.
Workflow
At its core, PromptMule uses semantic caching to identify and store similar queries and responses. It leverages AI embeddings, which are numerical representations of text, to enable fast similarity search of cached data. This approach, similar to Retrieval Augmented Generation (RAG), dramatically increases cache hit rates and performance.
This allows the application to reference the results when the user asks a semantically similar question in the future, thereby reducing the number of API calls made to the LLM provider. The advantage of semantic caching is that it reduces both the latency and cost of the overall RAG application.
Architecture
Under the hood, PromptMule leverages various AWS services to ensure scalable, secure, and reliable operation.
UI / Application
- features a hosted website, user profile pages, API documentation, and other pages such as user feedback, pricing, etc.
- The domain is managed using AWS Route 53 and employs CloudFront for service requests.
- The web pages are developed with a ReactJS application hosted on Amazon S3.
Security and Access
- API requests, whether direct from users or via the application, are routed through the API Gateway through cognito authentication & authorisation.
- User management is handled through AWS Cognito, and sensitive information such as passwords and API keys are securely stored using AWS Secrets Manager.
Orchestration
- The orchestration layer is critical for the cache system, encompassing all the logic required for cache delivery.
- Serverless AWS Lambda functions are used for orchestration to ensure cost efficiency.
Datastore
- A MySQL database acts as the cache system, storing responses and all user request information.
- Multi-tenant enabled, allowing for secure and isolated data handling across different users.
Event Management
- Events are triggered by logs from AWS CloudWatch and routed to appropriate services such as Amazon SES, Slack, and GitHub through SES.
External System integrations
- Pinecone : application uses the serverless version of the Pinecone vector database to store embeddings required for RAG.
- Stripe Payment Gateway: Stripe is integrated for managing subscriptions to different tiers of API usage.
- Slack/Git : Issues raised through user feedback that are logged into Git as well as are notified to appropriate user groups in Slack.
Result
Improved application responsiveness by 5x while reducing costs by 70%.
Developed in less than 90-days with CI/CD, achieving rapid time-to-market goals.
Seamless scalability, with capacity to handle billions of requests.
Enterprise-grade security.
Future State
The following shows the envisioned future state
Integration with LLM's
Integrating the platform with multiple large language model (LLM) providers to create a unified system
Use of blockchain technology
for ownership of digital assets & managing intellectual property rights
Conclusion
PromptMule represents a leap forward in making high-performance AI accessible and cost-effective for all developers. By democratizing access to state-of-the-art caching technology, it enables a new generation of engaging, real-time AI experiences. PromptMule eliminates the complexity and expense traditionally associated with optimizing AI applications, empowering teams to ship better products faster.
What PromptMule Customers Say
PromptMule transformed our customer support. Our users now enjoy instant, reliable help, and our support team can focus on what really matters. It's been a game-changer for our customer satisfaction and operational efficiency.
