Real-time price chart computation at 16,000+ charts per minute

Overview

The client is one of the top retailers in precious metals in the USA and has an online retail store to sell precious metals. Over time the online retail store has reached its limitations, and is currently unable to achieve required uptime,  performance and scale consequently resulting in negative revenue and brand impact and loss of credibility in the market. The client partnered with Numino Labs to transform its online retail store infrastructure to be able to meet his current and future business goals.

Project Info

Category

Cloud Backend

Tags

  • AWS
  • Serverless
  • Nodejs
  • EFS

Problem Statement

The online store has the following capabilities
The technology infrastructure representing the online retail store is hosted in the Amazon AWS cloud. The system has a legacy monolithic design and is hosted on a single server instance.
Enables consumers to understand the latest price for precious metals in real time
Enables consumers to purchase precious metals in physical form
Supports 10+ different languages to target customers from all over the world
Currently supports 500k to 1m consumers
The system currently handles the following load
Data files

Generates 16000+ text file every minute corresponding to precious metal price data point it receives every minute from external vendors

Price Charts
Generates 10000+ pricing chart images per minute for intraday changes and over 69000+ historical chart images overall across all precious metals
Pre-generating pricing charts are used to

         01.     Are consumed by the other direct clients.

         02.    Allow the system to efficiently handle thousands of concurrent requests.

         03.    Allow consumers to view the latest precious metal prices instantly, without experiencing delays associated with real-time rendering, which can be resource-intensive and slow.

Numino Labs assessment of the current system revealed the following challenges
Scalability Issues
The computing instance struggled to handle the load of generating the enormous volume of image & text files and could no longer be vertically scaled at manageable cost.
Reliability Problems
Frequent service failures due to the monolithic structure, with no efficient failover mechanisms.
Complex Maintenance
High maintenance overhead with a single point of failure, leading to frequent customer complaints and manual interventions to resolve the issue.

The idea was to move this backend to an alternate serverless architecture that allowed the system to operate at such scale & volume of requests at significantly lower cost.

Solution Features

Numino Labs came up with a new architecture and design for the online retail store backend based on serverless architecture capabilities supported by Amazon AWS cloud platform. Numino Labs also upgraded the client technology stack to use modern day technologies that work well with serverless architectures on Amazon AWS

 

Performance Comparison between old and new architecture

Scalability
Scalability
The serverless architecture allows for automatic scaling of resources to handle the generation of over 25,000+ charts per minute without performance degradation. AWS Lambda ensures the system can adapt to varying loads dynamically.
Reliability
The distributed nature of AWS Lambda functions and use of AWS EBS storage eliminates the single point of failure present in the previous architecture, reducing downtime and enhancing the reliability of chart generation and delivery.
Cost-Efficiency:
Previous storage solutions like S3 and EFS proved costly for the site’s needs. By leveraging EBS (GP3) storage shared across instances via NFS within a VPC, the storage cost has been reduced to nearly zero, significantly lowering operational expenses.
Improved Performance
Improved Performance
The transition to lambda with Node.js runtime for backend services results in faster execution and lower latency, contributing to a more responsive user experience across web and mobile platforms.

Below is a timeline of the events for the price charts generation –

Technology Stack

Following the architectural overhaul, the technology stack now includes

Development

Database

Platform Service

Key Components

Framework
Framework
Migrated the old platform to Node.js which is lightweight and suitable for microservices architecture where individual components are deployed independently.
It uses an event-driven, non-blocking I/O model that makes it highly efficient for handling concurrent requests, making it suitable for applications that require real-time updates or handle a large number of simultaneous connections.
Storage
Storage
EBS (GP3) with NFS sharing was chosen as a more cost-effective solution over S3 and EFS. The shared EBS (GP3) storage solution is more suitable for applications requiring frequent, consistent file access without incurring the high costs associated with S3 or EFS.


Orchestration
Orchestration
Leveraging AWS Lambda for backend processes, enabling automatic scaling and high availability & improving fault tolerance.
Allowed memory allocation of up to 10GB thus helping improve the function's performance, especially for compute-intensive tasks allowing faster execution times.
Utilizing lambdas concurrency for concurrent processing and completion in under 1 min.
Database
Database
Microsoft SQL Server (RDS) and MongoDB for data storage (unchanged).

Architecture

The new architectural style focuses on serverless and distributed computing, ensuring that each component can scale independently and that failures in one service do not affect the overall system.

Serverless Processing
Serverless Processing
AWS Lambda functions are triggered to process incoming data and generate charts, leveraging Node.js for its asynchronous capabilities. Allowed isolation of text and image generation process. Each process runs independently and scales dynamically based on the system's load, handling thousands of concurrent requests efficiently.
Reliability and Logging Mechanisms
Reliability and Logging Mechanisms
he system's reliability has improved with automated retries and robust logging mechanisms. This helps ensure that even in the event of occasional errors, the system can recover quickly without downtime or loss of data integrity, improving the overall user experience.

Data Ingestion
Data Ingestion
The system ingests data points from external vendors in real-time to keep the price charts up-to-date. Over 1,000+ text files are generated every minute, structuring vendor data for further processing and storage.
Database Management
Database Management
Data is stored in Microsoft SQL Server (RDS) and MongoDB, allowing structured and unstructured data to be managed effectively.


Storage with EBS (GP3) and NFS
Storage with EBS (GP3) and NFS
Instead of using Amazon EFS or S3, which proved to be expensive for frequent small file operations, the system now utilizes EBS (GP3) storage shared via NFS within a VPC. This solution significantly reduces storage costs, which had previously been a major issue with S3 and EFS pricing models.

EBS (GP3) provides reliable, low-latency access to pre-generated charts and text files, with the added benefit of scalability for both read and write operations. The decision to use NFS for sharing EBS across instances allows for concurrent access while minimizing infrastructure costs, making it an optimal solution for handling real-time updates and frequent small file operations.

Summary

The architectural shift ensures that the retailer can continue to provide real-time, accurate data to its customers, enhancing their user experience across all platforms. The move to serverless computing and the adoption of EBS (GP3) storage have resolved the scalability and reliability issues present in the previous system.

The Numino team’s innovative approach has resolved scalability and reliability issues, resulting in a more robust and maintainable system.

Read More

Pixm Anti Phishing v2.0
Cloud Backend,Web Apps,
Pixm Anti Phishing v2.0

Pixm is an AI tool that stops phishing attacks instantly when clicked, using serverless technology for scalable, cloud-based threat detection and comprehensive enterprise security management.
StrategicIT ePMOLite Project Management
Cloud Backend,Web Apps,
StrategicIT ePMOLite Project Management

ePMOLite transforms IT PMO for small to mid-sized organizations with a cloud-based, serverless solution that tracks progress and financials, facilitating informed decision-making.
PromptMule Prompt Cache LLMs
AI,Cloud Backend,
PromptMule Prompt Cache LLMs

PromptMule, a managed caching service, leverages semantic caching and vector embeddings to deliver lightning-fast responses, reducing costs by 70% while improving application responsiveness by 5x.