Enhancing Scalability and Reliability
of backend for Precious Metals Giant

Overview

The client is one of the top retailers in precious metals in the USA and has an online retail store to sell precious metals. Over time the online retail store has reached its limitations, and is currently unable to achieve required uptime, performance and scale consequently resulting in negative revenue and brand impact and loss of credibility in the market. The client partnered with Numino Labs to transform its online retail store infrastructure to be able to meet his current and future business goals.

Project Info

Category

Cloud Backend

Client

Precious Metals retailer

Tags

  • AWS
  • Nodejs
  • Serverless
  • EFS

Problem Statement

The online store has the following capabilities

  • Enables consumers to understand the latest price for precious metals in real time
  • Enables consumers to purchase precious metals in physical form
  • Supports 10+ different languages to target customers from all over the world
  • Currently supports 500k to 1m consumers

The technology infrastructure representing the online retail store is hosted in the Amazon AWS cloud. The system has a legacy monolithic design and is hosted on a single server instance.


The system currently handles the following load

  • Generates 1000+ text file every minute corresponding to precious metal price data point it receives every minute from external vendors
  • Generates 15000+ pricing chart images per minute for a single precious metal, and over 69000+ chart images overall across all precious metals. The pricing charts are pre generated as images

Pre-generating pricing charts are used to

  • Allow consumers to view the latest precious metal prices instantly, without experiencing delays associated with real-time rendering, which can be resource-intensive and slow.
  • Are consumed by the other direct clients.
  • Allow the system to efficiently handle thousands of concurrent requests,

Numino Labs assessment of the current system revealed the following challenges

Scalability
Scalability

The computing instance struggled to handle the load of generating the enormous volume of image & text files and could no longer be vertically scaled at manageable cost.

Reliability
Reliability

Frequent service failures due to the monolithic structure, with no efficient failover mechanisms.

Maintenance
Maintenance

High maintenance overhead with a single point of failure, leading to frequent customer complaints and manual interventions to resolve the issue.

The idea was to move this backend to an alternate serverless architecture that allowed the system to operate at such scale & volume of requests at significantly lower cost.

Solution Features

Numino Labs came up with a new architecture and design for the online retail store backend based on serverless architecture capabilities supported by Amazon AWS cloud platform. We also upgraded the client technology stack to use modern day technologies that work well with serverless architectures on Amazon AWS

Scalability
Scalability
The serverless architecture allows for automatic scaling of resources to handle the generation of over 69,000+ charts per minute without performance degradation. AWS Lambda ensures the system can adapt to varying loads dynamically.
Reliability
Reliability
The distributed nature of AWS Lambda functions and AWS EFS storage eliminates the single point of failure present in the previous architecture, reducing downtime and enhancing the reliability of chart generation and delivery.
Cost-Efficiency
Cost-Efficiency
The client benefits from a pay-as-you-go pricing structure, significantly reducing operational costs compared to maintaining large & costly computing instances. The choice of EFS over S3 as storage also significantly reduces the cost.
Improved Performance
Improved Performance
The transition to lambda with Node.js runtime for backend services results in faster execution and lower latency, contributing to a more responsive user experience across web and mobile platforms.

Results

Below is a statistical comparison of the two architectural design patterns.

Technology Stack

Framework

Migrated the old platform to Node.js which is lightweight and suitable for microservices architecture where individual components are deployed independently.

 

It uses an event-driven, non-blocking I/O model that makes it highly efficient for handling concurrent requests, making it suitable for applications that require real-time updates or handle a large number of simultaneous connections.

Database

Microsoft SQL Server (RDS) and MongoDB for data storage (unchanged).

Orchestration

Leveraging AWS Lambda for backend processes, enabling automatic scaling and high availability & improving fault tolerance.

  • Allowed memory allocation of up to 10GB thus helping improve the function's performance, especially for compute-intensive tasks allowing faster execution times.
  • Utilizing lambdas concurrency for concurrent processing and completion in under 1 min.
Storage

EFS was selected over S3 to provide a more suitable file system interface as

  • It is ideal for applications(such as chart generation service) that require frequent read and write access with consistency for low file sizes.
  • Ease of Integration with AWS Lambda, allowing Lambda functions to mount the EFS file system directly and read from or write to it as needed.
  • More cost-efficient for applications with frequent small file operations, reducing costs compared to S3's request-based pricing mode.

Architecture

The new architectural style focuses on serverless and distributed computing, ensuring that each component can scale independently and that failures in one service do not affect the overall system.

The system’s reliability has improved with automated retries and robust logging mechanisms.
Data Ingestion
The system ingests data points from external vendors in real-time to keep the price charts up-to-date. Over 1,000+ text files are generated every minute, structuring vendor data for further processing and storage.
Serverless Processing
AWS Lambda functions are triggered to process incoming data and generate charts, leveraging Node.js for its asynchronous capabilities. Allowed isolation of text and image generation process.
Database Management
Data is stored in Microsoft SQL Server (RDS) and MongoDB, allowing structured and unstructured data to be managed effectively.
Storage with Amazon EFS
Amazon EFS is used to store pre-generated charts and text files, supporting concurrent access and frequent read/write operations.

Summary

The architectural shift ensures that the retailer can continue to provide real-time, accurate data to its customers, enhancing their user experience across all platforms. The Numino team’s innovative approach has resolved scalability and reliability issues, resulting in a more robust and maintainable system.

Read More

Pixm Anti Phishing v2.0
Cloud Backend,Web Apps,
Pixm Anti Phishing v2.0

Pixm is an AI tool that stops phishing attacks instantly when clicked, using serverless technology for scalable, cloud-based threat detection and comprehensive enterprise security management.
StrategicIT ePMOLite Project Management
Cloud Backend,Web Apps,
StrategicIT ePMOLite Project Management

ePMOLite transforms IT PMO for small to mid-sized organizations with a cloud-based, serverless solution that tracks progress and financials, facilitating informed decision-making.
PromptMule Prompt Cache LLMs
AI,Cloud Backend,
PromptMule Prompt Cache LLMs

PromptMule, a managed caching service, leverages semantic caching and vector embeddings to deliver lightning-fast responses, reducing costs by 70% while improving application responsiveness by 5x.