e-Commerce/Social Media AWS Enterprise Architecture with Machine Learning Full Life Cycle Illustration

https://app.diagrams.net/#G1nf2mKcHb_uwJw_QT93nEDJqFtn6AZKMV

1. AWS Route 53 provides DNS Configuration of hosted Domain
2. AWS WAF is a web application firewall that prevents Web App against common web exploits. Leverage also on the AWS Shield for DDOS attacks.
3. AWS Rekognition to detect content that is inappropriate, unwanted, or offensive.
4. AWS Elemental MediaConvert used to transcode service with broadcast-grade features, Lambda used to convert the media files with mobile friendly features.
5. AWS CloudFront is a fast content delivery network (CDN) that speeds up the distribution of static and dynamic web/mobile edge content
6. Elastic Load Balancing (Web Servers) distributes traffic across Web Servers Instances in an Auto Scaling Group of Multi-Availability Zones
7. Application Load Balancing (Application Servers) distributes traffic across App Servers Instances in an Auto Scaling Group of Multi-Availability Zones
8. AWS Cognito provides solutions to control access to AWS resources from the web apps
9. AWS Lambda serverless compute service used to trigger the API Gateway for Data retrieving micro-services, or machine learning services for external partners or specific group of users
10. AWS Simple Email Service (SES) / AWS Simple Notification Service (SNS) are email / SMS services to send mail / SMS from within any application.
11. NAT gateway enables instances in a private subnet can connect to services outside of the VPC but external services cannot initiate a connection with those instances
12. AWS RDS (SQL) / AWS DynamoDB (NoSQL) used to simply the database administration with multiple master and high availability configuration.

• Use RDS/DynamoDB Automatic Backup for Disaster Recovery ( with RPO & RTO Design goal).
• Use RDS/DynamoDB Encryption for Data Security with Data Encryption in Transit and at Rest.
• Use Elastic Cache to enable in-memory data store and cache to support the most demanding applications requiring sub-millisecond response times from SQL or NoSQL database.

13. AWS Kinesis Data Stream (KDS) capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. And all the data collected is available for real-time dashboards, real-time anomaly detection, dynamic pricing etc.

• When to use AWS Kinesis or AWS MSK (Managed Streaming for Kafka) considerations – https://farmoutain.wordpress.com/2021/08/04/aws-enterprise-data-lake-solution-architecture-design-principles-and-usage-patterns/
• Kinesis Data Firehose (KDF) capture, transform and load streaming data into data lakes (S3), data stores (Redshift, Elasticsearch), and analytics services (generic HTTP endpoints, Data Dog, Splunk etc).
• Kinesis Video Streams makes it easy to securely stream video from connected devices to AWS for analytics, machine learning (ML), playback, and other processing.
• Kinesis Data Analytics enable the transformations and analyzation of streaming data in real time with Apache Flink Events processing Graphs and Machine Learning Library.

14. S3 as a data lake – I have previously shared a reference architecture of data lake, please refer to

(Items 14 to 17)
https://farmoutain.wordpress.com/2021/08/04/aws-enterprise-data-lake-solution-architecture-design-principles-and-usage-patterns/

18. AWS SageMaker provide a full life cycle for data scientists and developers to prepare data, build, train, test/fine tune, deploy, monitor and re-train machine learning models. Full Machine Learning Life Cycle is illustrated in item 20 below.
19. AWS API Gateway makes it easy to create, publish, maintain, monitor, and secure APIs at any scale. Basically design for Open API data/machine learning/external system flow integration with partners. This includes innovative business model to monetize the data, financial services, machine learning models by providing relevant APIs as a services.
20. AWS Full Machine Learning Life Cycle Illustration:

• SageMaker Data Wrangler used for cleaning, normalizing, transforming and encoding data, as well as joining datasets.
• SageMaker Clarify used for detecting pre-training (data bias) and post-training bias (model bias) in the data preparation and training stage. It is also used to generate insights for Prediction influential factors in the inference stage.
• SageMaker Feature Store allows to create offline feature groups that keep all the historical data and can be used as inputs features for training models.
• SageMaker Notebooks are used to perform advanced data exploration, create training jobs, deploy models to Amazon SageMaker hosting, and test or validate the machine learning models
• SageMaker ML Lineage Tracking associate all the artifacts (such as data, models, and parameters) with a trained model to produce metadata
• SageMaker Model Registry stores the metadata around all the artifacts that includes in the process of creating the models, along with the trained models.
• SageMaker Pipelines After experiment and decide on the various options in the lifecycle (such as which transforms to apply to the machine learning features, determine imbalance or bias in the data, which algorithms to choose to train with, or which hyperparameters are giving us the best performance metrics), SageMaker Pipelines can automate the various tasks across the entire machine learning lifecycle to the SageMaker Hosted Endpoints. This is business ready deployment of Machine Learning as a services, leverage on the API Gateway to securely manage the Machine Learning APIs for partners/clients.

21. To build a set of security best practices across Infrastructure Protection, Data Security, Identity & Access Management, Incident Detection and responses. Collect security and access events logs and store centrally, monitor the security events for regular review and generate (SNS) alerts on certain events. Automate some incidents response based on SNS Alerts
22. AWS CloudTrail – enables governance, compliance, operational auditing, and risk auditing of the AWS account. CloudTrail can log, continuously monitor, and retain account activity related to actions across the AWS infrastructure.