This can be a visitor weblog put up co-authored with Atul Khare and Bhupender Panwar from Salesforce.
Headquartered in San Francisco, Salesforce, Inc. is a cloud-based buyer relationship administration (CRM) software program firm constructing synthetic intelligence (AI)-powered enterprise purposes that enable companies to attach with their prospects in new and customized methods.
The Salesforce Belief Intelligence Platform (TIP) log platform crew is chargeable for information pipeline and information lake infrastructure, offering log ingestion, normalization, persistence, search, and detection functionality to make sure Salesforce is secure from risk actors. It runs miscellaneous companies to facilitate investigation, mitigation, and containment for safety operations. The TIP crew is crucial to securing Salesforce’s infrastructure, detecting malicious risk actions, and offering well timed responses to safety occasions. That is achieved by gathering and inspecting petabytes of safety logs throughout dozens of organizations, some with 1000’s of accounts.
On this put up, we talk about how the Salesforce TIP crew optimized their structure utilizing Amazon Internet Providers (AWS) managed companies to attain higher scalability, price, and operational effectivity.
TIP current structure chicken’s eye view and scale of the platform
The primary key efficiency indicator (KPI) for the TIP platform is its functionality to ingest a excessive quantity of safety logs from a wide range of Salesforce inside techniques in actual time and course of them with excessive velocity. The platform ingests greater than 1 PB of information per day, greater than 10 million occasions per second, and greater than 200 totally different log sorts. The platform ingests log recordsdata in JSON, textual content, and Frequent Occasion Format (CEF) codecs.
The message bus in TIP’s current structure primarily makes use of Apache Kafka for ingesting totally different log sorts coming from the upstream techniques. Kafka had a single subject for all of the log sorts earlier than they had been consumed by totally different downstream purposes together with Splunk, Streaming Search, and Log Normalizer. The Normalized Parquet Logs are saved in an Amazon Easy Storage Service (Amazon S3) information lake and cataloged into Hive Metastore (HMS) on an Amazon Relational Database Service (Amazon RDS) occasion based mostly on S3 occasion notifications. The info lake shoppers then use Apache Presto operating on Amazon EMR cluster to carry out one-time queries. Different groups together with the Knowledge Science and Machine Studying groups use the platform to detect, analyze, and management safety threats.
Challenges with the present TIP log platform structure
A few of the foremost challenges that TIP’s current structure was going through embrace:
- Heavy operational overhead and upkeep price managing the Kafka cluster
- Excessive price to serve (CTS) to satisfy rising enterprise wants
- Compute threads restricted by partitions’ numbers
- Troublesome to scale out when visitors will increase
- Weekly patching creates lags
- Challenges with HMS scalability
All these challenges motivated the TIP crew to embark on a journey to create a extra optimized platform that’s simpler to scale with much less operational overhead and decrease CTS.
New TIP log platform structure
The Salesforce TIP log platform engineering crew, in collaboration with AWS, began constructing the brand new structure to switch the Kafka-based message bus answer with the totally managed AWS messaging and notification options Amazon Easy Queue Service (Amazon SQS) and Amazon Easy Notification Service (Amazon SNS). Within the new design, the upstream techniques ship their logs to a central Amazon S3 storage location, which invokes a course of to partition the logs and retailer them in an S3 information lake. Shopper purposes resembling Splunk get the messages delivered to their system utilizing Amazon SQS. Equally, the partitioned log information via Amazon SQS occasions initializes a log normalization course of that delivers the normalized log information to open supply Delta Lake tables on an S3 information lake. One of many main modifications within the new structure is the usage of an AWS Glue Knowledge Catalog to switch the earlier Hive Metastore. The one-time evaluation purposes use Apache Trino on an Amazon EMR cluster to question the Delta Tables cataloged in AWS Glue. Different shopper purposes additionally learn the info from S3 information lake recordsdata saved in Delta Desk format. Extra particulars on a few of the vital processes are as follows:
Log partitioner (Spark structured stream)
This service ingests logs from the Amazon S3 SNS SQS-based retailer and shops them within the partitioned (by log sorts) format in S3 for additional downstream consumptions from the Amazon SNS SQS subscription. That is the bronze layer of the TIP information lake.
Log normalizer (Spark structured stream)
One of many downstream shoppers of log partitioner (Splunk Ingestor is one other one), the log normalizer ingests the info from Partitioned Output S3, utilizing Amazon SNS SQS notifications, and enriches them utilizing Salesforce customized parsers and tags. Lastly, this enriched information is landed within the information lake on S3. That is the silver layer of the TIP information lake.
Machine studying and different information analytics shoppers (Trino, Flink, and Spark Jobs)
These shoppers eat from the silver layer of the TIP information lake and run analytics for safety detection use circumstances. The sooner Kafka interface is now transformed to delta streams ingestion, which concludes the whole removing of the Kafka bus from the TIP information pipeline.
Benefits of the brand new TIP log platform structure
The primary benefits realized by the Salesforce TIP crew based mostly on this new structure utilizing Amazon S3, Amazon SNS, and Amazon SQS embrace:
- Price financial savings of roughly $400 thousand per 30 days
- Auto scaling to satisfy rising enterprise wants
- Zero DevOps upkeep overhead
- No mapping of partitions to compute threads
- Compute assets could be scaled up and down independently
- Totally managed Knowledge Catalog to cut back the operational overhead of managing HMS
Abstract
On this weblog put up we mentioned how the Salesforce Belief Intelligence Platform (TIP) optimized their information pipeline by changing the Kafka-based message bus answer with totally managed AWS messaging and notification options utilizing Amazon SQS and Amazon SNS. Salesforce and AWS groups labored collectively to verify this new platform seamlessly scales to ingest greater than 1 PB of information per day, greater than 10 thousands and thousands occasions per second, and greater than 200 totally different log sorts. Attain out to your AWS account crew if in case you have comparable use circumstances and also you need assistance architecting your platform to attain operational efficiencies and scale.
Concerning the authors
Atul Khare is a Director of Engineering at Salesforce Safety, the place he spearheads the Safety Log Platform and Knowledge Lakehouse initiatives. He helps numerous safety prospects by constructing strong huge information ETL pipeline that’s elastic, resilient, and simple to make use of, offering uniform & constant safety datasets for risk detection and response operations, AI, forensic evaluation, analytics, and compliance wants throughout all Salesforce clouds. Past his skilled endeavors, Atul enjoys performing music along with his band to boost funds for native charities.
Bhupender Panwar is a Large Knowledge Architect at Salesforce and seasoned advocate for large information and cloud computing. His background encompasses the event of data-intensive purposes and pipelines, fixing intricate architectural and scalability challenges, and extracting precious insights from intensive datasets throughout the know-how business. Exterior of his huge information work, Bhupender likes to hike, bike, take pleasure in journey and is a superb foodie.
Avijit Goswami is a Principal Options Architect at AWS specialised in information and analytics. He helps AWS strategic prospects in constructing high-performing, safe, and scalable information lake options on AWS utilizing AWS managed companies and open-source options. Exterior of his work, Avijit likes to journey, hike within the San Francisco Bay Space trails, watch sports activities, and hearken to music.
Vikas Panghal is the Principal Product Supervisor main the product administration crew for Amazon SNS and Amazon SQS. He has deep experience in event-driven and messaging purposes and brings a wealth of information and expertise to his function, shaping the way forward for messaging companies. He’s obsessed with serving to prospects construct extremely scalable, fault-tolerant, and loosely coupled techniques. Exterior of labor, he enjoys spending time along with his household outdoor, enjoying chess, and operating.
