In 2013, Amazon Net Companies revolutionized the info warehousing business by launching Amazon Redshift, the primary fully-managed, petabyte-scale, enterprise-grade cloud knowledge warehouse. Amazon Redshift made it easy and cost-effective to effectively analyze massive volumes of knowledge utilizing present enterprise intelligence instruments. This cloud service was a big leap from the standard knowledge warehousing options, which had been costly, not elastic, and required vital experience to tune and function. Since then, buyer calls for for higher scale, larger throughput, and agility in dealing with all kinds of adjusting, however more and more enterprise vital analytics and machine studying use instances has exploded, and we now have been preserving tempo. At present, tens of hundreds of consumers use Amazon Redshift in AWS international infrastructure to collectively course of exabytes of knowledge each day and employs Amazon Redshift as a key part of their knowledge structure to drive use instances from typical dashboarding to self-service analytics, real-time analytics, machine studying, knowledge sharing and monetization, and extra
The developments to Amazon Redshift introduced at AWS re:Invent 2023 additional accelerates modernization of your cloud analytics environments, preserving our core tenet that can assist you obtain the perfect price-performance at any scale. These bulletins drive ahead the AWS Zero-ETL imaginative and prescient to unify all of your knowledge, enabling you to raised maximize the worth of your knowledge with complete analytics and ML capabilities, and innovate sooner with safe knowledge collaboration inside and throughout organizations. From price-performance enhancements to zero-ETL, to generative AI capabilities, we now have one thing for everybody. Let’s dive into the highlights.
Modernizing analytics for scale, efficiency, and reliability
“Our migration from legacy on-premises platform to Amazon Redshift permits us to ingest knowledge 88% sooner, question knowledge 3x sooner, and cargo each day knowledge to the cloud 6x sooner. Amazon Redshift enabled us to optimize efficiency, availability, and reliability—considerably easing operational complexity, whereas rising the speed of our end-users’ decision-making expertise on the Fab ground.”
– Sunil Narayan, Sr Dir, Analytics at GlobalFoundries
Diligently driving the perfect price-performance at scale with new enhancements
Since day 1, Amazon Redshift has been constructing revolutionary capabilities that can assist you get to optimum efficiency, whereas preserving prices decrease. Amazon Redshift continues to steer on the price-performance entrance with as much as 6x higher price-performance than different cloud knowledge warehouse and for sprint boarding functions with excessive concurrency and low latency. We carefully analyze question patterns within the fleet and search for alternatives to drive customer-focused innovation. For instance, earlier within the yr, we introduced velocity ups for string-based knowledge processing as much as 63x in comparison with different compression encodings corresponding to LZO (Lempel-Ziv-Oberhumer) or ZStandard. At AWS re:Invent 2023, we launched extra efficiency enhancements in question planning and execution corresponding to enhanced bloom filters , question rewrites, and help for write operations in auto scaling . For extra details about efficiency enchancment capabilities, consult with the record of bulletins under.
Amazon Redshift Serverless is extra clever than ever with new AI-driven scaling and optimizations
Talking of price-performance, new subsequent era AI-driven scaling and optimizations capabilities in Amazon Redshift Serverless can ship as much as 10x higher price-performance for variable workloads (based mostly on inside testing), with out handbook intervention. Amazon Redshift Serverless, typically out there since 2021, lets you run and scale analytics with out having to provision and handle the info warehouse. Since GA, Redshift Serverless executed over a billion queries to energy knowledge insights for hundreds of consumers. With these new AI optimizations, Amazon Redshift Serverless scales proactively and routinely with workload adjustments throughout all key dimensions —corresponding to knowledge quantity, concurrent customers, and question complexity. You simply specify your required price-performance targets to both optimize for price or optimize for efficiency or balanced and serverless does the remaining. Be taught extra about extra enhancements in Redshift Serverless, beneath the record of bulletins under.
Multi-data warehouse writes by way of knowledge sharing
Knowledge sharing is a broadly adopted function in Amazon Redshift with prospects operating tens of hundreds of thousands of queries on shared knowledge each day. Prospects share stay transactionally constant knowledge inside and throughout organizations and areas for learn functions with out knowledge copies or knowledge motion. Prospects are utilizing knowledge sharing to modernize their analytics architectures from monolithic architectures to multi-cluster, knowledge mesh deployments that allow seamless and safe entry throughout organizations to drive knowledge collaboration and highly effective insights. At AWS re:Invent 2023, we prolonged knowledge sharing capabilities to launch multi-data warehouse writes in preview. Now you can begin writing to Redshift databases from different Redshift knowledge warehouses in just some clicks, additional enabling knowledge collaboration, versatile scaling of compute for ETL/knowledge processing workloads by including warehouses of various sorts and sizes based mostly on price-performance wants. Expertise higher transparency of compute utilization as every warehouse is billed for its personal compute and consequently maintain your prices beneath management.
Multidimensional knowledge layouts
Amazon Redshift presents business main predictive optimizations that repeatedly monitor your workloads and seamlessly speed up efficiency and maximize concurrency by adjusting knowledge format and compute administration as you employ the info warehouse extra. Along with the highly effective optimizations Redshift already presents, corresponding to Computerized Desk Type, Computerized type and distribution keys, we’re introducing Multidimensional Knowledge Layouts, a brand new highly effective desk sorting mechanism that improves efficiency of repetitive queries by routinely sorting knowledge based mostly on the incoming question filters (for instance: Gross sales in a particular area). This technique considerably accelerates the efficiency of desk scans in comparison with conventional strategies.
Unifying all of your knowledge with zero-ETL approaches
“Utilizing the Aurora MySQL zero-ETL integration, we expertise close to real-time knowledge synchronization between Aurora MySQL databases and Amazon Redshift, making it doable to construct an evaluation setting in simply three hours as a substitute of the month of developer time it used to take earlier than”
– Cash Ahead i
JOYME makes use of Amazon Redshift’s streaming ingestion and different Amazon providers for threat management over customers’ monetary exercise corresponding to recharge, refund, and rewards.
“With Redshift, we’re capable of view threat counterparts and knowledge in close to actual time—
as a substitute of on an hourly foundation. Redshift considerably improved our enterprise ROI effectivity.”– PengBo Yang, CTO, JOYME
Knowledge pipelines could be difficult and dear to construct and handle and might create hours-long delays to acquire transactional knowledge for analytics. These delays can result in missed enterprise alternatives, particularly when the insights derived from analytics on transactional knowledge are related for less than a restricted period of time. Amazon Redshift employs AWS’s zero-ETL strategy that allows interoperability and integration between the info warehouse and operational databases and even your streaming knowledge providers, in order that the info is definitely and routinely ingested into the warehouse for you, or you’ll be able to entry the info in place, the place it lives.
Zero-ETL integrations with operational databases
We delivered zero-ETL integration between Amazon Aurora MySQL Amazon Redshift (normal availability) this yr, to allow close to real-time analytics and machine studying (ML) utilizing Amazon Redshift on petabytes of transactional knowledge from Amazon Aurora. Inside seconds of transactional knowledge being written into Aurora, the info is out there in Amazon Redshift, so that you don’t should construct and keep advanced knowledge pipelines to carry out extract, remodel, and cargo (ETL) operations. At AWS re:Invent, we prolonged zero-ETL integration to extra sources particularly Aurora PostgreSQL, Dynamo DB, and Amazon RDS MySQL. Zero-ETL integration additionally allows you to load and analyze knowledge from a number of operational database clusters in a brand new or present Amazon Redshift occasion to derive holistic insights throughout many functions.
Knowledge lake querying with help for Apache Iceberg tables
Amazon Redshift permits prospects to run a variety of workloads on knowledge warehouse and knowledge lakes utilizing its help for varied open file and desk codecs. At AWS re:Invent, we introduced the final availability of help for Apache Iceberg tables, so you’ll be able to simply entry your Apache Iceberg tables in your knowledge lake from Amazon Redshift and be a part of it with the info in your knowledge warehouse when wanted. Use one click on to entry your knowledge lake tables utilizing auto-mounted AWS Glue knowledge catalogs on Amazon Redshift for a simplified expertise. We’ve got improved knowledge lake question efficiency by integrating with AWS Glue statistics and introduce preview of incremental refresh for materialized views on knowledge lake knowledge to speed up repeated queries.
Be taught extra concerning the zero-ETL integrations, knowledge lake efficiency enhancements, and different bulletins under.
Maximize worth with complete analytics and ML capabilities
“Amazon Redshift is among the most essential instruments we had in rising Jobcase as an organization.”
– Ajay Joshi, Distinguished Engineer, Jobcase
With all of your knowledge built-in and out there, you’ll be able to simply construct and run close to real-time analytics to AI/ML/Generative AI functions. Right here’s a few highlights from this week and for the total record, see under.
Amazon Q Generative SQL functionality
Question Editor, an out-of-the-box web-based SQL expertise in Amazon Redshift is a well-liked device for knowledge exploration, visible evaluation, and knowledge collaboration. At AWS re:Invent, we launched Amazon Q Generative SQL capabilities in Amazon Redshift Question Editor (preview), to simplify question authoring and improve your productiveness by permitting you to specific queries in pure language and obtain SQL code suggestions. Generative SQL makes use of AI to investigate consumer intent, question patterns, and schema metadata to determine frequent SQL question patterns instantly permitting you to get insights sooner in a conversational format with out intensive information of your group’s advanced database metadata.
Amazon Redshift ML massive language mannequin (LLM) integration
Amazon Redshift ML permits prospects to create, prepare, and deploy machine studying fashions utilizing acquainted SQL instructions. Prospects use Redshift ML to run a median of over 10 billion predictions a day inside their knowledge warehouses. At AWS re:Invent, we introduced help for LLMs as preview. Now, you need to use pre-trained open supply LLMs in Amazon SageMaker JumpStart as a part of Redshift ML, permitting you to deliver the facility of LLMs to analytics. For instance, you may make inferences in your product suggestions knowledge in Amazon Redshift, use LLMs to summarize suggestions, carry out entity extraction, sentiment evaluation and product suggestions classification.
Innovate sooner with safe knowledge collaboration inside and throughout the organizations
“Hundreds of thousands of firms use Stripe’s software program and APIs to simply accept funds, ship payouts, and handle their companies on-line. Entry to their Stripe knowledge by way of main knowledge warehouses like Amazon Redshift has been a prime request from our prospects. Our prospects wanted safe, quick, and built-in analytics at scale with out constructing advanced knowledge pipelines or shifting and copying knowledge round. With Stripe Knowledge Pipeline for Amazon Redshift, we’re serving to our prospects arrange a direct and dependable knowledge pipeline in a couple of clicks. Stripe Knowledge Pipeline permits our prospects to routinely share their full, up-to-date Stripe knowledge with their Amazon Redshift knowledge warehouse, and take their enterprise analytics and reporting to the subsequent degree.”
– Tony Petrossian, Head of Engineering, Income & Monetary Administration at Stripe
With Amazon Redshift, you’ll be able to simply and securely share knowledge and collaborate regardless of the place your groups or knowledge is situated. And have the arrogance that your knowledge is safe regardless of the place you use or how extremely regulated your industries are. We’ve got enabled positive grained permissions, a simple authentication expertise with single sign-on in your organizational id—all supplied at no extra price to you.
Unified id with IAM id heart integration
We introduced Amazon Redshift integration with AWS IAM Identification Middle to allow organizations to help trusted id propagation between Amazon QuickSight,, Amazon Redshift Question Editor, and Amazon Redshift, . Prospects can use their group identities to entry Amazon Redshift in a single sign-on expertise utilizing third occasion id suppliers (IdP), corresponding to Microsoft Entra ID, Okta, Ping, OneLogin, and so forth. from Amazon QuickSight and Amazon Redshift Question Editor. Directors can use third-party id supplier customers and teams to handle positive grained entry to knowledge throughout providers and audit consumer degree entry in AWS CloudTrail. With trusted id propagation, a consumer’s id is handed seamlessly between Amazon QuickSight, Amazon Redshift decreasing time to insights and enabling a friction free analytics expertise.
For the total set of bulletins, see the next:
-
Modernizing analytics for scale, efficiency, and reliability
-
Unifying all of your knowledge with zero-ETL approaches
-
Maximize worth with complete analytics and ML capabilities
-
Innovate sooner with safe knowledge collaboration inside and throughout the organizations
Be taught extra: https://aws.amazon.com/redshift
Concerning the authors
Neeraja Rentachintala is a Principal Product Supervisor with Amazon Redshift. Neeraja is a seasoned Product Administration and GTM chief, bringing over 20 years of expertise in product imaginative and prescient, technique and management roles in knowledge merchandise and platforms. Neeraja delivered merchandise in analytics, databases, knowledge Integration, software integration, AI/Machine Studying, massive scale distributed techniques throughout On-Premise and Cloud, serving Fortune 500 firms as a part of ventures together with MapR (acquired by HPE), Microsoft SQL Server, Oracle, Informatica and Expedia.com.
Sunaina AbdulSalah leads product advertising and marketing for Amazon Redshift. She focuses on educating prospects concerning the impression of knowledge warehousing and analytics and sharing AWS buyer tales. She has a deep background in advertising and marketing and GTM capabilities within the B2B know-how and cloud computing domains. Outdoors of labor, she spends time together with her household and buddies and enjoys touring.
