How Eightfold AI carried out metadata safety in a multi-tenant knowledge analytics atmosphere with Amazon Redshift

This can be a visitor publish co-written with Arun Sudhir from Eightfold AI.

Eightfold is remodeling the world of labor by offering options that empower organizations to recruit and retain a various international workforce. Eightfold is a frontrunner in AI merchandise for enterprises to construct on their expertise’s present abilities. From Expertise Acquisition to Expertise Administration and expertise insights, Eightfold affords a single AI platform that does all of it.

The Eightfold Expertise Intelligence Platform powered by Amazon Redshift and Amazon QuickSight offers a full-fledged analytics platform for Eightfold’s clients. It delivers analytics and enhanced insights in regards to the buyer’s Expertise Acquisition, Expertise Administration pipelines, and rather more. Clients can even implement their very own customized dashboards in QuickSight. As a part of the Expertise Intelligence Platform Eightfold additionally exposes a knowledge hub the place every buyer can entry their Amazon Redshift-based knowledge warehouse and carry out advert hoc queries in addition to schedule queries for reporting and knowledge export. Moreover, clients who’ve their very own in-house analytics infrastructure can combine their very own analytics options with Eightfold Expertise Intelligence Platform by instantly connecting to the Redshift knowledge warehouse provisioned for them. Doing this offers them entry to their uncooked analytics knowledge, which might then be built-in into their analytics infrastructure regardless of the expertise stack they use.

Eightfold offers this analytics expertise to a whole lot of shoppers immediately. Securing buyer knowledge is a high precedence for Eightfold. The corporate requires the best safety requirements when implementing a multi-tenant analytics platform on Amazon Redshift.

The Eightfold Expertise Intelligence Platform integrates with Amazon Redshift metadata safety to implement visibility of knowledge catalog itemizing of names of databases, schemas, tables, views, saved procedures, and capabilities in Amazon Redshift.

On this publish, we focus on how the Eightfold Expertise Lake system crew carried out the Amazon Redshift metadata safety function of their multi-tenant atmosphere to allow entry controls for the database catalog. By linking entry to business-defined entitlements, they can implement knowledge entry insurance policies.

Amazon Redshift safety controls addresses limiting knowledge entry to customers who’ve been granted permission. This publish discusses limiting itemizing of knowledge catalog metadata as per the granted permissions.

The Eightfold crew wanted to develop a multi-tenant software with the next options:

• Implement visibility of Amazon Redshift objects on a per-tenant foundation, so that every tenant can solely view and entry their very own schema
• Implement tenant isolation and safety in order that tenants can solely see and work together with their very own knowledge and objects

Metadata safety in Amazon Redshift

Amazon Redshift is a completely managed, petabyte-scale knowledge warehouse service within the cloud. Many shoppers have carried out Amazon Redshift to assist multi-tenant functions. One of many challenges with multi-tenant environments is that database objects are seen to all tenants despite the fact that tenants are solely approved to entry sure objects. This visibility creates knowledge privateness challenges as a result of many purchasers need to conceal objects that tenants can’t entry.

The newly launched metadata safety function in Amazon Redshift allows you to conceal database objects from all different tenants and make objects solely seen to tenants who’re approved to see and use them. Tenants can use SQL instruments, dashboards, or reporting instruments, and likewise question the database catalog, however they’ll solely see applicable objects for which they’ve permissions to see.

Resolution overview

Exposing a Redshift endpoint to all of Eightfold’s clients as a part of the Expertise Lake endeavor concerned a number of design decisions that needed to be fastidiously thought of. Eightfold has a multi-tenant Redshift knowledge warehouse that had particular person buyer schemas for purchasers, which they may hook up with utilizing their very own buyer credentials to carry out queries on their knowledge. Information in every buyer tenant can solely be accessed by the shopper credentials that had entry to the shopper schema. Every buyer might entry knowledge below their analytics schema, which was named after the shopper. For instance, for a buyer named A, the schema identify can be A_analytics. The next diagram illustrates this structure.

Though buyer knowledge was secured by limiting entry to solely the shopper person, when clients used enterprise intelligence (BI) instruments like QuickSight, Microsoft Energy BI, or Tableau to entry their knowledge, the preliminary connection confirmed all the shopper schemas as a result of it was performing a catalog question (which couldn’t be restricted). Subsequently, Eightfold’s clients had issues that different clients might uncover that they had been Eightfold’s clients by merely making an attempt to hook up with Expertise Lake. This unrestricted database catalog entry posed a privateness concern to a number of Eightfold clients. Though this may very well be prevented by provisioning one Redshift database per buyer, that was a logistically tough and costly answer to implement.

The next screenshot reveals what a connection from QuickSight to our knowledge warehouse seemed like with out metadata safety turned on. All different buyer schemas had been uncovered despite the fact that the connection to QuickSight was made as customer_k_user.

Strategy for implementing metadata entry controls

To implement restricted catalog entry, and guarantee it labored with Expertise Lake, we cloned our manufacturing knowledge warehouse with all of the schemas and enabled the metadata safety flag within the Redshift knowledge warehouse by connecting to SQL instruments. After it was enabled, we examined the catalog queries by connecting to the information warehouse from BI instruments like QuickSight, Microsoft Energy BI, and Tableau and ensured that solely the shopper schemas present up because of the catalog question. We additionally examined by operating catalog queries after connecting to the Redshift knowledge warehouse from psql, to make sure that solely the shopper schema objects had been surfaced—It’s essential to validate that given tenants have entry to the Redshift knowledge warehouse instantly.

The metadata safety function was examined by first turning on metadata safety in our Redshift knowledge warehouse by connecting utilizing a SQL instrument or Amazon Redshift Question Editor v2.0 and issuing the next command:

ALTER SYSTEM SET metadata_security = TRUE;

Observe that the previous command is ready on the Redshift cluster degree or Redshift Serverless endpoint degree, which implies it’s utilized to all databases and schemas within the cluster or endpoint.

In Eightfold’s state of affairs, knowledge entry controls are already in place for every of the tenants for his or her respective database objects.

After turning on the metadata safety function in Amazon Redshift, Eightfold was capable of limit database catalog entry to solely present particular person buyer schemas for every buyer that was making an attempt to hook up with Amazon Redshift and additional validated by issuing a catalog question to entry schema objects as nicely.

We additionally examined by connecting by way of psql and making an attempt out varied catalog queries. All of them yielded solely the related buyer schema of the logged-in person because the end result. The next are some examples:

analytics=> choose * from pg_user;
usename | usesysid | usecreatedb | usesuper | usecatupd | passwd | valuntil | useconfig 
------------------------+----------+-------------+----------+-----------+----------+----------+-------------------------------------------
customer_k_user | 377 | f | f | f | ******** | | 
(1 row)

analytics=> choose * from information_schema.schemata;
catalog_name | schema_name | schema_owner | default_character_set_catalog | default_character_set_schema | default_character_set_name | sql_path 
--------------+----------------------+------------------------+-------------------------------+------------------------------+----------------------------+----------
analytics | customer_k_analytics | customer_k_user | | | | 
(1 row)

The next screenshot reveals the UI after metadata safety was enabled: solely customer_k_analytics is seen when connecting to the Redshift knowledge warehouse as customer_k_user.

This ensured that particular person buyer privateness was protected and elevated buyer confidence in Eightfold’s Expertise Lake.

Buyer suggestions

“Being an AI-first platform for purchasers to rent and develop folks to their highest potential, knowledge and analytics play a significant position within the worth offered by the Eightfold platform to its clients. We depend on Amazon Redshift as a multi-tenant Information Warehouse that gives wealthy analytics with knowledge privateness and safety by buyer knowledge isolation through the use of schemas. Along with the information being safe as at all times, we layered on Redshift’s new metadata entry management to make sure buyer schemas usually are not seen to different clients. This function really made Redshift the perfect alternative for a multi-tenant, performant, and safe Information Warehouse and is one thing we’re assured differentiates our providing to our clients.”

– Sivasankaran Chandrasekar, Vice President of Engineering, Information Platform at Eightfold AI

Conclusion

On this publish, we demonstrated how the Eightfold Expertise Intelligence Platform crew carried out a multi-tenant atmosphere for a whole lot of shoppers, utilizing the Amazon Redshift metadata safety function. For extra details about metadata safety, discuss with the Amazon Redshift documentation.

Check out the metadata safety function to your future Amazon Redshift implementations, and be happy to go away a remark about your expertise!

Concerning the authors

Arun Sudhir is a Workers Software program Engineer at Eightfold AI. He has greater than 15 years of expertise in design and improvement of backend software program techniques in corporations like Microsoft and AWS, and has a deep data of database engines like Amazon Aurora PostgreSQL and Amazon Redshift.

Rohit Bansal is an Analytics Specialist Options Architect at AWS. He focuses on Amazon Redshift and works with clients to construct next-generation analytics options utilizing AWS Analytics companies.

Anjali Vijayakumar is a Senior Options Architect at AWS specializing in EdTech. She is captivated with serving to clients construct well-architected options within the cloud.