Invoke AWS Lambda features from cross-account Amazon Kinesis Information Streams

A multi-account structure on AWS is crucial for enhancing safety, compliance, and useful resource administration by isolating workloads, enabling granular value allocation, and facilitating collaboration throughout distinct environments. It additionally mitigates dangers, improves scalability, and permits for superior networking configurations.

In a streaming structure, you might have occasion producers, stream storage, and occasion shoppers in a single account or unfold throughout completely different accounts relying on what you are promoting and IT necessities. For instance, your organization could wish to centralize its clickstream knowledge or log knowledge from a number of completely different producers throughout completely different accounts. Information shoppers from advertising, product engineering, or analytics require entry to the identical streaming knowledge throughout accounts, which requires the power to ship a multi-account streaming structure.

To construct a multi-account streaming structure, you should utilize Amazon Kinesis Information Streams because the stream storage and AWS Lambda because the occasion shopper. Amazon Kinesis Information Streams permits real-time processing of streaming knowledge at scale. When built-in with Lambda, it permits for serverless knowledge processing, enabling you to investigate and react to knowledge streams in actual time with out managing infrastructure. This integration helps numerous use circumstances, together with real-time analytics, log processing, Web of Issues (IoT) knowledge ingestion, and extra, making it helpful for companies requiring well timed insights from their streaming knowledge. On this publish, we show how one can course of knowledge ingested right into a stream in a single account with a Lambda perform in one other account.

The current launch of Kinesis Information Streams help for resource-based insurance policies permits invoking a Lambda from one other account. With a resource-based coverage, you’ll be able to specify AWS accounts, AWS Id and Entry Administration (IAM) customers, or IAM roles and the precise Kinesis Information Streams actions for which you wish to grant entry. After entry is granted, you’ll be able to configure a Lambda perform in one other account to begin processing the information stream belonging to your account. This reduces value and simplifies the information processing pipeline, since you now not have to repeat streaming knowledge utilizing Lambda features in each accounts. Sharing entry to your knowledge streams or registered shoppers doesn’t incur extra expenses to your account. Cross-account utilization of Kinesis Information Streams assets will proceed to be billed to the useful resource house owners.

On this publish, we use Kinesis Information Streams with enhanced fan-out characteristic, empowering shoppers with devoted learn throughput tailor-made to their functions. By default, Kinesis Information Streams presents shared learn throughput of two MB/sec per shard throughout shoppers, however with enhanced fan-out, every shopper can get pleasure from devoted throughput of two MB/sec per shard. This flexibility means that you can seamlessly adapt Kinesis Information Streams to your particular necessities, selecting between enhanced fan-out for devoted throughput or shared throughput in keeping with your wants.

Answer overview

For our answer, we deploy Kinesis Information Streams in Account 1 and Lambda as the patron in Account 2 to obtain knowledge from the information stream. The next diagram illustrates the high-level structure.

The setup requires the next key components:

• Kinesis knowledge stream in Account 1 and Lambda perform in Account 2
• Kinesis Information Streams useful resource insurance policies in Account 1, permitting a cross-account Lambda execution function to carry out operations on the Kinesis knowledge stream
• A Lambda execution function in Account 2 and an enhanced fan-out shopper useful resource coverage in Account 1, permitting the cross-account Lambda execution function to carry out operations on the Kinesis knowledge stream

For the setup, you employ three AWS CloudFormation templates to create the important thing assets:

• CloudFormation template 1 creates the next key assets in Account 1:
  • Kinesis knowledge stream
  • Kinesis knowledge stream enhanced fan-out shopper
• CloudFormation template 2 creates the next key assets in Account 2:
  • Client Lambda perform
  • Client Lambda perform execution function
• CloudFormation template 3 creates the next useful resource in Account 2:
  • Client Lambda perform occasion supply mapping

The answer helps single-Area deployment, and the CloudFormation templates have to be deployed in the identical Area throughout completely different AWS accounts. On this answer, we use Kinesis Information Streams enhanced fan-out, which is a greatest apply for deploying architectures requiring massive throughput throughout a number of shoppers. Full the steps within the following sections to deploy this answer.

Conditions

You need to have two AWS accounts and the required permissions to run a CloudFormation template to create the companies talked about within the answer structure. You additionally want the AWS Command Line Interface (AWS CLI) put in, model 2.15 and above.

Launch CloudFormation template 1

Full the next steps to launch the primary CloudFormation template:

1. Check in to the AWS Administration Console as Account 1 and choose the suitable AWS Area.
2. Obtain and launch CloudFormation template 1 the place you wish to deploy your Kinesis knowledge stream.
3. For LambdaConsumerAccountId, enter your Lambda shopper account ID and click on submit. The CloudFormation template deployment will take a couple of minutes to finish.
4. When the stack is full, on the AWS CloudFormation console, navigate to the stack Outputs tab and duplicate the values of following parameters:
   • KinesisStreamArn
   • KinesisStreamEFOConsumerArn
   • KMSKeyArn

You have to these values in later steps.

Launch CloudFormation template 2

Full the next steps to launch the second CloudFormation template:

1. Check in to the console as Account 2 and choose the suitable Area.
2. Obtain and launch CloudFormation template 2 the place you wish to host the Lambda shopper.
3. Present the next enter parameters captured from the earlier step:
   • KinesisStreamArn
   • KinesisStreamEFOConsumerArn
   • KMSKeyArn

The CloudFormation template creates the next key assets:

• Lambda shopper
• Lambda execution function

The Lambda perform’s execution function is an IAM function that grants the perform permission to entry AWS companies and assets. Right here, you create a Lambda execution function that has the required Kinesis Information Streams and Lambda invocation permissions.

The CloudFormation template deployment will take a couple of minutes to finish.

1. When the stack is full, on the AWS CloudFormation console, navigate to the stack Outputs tab and duplicate the values of following parameters:
   • KinesisStreamCreateResourcePolicyCommand
   • KinesisStreamEFOConsumerCreateResourcePolicyCommand
2. Run the next AWS CLI instructions in Account 1 utilizing AWS CloudShell. We suggest utilizing CloudShell as a result of it should have the newest model of the AWS CLI and keep away from any type of failures.
   • KinesisStreamCreateResourcePolicyCommand – This creates the useful resource coverage in Account 1 for Kinesis Information Stream. The next is a pattern useful resource coverage:

     {
     "Model": "2012-10-17",
     "Assertion": [
     {
     "Sid": "StreamEFOReadStatementID",
     "Effect": "Allow",
     "Principal": {
     "AWS": [
     "arn:aws:iam::<AWS Lambda - Consumer account id>:role/kds-cross-account-stream-consumer-lambda-execution-role"
     ]
     },
     "Motion": [
     "kinesis:DescribeStreamSummary",
     "kinesis:ListShards",
     "kinesis:DescribeStream",
     "kinesis:GetRecords",
     "kinesis:GetShardIterator"
     ],
     "Useful resource": "arn:aws:kinesis:<area id>:<Account 1 - Amazon KDS account id>:stream/kds-cross-account-stream"
     }
     ]
     }

   • KinesisStreamEFOConsumerCreateResourcePolicyCommand – This creates the useful resource coverage for the improved fan-out shopper for the Kinesis knowledge stream in Account 1. The next is a pattern useful resource coverage:

     {
     "Model": "2012-10-17",
     "Assertion": [
     {
     "Sid": "ConsumerEFOReadStatementID",
     "Effect": "Allow",
     "Principal": {
     "AWS": [
     " arn:aws:iam::<AWS Lambda - Consumer account id>:role/kds-cross-account-stream-consumer-lambda-execution-role"
     ]
     },
     "Motion": [
     "kinesis:DescribeStreamConsumer",
     "kinesis:SubscribeToShard"
     ],
     "Useful resource": "arn:aws:kinesis:<area id>:<Account 1 - Amazon KDS account id>:stream/kds-cross-account-stream/shopper/kds-cross-account-stream-efo-consumer:1706616477"
     }
     ]
     }

You may also entry this coverage on the Kinesis Information Streams console, beneath Enhanced fan-out, Client identify, and Client sharing resource-based coverage.

Launch CloudFormation template 3

Now that you’ve created useful resource insurance policies in Account 1 for the Kinesis knowledge stream and its enhanced fan-out shopper, you’ll be able to create Lambda occasion supply mapping for the patron Lambda perform in Account 2. Full the next steps:

1. Check in to the console as Account 2 and choose the suitable Area.
2. Obtain and launch CloudFormation template 3 to replace the stack you created utilizing CloudFormation template 2.

The CloudFormation template creates the Lambda occasion supply mapping.

Validate the answer

At this level, the deployment is full. A Kinesis knowledge stream is obtainable to eat the messages and a Lambda perform receives these messages within the vacation spot account. To ship pattern messages to the information stream in Account 1, run the next AWS CLI command utilizing CloudShell:

aws kinesis put-record --stream-name kds-cross-account-stream --data sampledatarecord --partition-key samplepartitionkey3 --region <area id>

The Lambda perform in Account 2 is ready to obtain the messages, and it is best to be capable to confirm the identical utilizing Amazon CloudWatch logs:

1. On the CloudWatch console, select Log teams within the navigation pane.
2. Find the log group /aws/lambda/kds-cross-account-stream-efo-consumer.
3. Select Search log group to view the related log messages. The next is an instance message:

   "Information": [
   {
   "kinesis": {
   "kinesisSchemaVersion": "1.0",
   "partitionKey": "samplepartitionkey3",
   "sequenceNumber": "49648798411111169765201534322676841348246990356337393698",
   "data": "sampledatarecord",
   "approximateArrivalTimestamp": 1706623274.658
   },

Clean up

It’s always a good practice to clean up all the resources you created as part of this post to avoid any additional cost.

To clean up your resources, delete the respective CloudFormation stacks from Accounts 1 and 2, and stop the producer from pushing events to the Kinesis data stream. This makes sure that you are not charged unnecessarily.

Summary

In this post, we demonstrated how to configure a cross-account Lambda integration with Kinesis Data Streams using AWS resource-based policies. This enables processing of data ingested into a stream within one AWS account through a Lambda function located in another account. To support customers who use a Kinesis data stream in their central account and have multiple consumers reading data from it, we have used the Kinesis Data Streams enhanced fan-out feature.

To get started, open the Kinesis Data Streams console or use the new API PutResourcePolicy to attach a resource policy to your data stream or consumer.

About the authors

Pratik Patel is Sr. Technical Account Manager and streaming analytics specialist. He works with AWS customers and provides ongoing support and technical guidance to help plan and build solutions using best practices and proactively keep customers’ AWS environments operationally healthy.

Amar is a Senior Solutions Architect at Amazon AWS in the UK. He works across power, utilities, manufacturing and automotive customers on strategic implementations, specializing in using AWS Streaming and advanced data analytics solutions, to drive optimal business outcomes.