Improve monitoring and debugging for AWS Glue jobs utilizing new job observability metrics: Half 2

Monitoring information pipelines in actual time is crucial for catching points early and minimizing disruptions. AWS Glue has made this extra easy with the launch of AWS Glue job observability metrics, which give priceless insights into your information integration pipelines constructed on AWS Glue. Nevertheless, you would possibly want to trace key efficiency indicators throughout a number of jobs. On this case, a dashboard that may visualize the identical metrics with the power to drill down into particular person points is an efficient resolution to observe at scale.

This submit, walks via the way to combine AWS Glue job observability metrics with Grafana utilizing Amazon Managed Grafana. We talk about the forms of metrics and charts out there to floor key insights together with two use circumstances on monitoring error courses and throughput of your AWS Glue jobs.

Resolution overview

Grafana is an open supply visualization software that permits you to question, visualize, alert on, and perceive your metrics irrespective of the place they’re saved. With Grafana, you’ll be able to create, discover, and share visually wealthy, data-driven dashboards. The brand new AWS Glue job observability metrics might be effortlessly built-in with Grafana for real-time monitoring objective. Metrics like employee utilization, skewness, I/O charge, and errors are captured and visualized in easy-to-read Grafana dashboards. The combination with Grafana supplies a versatile solution to construct customized views of pipeline well being tailor-made to your wants. Observability metrics open up monitoring capabilities that weren’t potential earlier than for AWS Glue. Corporations counting on AWS Glue for crucial information integration pipelines can have better confidence that their pipelines are operating effectively.

AWS Glue job observability metrics are emitted as Amazon CloudWatch metrics. You possibly can provision and handle Amazon Managed Grafana, and configure the CloudWatch plugin for the given metrics. The next diagram illustrates the answer structure.

Implement the answer

Full following steps to arrange the answer:

1. Arrange an Amazon Managed Grafana workspace.
2. Sign up to your workspace.
3. Select Administration.
4. Select Add new information supply.
5. Select CloudWatch.
6. For Default Area, choose your most popular AWS Area.
7. For Namespaces of Customized Metrics, enter Glue.
8. Select Save & take a look at.

Now the CloudWatch information supply has been registered.

9. Copy the information supply ID from the URL https://g-XXXXXXXXXX.grafana-workspace.<area>.amazonaws.com/datasources/edit/<data-source-ID>/.

The following step is to organize the JSON template file.

10. Obtain the Grafana template.
11. Change <data-source-id> within the JSON file together with your Grafana information supply ID.

Lastly, configure the dashboard.

12. On the Grafana console, select Dashboards.
13. Select Import on the New menu.
14. Add your JSON file, and select Import.

The Grafana dashboard visualizes AWS Glue observability metrics, as proven within the following screenshots.

The pattern dashboard has the next charts:

• [Reliability] Job Run Errors Breakdown
• [Throughput] Bytes Learn & Write
• [Throughput] Information Learn & Write
• [Resource Utilization] Employee Utilization
• [Job Performance] Skewness
• [Resource Utilization] Disk Used (%)
• [Resource Utilization] Disk Obtainable (GB)
• [Executor OOM] OOM Error Rely
• [Executor OOM] Heap Reminiscence Used (%)
• [Driver OOM] OOM Error Rely
• [Driver OOM] Heap Reminiscence Used (%)

Analyze the causes of job failures

Let’s strive analyzing the causes of job run failures of the job iot_data_processing.

First, have a look at the pie chart [Reliability] Job Run Errors Breakdown. This pie chart shortly identifies which errors are most typical.

Then filter with the job identify iot_data_processing to see the frequent errors for this job.

We will observe that almost all (75%) of failures have been on account of glue.error.DISK_NO_SPACE_ERROR.

Subsequent, have a look at the road chart [Resource Utilization] Disk Used (%) to grasp the driving force’s used disk house throughout the job runs. For this job, the inexperienced line reveals the driving force’s disk utilization, and the yellow line reveals the common of the executors’ disk utilization.

We will observe that there have been thrice when 100% of disk was utilized in executors.

Subsequent, have a look at the road chart [Throughput] Information Learn & Write to see whether or not the information quantity was modified and whether or not it impacted disk utilization.

The chart reveals that round 4 billion data have been learn in the beginning of this vary; nevertheless, round 63 billion data have been learn on the peak. Which means the incoming information quantity has considerably elevated, and prompted native disk house scarcity within the employee nodes. For such circumstances, you’ll be able to enhance the variety of staff, allow auto scaling, or select bigger employee varieties.

After implementing these ideas, we will see decrease disk utilization and a profitable job run.

(Optionally available) Configure cross-account setup

We will optionally configure a cross-account setup. Cross-account metrics depend upon CloudWatch cross-account observability. On this setup, we anticipate the next atmosphere:

• AWS accounts aren’t managed in AWS Organizations
• You will have two accounts: one account is used because the monitoring account the place Grafana is positioned, one other account is used because the supply account the place the AWS Glue-based information integration pipeline is positioned

To configure a cross-account setup for this atmosphere, full the next steps for every account.

Monitoring account

Full the next steps to configure your monitoring account:

1. Sign up to the AWS Administration Console utilizing the account you’ll use for monitoring.
2. On the CloudWatch console, select Settings within the navigation pane.
3. Underneath Monitoring account configuration, select Configure.
4. For Choose information, select Metrics.
5. For Checklist supply accounts, enter the AWS account ID of the supply account that this monitoring account will view.
6. For Outline a label to determine your supply account, select Account identify.
7. Select Configure.

Now the account is efficiently configured as a monitoring account.

8. Underneath Monitoring account configuration, select Sources to hyperlink accounts.
9. Select Any account to get a URL for organising particular person accounts as supply accounts.
10. Select Copy URL.

You’ll use the copied URL from the supply account within the subsequent steps.

Supply account

Full the next steps to configure your supply account:

1. Sign up to the console utilizing your supply account.
2. Enter the URL that you just copied from the monitoring account.

You possibly can see the CloudWatch settings web page, with some info stuffed in.

3. For Choose information, select Metrics.
4. Don’t change the ARN in Enter monitoring account configuration ARN.
5. The Outline a label to determine your supply account part is pre-filled with the label selection from the monitoring account. Optionally, select Edit to alter it.
6. Select Hyperlink.
7. Enter Verify within the field and select Verify.

Now your supply account has been configured to hyperlink to the monitoring account. The metrics emitted within the supply account will present on the Grafana dashboard within the monitoring account.

To study extra, see CloudWatch cross-account observability.

Concerns

The next are some concerns when utilizing this resolution:

• Grafana integration is outlined for real-time monitoring. When you have a fundamental understanding of your jobs, will probably be easy so that you can monitor efficiency, errors, and extra on the Grafana dashboard.
• Amazon Managed Grafana is dependent upon AWS IAM Determine Middle. This implies it’s essential to handle single sign-on (SSO) customers individually, not simply AWS Identification and Entry Administration (IAM) customers and roles. It additionally requires one other sign-in step from the AWS console. The Amazon Managed Grafana pricing mannequin is dependent upon an energetic consumer license per workspace. Extra customers may cause extra costs.
• Graph traces are visualized per job. If you wish to see the traces throughout all the roles, you’ll be able to select ALL within the management.

Conclusion

AWS Glue job observability metrics provide a strong new functionality for monitoring information pipeline efficiency in actual time. By streaming key metrics to CloudWatch and visualizing them in Grafana, you achieve extra fine-grained visibility that wasn’t potential earlier than. This submit confirmed how easy it’s to allow observability metrics and combine the information with Grafana utilizing Amazon Managed Grafana. We explored the totally different metrics out there and the way to construct custom-made Grafana dashboards to floor actionable insights.

Observability is now a vital a part of strong information orchestration on AWS. With the power to observe information integration traits in actual time, you’ll be able to optimize prices, efficiency, and reliability.

Concerning the Authors

Noritaka Sekiyama is a Principal Huge Information Architect on the AWS Glue crew. He’s answerable for constructing software program artifacts to assist clients. In his spare time, he enjoys biking together with his new highway bike.

Xiaoxi Liu is a Software program Improvement Engineer on the AWS Glue crew. Her ardour is constructing scalable distributed techniques for effectively managing massive information on the cloud, and her concentrations are distributed system, massive information, and cloud computing.

Akira Ajisaka is a Senior Software program Improvement Engineer on the AWS Glue crew. He likes open supply software program and distributed techniques. In his spare time, he enjoys taking part in arcade video games.

Shenoda Guirguis is a Senior Software program Improvement Engineer on the AWS Glue crew. His ardour is in constructing scalable and distributed information infrastructure and processing techniques. When he will get an opportunity, Shenoda enjoys studying and taking part in soccer.

Sean Ma is a Principal Product Supervisor on the AWS Glue crew. He has an 18-year observe file of innovating and delivering enterprise merchandise that unlock the facility of knowledge for customers. Outdoors of labor, Sean enjoys scuba diving and school soccer.

Mohit Saxena is a Senior Software program Improvement Supervisor on the AWS Glue crew. His crew focuses on constructing distributed techniques to allow clients with interactive and easy to make use of interfaces to effectively handle and rework petabytes of knowledge seamlessly throughout information lakes on Amazon S3, databases and data-warehouses on cloud.