It is a visitor publish co-authored with Kanehito Miyake, Engineer at Fujitsu Japan.
Fujitsu Restricted was established in Japan in 1935. At the moment, we’ve roughly 120,000 workers worldwide (as of March 2023), together with group firms. We develop enterprise in varied areas around the globe, beginning with Japan, and supply digital providers globally. To offer quite a lot of merchandise, providers, and options which might be higher suited to clients and society in every area, we’ve constructed enterprise processes and programs which might be optimized for every area and its market.
Nonetheless, in recent times, the IT market surroundings has modified drastically, and it has turn into troublesome for your complete group to reply flexibly to the person market scenario. Furthermore, we’re challenged not solely to revisit particular person merchandise, providers, and options, but in addition to reinvent total enterprise processes and operations.
To rework Fujitsu from an IT firm to a digital transformation (DX) firm, and to turn into a world-leading DX associate, Fujitsu has declared a shift to data-driven administration. We constructed the OneFujitsu program, which standardizes enterprise tasks and programs all through the corporate, together with the home and abroad group firms, and tackles the main transformation of your complete firm below this system.
To realize data-driven administration, we constructed OneData, a knowledge utilization platform used within the 4 world AWS Areas, which began operation in April 2022. As of November 2023, greater than 200 tasks and 37,000 customers have been onboarded. The platform consists of roughly 370 dashboards, 360 tables registered within the knowledge catalog, and 40 linked programs. The information measurement saved in Amazon Easy Storage Service (Amazon S3) exceeds 100 TB, together with knowledge processed to be used in every venture.
On this publish, we introduce our OneData initiative. We clarify how Fujitsu labored to unravel the aforementioned points and introduce an summary of the OneData design idea and its implementation. We hope this publish will present some steering for architects and engineers.
Challenges
Like many different firms battling knowledge utilization, Fujitsu confronted some challenges, which we talk about on this part.
Siloed knowledge
In Fujitsu’s lengthy historical past, we restructured organizations by merging affiliated firms into Fujitsu. Though organizational integration has progressed, there are nonetheless many programs and mechanisms custom-made for particular person context. There are additionally many programs and mechanisms overlapping throughout totally different organizations. Because of this, it takes plenty of effort and time to find, search, and combine knowledge when analyzing your complete firm utilizing a standard normal. This example makes it troublesome for administration to understand enterprise traits and make selections in a well timed method.
Below these circumstances, the OneFujitsu program is designed have one system per one enterprise globally. Core programs resembling ERP and CRM are being built-in and unified with a purpose to not have silos. It’s going to make it simpler for customers to make the most of knowledge throughout totally different organizations for particular enterprise areas.
Nonetheless, to unfold a tradition of data-driven decision-making not solely in administration but in addition in each group, it’s essential to have a mechanism that allows customers to simply uncover varied kinds of knowledge in organizations, after which analyze the information rapidly and flexibly when wanted.
Excel-based knowledge utilization
Microsoft Excel is out there on virtually everybody’s PC within the firm, and it helps decrease the hurdles when beginning to make the most of knowledge. Nonetheless, Excel is principally designed for spreadsheets; it’s not designed for large-scale knowledge analytics and automation. Excel recordsdata are likely to include a combination of knowledge and procedures (capabilities, macros), and plenty of customers casually copy recordsdata for one-time use circumstances. It introduces complexity to maintain each knowledge and procedures updated. Moreover, it tends to require domain-specific data to handle the Excel recordsdata for particular person context.
For these causes, it was extraordinarily troublesome for Fujitsu to handle and make the most of knowledge at scale with Excel.
Answer overview
OneData defines three personas:
- Writer – This position contains the organizational and administration crew of programs that function knowledge sources. Obligations embody:
- Load uncooked knowledge from the information supply system on the acceptable frequency.
- Present and preserve updated with technical metadata for loaded knowledge.
- Carry out the cleaning course of and format conversion of uncooked knowledge as wanted.
- Grant entry permissions to knowledge based mostly on the requests from knowledge customers.
- Shopper – Shoppers are organizations and tasks that use the information. Obligations embody:
- Search for the information for use from the technical knowledge catalog and request entry to the information.
- Deal with the method and conversion of knowledge right into a format appropriate for their very own use (resembling fact-dimension) with granted referencing permissions.
- Configure enterprise intelligence (BI) dashboards to offer data-driven insights to end-users focused by the buyer’s venture.
- Use the most recent knowledge revealed by the writer to replace knowledge as wanted.
- Promote and increase using databases.
- Basis – This position encompasses the information steward and governance crew. Obligations embody:
- Present a preprocessed, generic dataset of knowledge generally utilized by many customers.
- Handle and information metrics for the standard of knowledge revealed by every writer.
Every position has sub-roles. For instance, the buyer position has the next sub-roles with totally different obligations:
- Information engineer – Create knowledge course of for evaluation
- Dashboard developer – Create a BI dashboard
- Dashboard viewer – Monitor the BI dashboard
The next diagram describes how OneData platform works with these roles.
Let’s take a look at the important thing elements of this structure in additional element.
Writer and shopper
Within the OneData platform, the writer is per every knowledge supply system, and the buyer is outlined per every knowledge utilization venture. OneData offers an AWS account for every.
This allows the writer to cleanse knowledge and the buyer to course of and analyze knowledge at scale. As well as, by correctly separating knowledge and processing, it turns into easy for the groups and organizations to share, handle, and inherit processes that have been historically confined to particular person PCs.
Basis
When the groups don’t have a strong sufficient skillset, it will probably require extra time to mannequin and course of knowledge, and trigger longer latency and decrease knowledge high quality. It could actually additionally contribute to decrease utilization by end-users. To deal with this, the muse position offers an already processed dataset as a generic knowledge mannequin for knowledge generally use circumstances utilized by many customers. This allows high-quality knowledge obtainable to every shopper. Right here, the muse position takes the lead in compiling the data of area specialists and making knowledge appropriate for evaluation. Additionally it is an efficient strategy that eliminates duplicates for customers. As well as, the muse position displays the state of the metadata, knowledge high quality indicators, knowledge permissions, data classification labels, and so forth. It’s essential in knowledge governance and knowledge administration.
BI and visualization
Particular person customers have a devoted area in a BI device. Up to now, if customers needed to transcend easy knowledge visualization utilizing Excel, they needed to construct and preserve their very own BI instruments, which induced silos. By unifying these BI instruments, OneData lowers the problem for customers to make use of BI instruments, and centralizes operation and upkeep, attaining optimization on a company-wide scale.
Moreover, to maintain portability between BI instruments, OneData recommends customers rework knowledge throughout the shopper AWS account as a substitute of reworking knowledge within the BI device. With this strategy, BI device hundreds knowledge from AWS Glue Information Catalog tables by an Amazon Athena JDBC/ODBC driver with none additional transformations.
Deployment and operational excellence
To offer OneData as a standard service for Fujitsu and group firms around the globe, Regional OneData has been deployed in a number of places. Regional OneData represents a unit of system configurations, and is designed to offer decrease community latency for platform customers, and be optimized for native languages, working hours for system operations and assist, and region-specific authorized restrictions, resembling knowledge residency and private data safety.
The Regional Operations Unit (ROU), a digital group that brings collectively members from every area, is answerable for working regional OneData in every of those areas. OneData HQ is answerable for supervising these ROUs, in addition to planning and managing your complete OneData.
As well as, we’ve a specifically positioned OneData known as World OneData, the place world knowledge utilization spans every area. Solely the correctly cleansed and sanitized knowledge is transferred between every Regional OneData and World OneData.
Techniques resembling ERP and CRM are accumulating knowledge as a writer for World OneData, and the dashboards for executives in varied areas to watch enterprise circumstances with world metrics are additionally appearing as a shopper for World OneData.
Technical ideas
On this part, we talk about a few of the technical ideas of the answer.
Massive scale multi-account
We have now adopted a multi-account technique to offer AWS accounts for every venture. Many publishers and customers are already onboarded into OneData, and the quantity is anticipated to extend sooner or later. With this technique, future utilization growth at scale might be achieved with out affecting the customers.
Additionally, this technique allowed us to have clear boundaries in safety, prices, and repair quotas for every AWS service.
All of the AWS accounts are deployed and managed by AWS Organizations and AWS Management Tower.
Serverless
Though we offer unbiased AWS accounts for every writer and shopper, each operational prices and useful resource prices can be huge if we accommodated particular person person requests, resembling, “I need a digital machine or RDBMS to run particular instruments for knowledge processing.” To keep away from such steady operational and useful resource prices, we’ve adopted AWS serverless providers for all of the computing sources obligatory for our actions as a writer and shopper.
We use AWS Glue to preprocess, cleanse, and enrich knowledge. Optionally, AWS Lambda or Amazon Elastic Container Service (Amazon ECS) with AWS Fargate may also be used based mostly on preferences. We enable customers to arrange AWS Step Features for orchestration and Amazon CloudWatch for monitoring. As well as, we offer Amazon Aurora Serverless PostgreSQL as normal for customers, to fulfill their wants for knowledge processing with extract, load, and rework (ELT) jobs. With this strategy, solely the buyer who requires these providers will incur expenses based mostly on utilization. We’re capable of make the most of decrease operational and useful resource prices because of the distinctive good thing about serverless (or extra precisely, pay-as-you-go) providers.
AWS offers many serverless providers, and OneData has built-in them to offer scalability that enables lively customers to rapidly present the required functionality as wanted, whereas minimizing the price for non-frequent customers.
Information possession and entry management
In OneData, we’ve adopted a knowledge mesh structure the place every writer maintains possession of knowledge in a distributed and decentralized method. When the buyer discovers the information they need to use, they request entry from the writer. The writer accepts the request and grants permissions solely when the request meets their very own standards. With the AWS Glue Information Catalog and AWS Lake Formation, there isn’t a have to replace S3 bucket insurance policies or AWS Id and Entry Administration (IAM) insurance policies each time we enable entry for particular person knowledge on an S3 knowledge lake, and we will effortlessly grant the required permissions for the databases, tables, columns, and rows when wanted.
Conclusion
Because the launch of OneData in April 2022, we’ve been persistently finishing up instructional actions to increase the variety of customers and introducing success tales on our portal website. Because of this, we’ve been selling change administration throughout the firm and are actively using knowledge in every division. Regional OneData is being rolled out progressively, and we plan to additional increase the size of use sooner or later.
With its world growth, the event of fundamental capabilities as a knowledge utilization platform will attain a milestone. As we transfer ahead, it will likely be essential to make it possible for OneData platform is used successfully all through Fujitsu, whereas incorporating new applied sciences associated to knowledge evaluation as acceptable. For instance, we’re getting ready to offer extra superior machine studying capabilities utilizing Amazon SageMaker Studio with OneData customers and investigating the applicability of AWS Glue Information High quality to scale back the guide high quality monitoring efforts. Moreover, we’re at present within the technique of implementing Amazon DataZone by varied initiatives and efforts, resembling verifying its performance and analyzing the way it can function whereas bridging the hole between OneData’s current processes and to the perfect course of we’re aiming for beliefs.
We have now had the chance to debate knowledge utilization with varied companions and clients and though particular person challenges could differ in measurement and its context, the problems that we’re at present making an attempt to unravel with OneData are frequent to a lot of them.
This publish describes solely a small portion of how Fujitsu tackled challenges utilizing the AWS Cloud, however we hope the publish gives you some inspiration to unravel your personal challenges.
Concerning the Writer
Kanehito Miyake is an engineer at Fujitsu Japan and in command of OneData’s resolution and cloud structure. He spearheaded the architectural research of the OneData venture and contributed enormously to selling knowledge utilization at Fujitsu together with his experience. He loves rockfish fishing.
Junpei Ozono is a Go-to-market Information & AI options architect at AWS in Japan. Junpei helps clients’ journeys on the AWS Cloud from Information & AI elements and guides them to design and develop data-driven architectures powered by AWS providers.
