However this results in our fifth downside, which is similar-yet-different information units. Why are there multiples? Which one ought to I take advantage of? Is that this information set nonetheless maintained, or is it a zombie information set that’s nonetheless frequently up to date however with out anybody overseeing it? The issue involves a head when you could have necessary computations that disagree with one another, as a result of counting on information units that needs to be similar however are usually not. Offering conflicting experiences, dashboards, or metrics to prospects will lead to a lack of belief, and in a worst-case state of affairs, lack of enterprise and even authorized motion.
Even in the event you kind out all of those issues—lowering latency, lowering prices, eradicating duplicate pipelines and information units, and eliminating break-fix work—you continue to haven’t offered something that operations can use. They’re nonetheless on their very own, upstream of your ETLs, as a result of all the cleansing, structuring, transforming, and distribution work is just actually helpful for these within the information analytics area.
Shift left for a headless information structure
Constructing a headless information structure requires a rethink of how we flow into, share, and handle information in our organizations—a shift left. We extract the ETL->bronze->silver work from downstream and put it upstream inside our information merchandise, a lot nearer to the supply.
