We are thrilled to witness general availability announcement of Microsoft Fabric today! This also marks 1 year of amazing journey with Microsoft Fabric. Thanks to product engineering, partner & field teams and internal implementation teams at Microsoft, customers and prospects who engaged with us on early evaluations, pilots and brainstorming; this has really helped Sonata in early learning & adoption, contributing to the product, develop subject matter experts, building our offerings and accelerators.
Excited to be a featured partner for Microsoft Fabric!
Enterprises have already started realizing the immense need of complete and high quality, connected data ecosystem for AI innovations to succeed.
Providing connected data foundation for analytics and AI to our customers is a key focus area for us. Microsoft Fabric provides a perfect technology stack to achieve this.
Real world objects are inherently interrelated. For a true reflection and complete insights, data need to be connected too.
Silos of information can’t provide real world reflections. Because real world entities and events have complex and many-to-many relationship. Insights gathered, inferences derived or AI predictions from silo datasets are at the risk of being inaccurate, incomplete, or impossible!
Here is an example. A retail store chain struggled to estimate their assembled product delivery date to customers at the time of order. Analytics specialists experimented with number of forecasting models to predict the order fulfilment and delivery estimates with limited success.
One of key challenges that prevented the predictability is disconnected information systems. This is a quite common challenge among the enterprises and bridging the silos is often a complex and time-consuming project. Reflecting an integrated picture of the business is key to solving such issues.
Building connected data model for each use case will be time consuming and expensive.
When there are several such needs, implementing integrations for each use case will be slow and expensive. Building connected data foundation on a SaaS platform like Microsoft Fabric is a very relevant solution to solve multiple such challenges quickly and cost-effectively.
Is lack of connected information system the only problem here? In the above example, where the product delivery involved assembling various components from existing inventory or from procurement channels, finding nearest or fastest sources of the parts, shipment routes, predictability of order fulfilment is dependent on variability of many sub-processes. It could appear to be process related challenge but connected data and insights has significant role to play in those sub-processes too. For example:
- Unpredictable demands
- Complex supply chain processes.
Demand prediction involves customer and market intelligence. Connecting complex supply chain process involves connecting their data in a business domain context.
Predictability often involves data universe beyond the organizational boundary.
Predicting product demands requires customer intelligence, market intelligence, competition, pricing, discounts and many more. Product or brand’s social sentiment can bring a drastic change in the demand too. Most of these are not internal data sources. Expanding the scope of information and intelligence hence becomes essential to solving some of the key business problems. This is where, what seems to be not related to information challenge, is actually related to it.
Building data mesh is a great way to connect disparate datasets while they physically can’t be all integrated or stored together.
Fabric brings number of key features necessary to build connected data mesh.
Plugging in required dataset into contexts – internal or external through connectors or shortcuts helps quickly build a mesh.
- Shortcuts across workspaces in Fabric allows intra-company datasets connected.
- Cross-platform shortcuts and rich set of data connectors in Fabric allows building an enterprise data model beyond the boundary of technology and systems stack.
- Sonata’s Lightning Data Suite further helps map the datasets into connected domain models and intelligently identify potential relationship across the disparate datasets. More on this in the last section of this blog
- Fabric’s focus is on enabling open and connected data ecosystem. Specialized features such as Synapse Link for Dataverse further simplify and speed up data integration into OneLake.
Connecting business processes involve connecting their data in a business domain context
While connecting datasets coming from supply chain, inventory locations, product part compatibility, pricing and shipment information, relationship between these datasets would seem really complex and hard to define unless they are put into a context of an integrated domain context. Hence, domain modeling is an important aspect of building a connected data foundation.
There are more reasons for organizing data into a connected domain models
- Domain model simplifies self-service analytics
- It broadens the audience for usage of data and helps adoption & ROI
- Improves usage of co-pilot and NLP based data exploration
- Reduces load on Engineering teams so that they can focus on more innovations and new build rather than getting caught up into fulfilling ad hoc insight needs
- Greatly improves agility. Users could get their insights in seconds rather than waiting on SME team to deliver it through build cycles
- Improves role experience by providing relevant perspectives of the enterprise data model
- Improves security. Fragmented data pieces could fall through the loopholes. Data entities well mapped to the functional domain can be better secured and governed.
Sonata’s domain data model approach is precisely for building that business friendly data structure on a connected data mesh in Fabric platform.
Build connected data foundation using Sonata’s Lightning Data Suite (LDS) for Fabric
LDS connects data silos in many ways.
- Domain data models bring functional connections among system generated silos.
- Intelligent mappings connect source data structures to domain models
- Labeling & tagging, label classifications and keyword synonyms establish similarity
- Semantic connector discover hidden connections across structured & unstructured data elements
Vision of Lightning Data Suite is to optimize data engineering and sustenance effort in order to leverage more bandwidth towards deriving insights and AI innovations. LDS brings capabilities necessary to build connected data foundation faster.
Domain data dictionary
This transforms system data model to business-friendly semantic data model for analytics and co-pilot, natural language querying and summarization in Fabric Power BI
Automatic Data Pipelines
Using the business dictionary-based metadata, LDS Automatic data pipeline does the job of hundreds of typical data flows or data pipelines that ingest data into data lake and build delta parquet based Lakehouse tables in Fabric.
Semantic connector
Connected data ecosystem is not only about bringing datasets from heterogeneous systems and sources; unless the datasets are meaningfully connected, real data mesh is not formed. LDS Semantic connector intelligently establishes relationships across heterogenous datasets coming together into OneLake.
Sonata LDS also brings various horizontal and vertical use case solutions based on our market research of key gaps and pain-points.
We are helping customers globally with their Fabric based data modernization journey.
We have been doing Fabric briefing sessions to help enterprise stakeholders explore what is in Fabric and how it helps with data modernization. For an enterprise to take a step back and revisit data strategy for the era of transformations with AI, customers leverage joint envisioning sessions. For further deep dive analysis of applicability and validate the possibilities with Fabric based solutions, we provide assessment and Proof of concept services. Data modernization using Sonata’s lightning data suite helps modern enterprise realize business friendly, connected data foundation on Fabric for agile and cost-efficient data modernization.
Behalf of team Sonata, I sincerely thank our early adoption customers for their invaluable questions and inputs, Microsoft Fabric Product Engineering teams for all the guidance and collaboration, Microsoft Field teams for joint GTM efforts and partnership, Microsoft internal solution implementation teams for the opportunity and collaborative early adoption programs. This has helped us in building specializations, co-create and add value to Fabric product ecosystem. Past 12 months of journey with Microsoft Fabric has been amazing and we are looking forward to delivering customer success through innovative data modernization solutions on Microsoft Fabric.