MS Fabric - In the Real World

In May 2023, Microsoft unveiled Fabric, a comprehensive platform that simplifies and unifies various Microsoft, Azure, and Power BI functionalities under one umbrella, providing a centralised hub for data and analytics.

Of course, as someone who has been using Power BI and Azure products primarily, I was super excited about the possibilities. Since then, my team and I have been experimenting with Fabric and delivered POC’s for our clients. We’ve learned quite a bit about Microsoft Fabric and the opportunities it represents.

One of the tests we ran for our POC work for a client, was: Can we replicate a classic Lakehouse Architecture for an on-premises set of data sources? The answer: a resounding yes!

What worked well?

• Our team’s existing skills in Azure smoothly translated to the equivalent content items.
• Power BI Developers familiar with Power Query, were able to contribute to the data engineering components by developing the Gen2 Dataflows required to move on-premises data coming from API sources.
• Existing load patterns from their existing Synapse environment were easily adapted into Fabric.
• Getting reference data into OneLake was a breeze.
• Orchestration of the data ingestion processes and building the presentation layer was straightforward using Pipelines.

What were the challenges?

• On-premises sources in Pipelines are not yet supported, meaning we needed to fall back to Gen2 Dataflows for these sources.
• A few elements around security needed some attention – we needed to unblock some outbound ports on the Gateway to write to a Lakehouse from a Gen2 Dataflow.
• Microsoft Fabric is still in public preview, so there were some bugs in the interface – usually resolved by refreshing the page. This led to a few times where we lost a bit of work!

Fabric was intuitive and straightforward to adopt, though there were some challenges for sure, I feel we’ve only scratched the surface so far of what it’s capable of as we continue to develop our Fabric framework and continue to deliver pilots and POCs for our clients.