Comparing cost of Dataflows (Gen1 vs. Gen2) in Power BI and Fabric

I have been using a Dataflow for years now which transforms a web log (.txt file) with many applied steps in Power Query and finally producing a nice table format to be used in my Power BI report. My raw data is about 15 GB or 100M rows.

This Dataflow Gen1 has been consuming lot of resources from our small F2 capacity. I have heard a lot about Dataflows Gen2 being expensive as well so I wanted to see what happens when I put exactly the same query in a new Dataflow Gen2.

Here is how you can migrate a Dataflow Gen1 -> Gen2

Dataflows export function (Export template)

This will download a .json file -> when creating a new dataflow you can import this file

Importing Dataflows defintion .json

I had to fix the permissions after that but once done my flow was refreshing just fine.

Now let us look at the costs of running both Dataflows (in terms of CUs) and also the average refresh times.

Refresh time comparison (Dataflow Gen1 vs Gen2)

	Gen1	Gen2
Low	26 min	16 min
High	32 min	19 min
Average	28 min	17 min

As you can see above, my Dataflow Gen2 refreshes faster, about 65% actually which was quite surprising.

Next let us look at the costs, we measure them in Compute Unit seconds (CUs) which is what you get charged by in any Fabric capacity. You can find some basics here.

Compute cost comparison in CUs (Dataflow Gen1 vs Gen2)

	Gen1	Gen2
Low	14,623	25,812
High	21,286	32,293
Average	17,988	27,832

The cost however, is much higher. On average it consumed 55% more than the Dataflow Gen1.

Since I was already worried about the cost of refreshing the Dataflow every day before and now it went up by 55% I really have to figure something out. For our F2 capacity one refresh already takes about 15,6% of our daily allowance. And usually we do not have it running through the night so this can go up to 25-30% of what we are paying for (to not pay more when pausing the capacity).

Now this example with 15GB of text files does not have to mean that all Dataflows Gen2 are faster and more expensive than Dataflows Gen1. But the results, were interesting in any case.

I would recommend you test the refresh time and CU usage before migrating your Dataflows to test if it will meet your expectations.