PowerBI - Split a column in multiple rows when newline is the value delimiter

I started to use PowerBI for reporting for more than 1 year. Because of my lack of deep knowledge of PowerBI and time limitations I preferred to have the following approach for data consolidation.

What is important to mention is:

Azure SQL DB was not running all the time, the DB was started only when I wanted to import a new version of data
The stored procedures were doing a lot of reformating, cleanup, transformation and data rotation

The real blocker, that forced me to use Azure SQL DB was of the columns where the data were a list of items that were separated by the new-line. For each item on the list, I had to expend the table with an additional row.

A few months ago I had some time and I decided to look at how:

Split a column by a specific delimiter
Do a column rotation (unpivotal).

The solution that I found is based on three specific actions:

Table.SplitColumn that enables us to split a column based on a delimiter

Table.SplitColumn(#"Filtered Rows", "Column name", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"})

In the above example, I add 50 new columns with the value that I get from split action. The output of this action is 50 new columns added with name D1, D2, ... D3 and the content represents the values that were delimited by the new-line. Inside PowerBI the new-line delimiter is "#(lf)".

Table.UnpivotOtherColumns that it is used to do a rotation of column to rows.

Table.UnpivotOtherColumns(#"Changed Type D", {"Column that I want to Keep 1","Column that I want to Keep 2" }, "Attribute", "Cloud Service", "Name of the new column")

Important here to remember that you add an extra column that you will not need "Attribute"

Table.RemoveColumns removes the newly added column that it is not required - "Attribute"

Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})

If you have multiple columns where you need to apply this, take into consideration that you might end up with a high number of tables (one for each case).

You can find below the full query:

let

Source = Excel.Workbook(File.Contents("C:\Users\blabla\file.xlsx"), null, true),

Projects_Sheet = ...,

#"Renamed Columns" = ...,

#"Replaced Value" = ...,

#"Replaced Value1" = ...,

#"Replaced Value2" = ...,

#"Filtered Rows" = ...,

#"Split Column by Delimiter" = Table.SplitColumn(#"Filtered Rows", "BigColumnWithNewLineAsDelimiterName", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"}),

#"Changed Type D" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"D.1", type text}, {"D.2", type text}, {"D.3", type text}, {"D.4", type text}, {"D.5", type text}, {"D.6", type text}, {"D.7", type text}, {"D.8", type text}, {"D.9", type text}, {"D.10", type text} , {"D.11", type text}, {"D.12", type text}, {"D.13", type text}, {"D.14", type text}, {"D.15", type text}, {"D.16", type text}, {"D.17", type text}, {"D.18", type text}, {"D.19", type text}, {"D.20", type text}, {"D.21", type text}, {"D.22", type text}, {"D.23", type text}, {"D.24", type text}, {"D.25", type text}, {"D.26", type text}, {"D.27", type text}, {"D.28", type text}, {"D.29", type text}, {"D.30", type text}, {"D.31", type text}, {"D.32", type text}, {"D.33", type text}, {"D.34", type text}, {"D.35", type text}, {"D.36", type text}, {"D.37", type text}, {"D.38", type text}, {"D.39", type text}, {"D.40", type text}, {"D.41", type text}, {"D.42", type text}, {"D.43", type text}, {"D.44", type text}, {"D.45", type text}, {"D.46", type text}, {"D.47", type text}, {"D.48", type text}, {"D.49", type text}, {"D.50", type text}}),

#"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Changed Type D", {"C","P" }, "Attribute", "SplitValuesOutput"),

#"Removed Columns" = Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})

#"Removed Columns"

In this way, I was able to remove the Azure SQL DB and have a more simple data update flow.

How to audit an Azure Cosmos DB

In this post, we will talk about how we can audit an Azure Cosmos DB database. Before jumping into the problem let us define the business requirement: As an Administrator I want to be able to audit all changes that were done to specific collection inside my Azure Cosmos DB. The requirement is simple, but can be a little tricky to implement fully. First of all when you are using Azure Cosmos DB or any other storage solution there are 99% odds that you’ll have more than one system that writes data to it. This means that you have or not have control on the systems that are doing any create/update/delete operations. Solution 1: Diagnostic Logs Cosmos DB allows us activate diagnostics logs and stream the output a storage account for achieving to other systems like Event Hub or Log Analytics. This would allow us to have information related to who, when, what, response code and how the access operation to our Cosmos DB was done. Beside this there is a field that specifies what was th...

Can User Technical Profile Influence AI Architecture Decisions?

In the last few days, I ran a small experiment on Devin/Windsurf/Cascade, BMAD, and LLM-based architecture decisions. The initial concern was simple: if an AI coding assistant knows something about the user, for example, that the user is a Java developer or a .NET developer, can this information influence the architecture and technology stack that the AI will propose? This question is important because many companies are starting to use AI tools not only for coding but also for product discovery, architecture, documentation, and technical decision-making. If the AI is influenced by the developer's personal profile, the result may not be fully neutral. It may look like a business or architecture decision, but in reality, it may be partially shaped by the context of the person using the tool. The test was done using BMAD with Devin/Windsurf/Cascade. The scenario was a unified field service operations platform. The task was to let BMAD refine the product, define the architecture and s...

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

Today blog post will be started with the following error when running DB tests on the CI machine: threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration: TeamCity .NET 4.51 EF 6.0.2 VS2013 It see...

Cloud as a Story - Vunvulea Radu

Search This Blog