Skip to main content

PowerBI - Split a column in multiple rows when newline is the value delimiter

By

I started to use PowerBI for reporting for more than 1 year. Because of my lack of deep knowledge of PowerBI and time limitations I preferred to have the following approach for data consolidation.


What is important to mention is:

  • Azure SQL DB was not running all the time, the DB was started only when I wanted to import a new version of data
  • The stored procedures were doing a lot of reformating, cleanup, transformation and data rotation
The real blocker, that forced me to use Azure SQL DB was of the columns where the data were a list of items that were separated by the new-line. For each item on the list, I had to expend the table with an additional row.  

A few months ago I had some time and I decided to look at how:
  • Split a column by a specific delimiter
  • Do a column rotation (unpivotal).
The solution that I found is based on three specific actions:
  • Table.SplitColumn that enables us to split a column based on a delimiter
Table.SplitColumn(#"Filtered Rows", "Column name", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"})
In the above example, I add 50 new columns with the value that I get from split action. The output of this action is 50 new columns added with name D1, D2, ... D3 and the content represents the values that were delimited by the new-line. Inside PowerBI the new-line delimiter is "#(lf)".
  • Table.UnpivotOtherColumns that it is used to do a rotation of column to rows.
Table.UnpivotOtherColumns(#"Changed Type D", {"Column that I want to Keep 1","Column that I want to Keep 2" }, "Attribute", "Cloud Service", "Name of the new column") 
Important here to remember that you add an extra column that you will not need "Attribute"
  • Table.RemoveColumns removes the newly added column that it is not required - "Attribute"
Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})

If you have multiple columns where you need to apply this, take into consideration that you might end up with a high number of tables (one for each case). 
You can find below the full query:
let
    Source = Excel.Workbook(File.Contents("C:\Users\blabla\file.xlsx"), null, true),
    Projects_Sheet = ...,
    #"Renamed Columns" = ...,
    #"Replaced Value" = ...,
    #"Replaced Value1" = ...,
    #"Replaced Value2" = ...,
    #"Filtered Rows" = ...,
    #"Split Column by Delimiter" = Table.SplitColumn(#"Filtered Rows", "BigColumnWithNewLineAsDelimiterName", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"}),
    #"Changed Type D" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"D.1", type text}, {"D.2", type text}, {"D.3", type text}, {"D.4", type text}, {"D.5", type text}, {"D.6", type text}, {"D.7", type text}, {"D.8", type text}, {"D.9", type text}, {"D.10", type text} , {"D.11", type text}, {"D.12", type text}, {"D.13", type text}, {"D.14", type text}, {"D.15", type text}, {"D.16", type text}, {"D.17", type text}, {"D.18", type text}, {"D.19", type text}, {"D.20", type text}, {"D.21", type text}, {"D.22", type text}, {"D.23", type text}, {"D.24", type text}, {"D.25", type text}, {"D.26", type text}, {"D.27", type text}, {"D.28", type text}, {"D.29", type text}, {"D.30", type text}, {"D.31", type text}, {"D.32", type text}, {"D.33", type text}, {"D.34", type text}, {"D.35", type text}, {"D.36", type text}, {"D.37", type text}, {"D.38", type text}, {"D.39", type text}, {"D.40", type text}, {"D.41", type text}, {"D.42", type text}, {"D.43", type text}, {"D.44", type text}, {"D.45", type text}, {"D.46", type text}, {"D.47", type text}, {"D.48", type text}, {"D.49", type text}, {"D.50", type text}}),
    #"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Changed Type D", {"C","P" }, "Attribute", "SplitValuesOutput"),
    #"Removed Columns" = Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})
in
    #"Removed Columns"

In this way, I was able to remove the Azure SQL DB and have a more simple data update flow.







Comments

Post a Comment

Popular posts from this blog

Windows Docker Containers can make WIN32 API calls, use COM and ASP.NET WebForms

By
After the last post , I received two interesting questions related to Docker and Windows. People were interested if we do Win32 API calls from a Docker container and if there is support for COM. WIN32 Support To test calls to WIN32 API, let’s try to populate SYSTEM_INFO class. [StructLayout(LayoutKind.Sequential)] public struct SYSTEM_INFO { public uint dwOemId; public uint dwPageSize; public uint lpMinimumApplicationAddress; public uint lpMaximumApplicationAddress; public uint dwActiveProcessorMask; public uint dwNumberOfProcessors; public uint dwProcessorType; public uint dwAllocationGranularity; public uint dwProcessorLevel; public uint dwProcessorRevision; } ... [DllImport("kernel32")] static extern void GetSystemInfo(ref SYSTEM_INFO pSI); ... SYSTEM_INFO pSI = new SYSTEM_INFO(...
Post a Comment
Read more

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

By
Today blog post will be started with the following error when running DB tests on the CI machine: threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration: TeamCity .NET 4.51 EF 6.0.2 VS2013 It see...
8 comments
Read more

Navigating Cloud Strategy after Azure Central US Region Outage

By
 Looking back, July 19, 2024, was challenging for customers using Microsoft Azure or Windows machines. Two major outages affected customers using CrowdStrike Falcon or Microsoft Azure computation resources in the Central US. These two outages affected many people and put many businesses on pause for a few hours or even days. The overlap of these two issues was a nightmare for travellers. In addition to blue screens in the airport terminals, they could not get additional information from the airport website, airline personnel, or the support line because they were affected by the outage in the Central US region or the CrowdStrike outage.   But what happened in reality? A faulty CrowdStrike update affected Windows computers globally, from airports and healthcare to small businesses, affecting over 8.5m computers. Even if the Falson Sensor software defect was identified and a fix deployed shortly after, the recovery took longer. In parallel with CrowdStrike, Microsoft provi...
Post a Comment
Read more