Skip to main content

PowerBI - Split a column in multiple rows when newline is the value delimiter

By

I started to use PowerBI for reporting for more than 1 year. Because of my lack of deep knowledge of PowerBI and time limitations I preferred to have the following approach for data consolidation.


What is important to mention is:

  • Azure SQL DB was not running all the time, the DB was started only when I wanted to import a new version of data
  • The stored procedures were doing a lot of reformating, cleanup, transformation and data rotation
The real blocker, that forced me to use Azure SQL DB was of the columns where the data were a list of items that were separated by the new-line. For each item on the list, I had to expend the table with an additional row.  

A few months ago I had some time and I decided to look at how:
  • Split a column by a specific delimiter
  • Do a column rotation (unpivotal).
The solution that I found is based on three specific actions:
  • Table.SplitColumn that enables us to split a column based on a delimiter
Table.SplitColumn(#"Filtered Rows", "Column name", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"})
In the above example, I add 50 new columns with the value that I get from split action. The output of this action is 50 new columns added with name D1, D2, ... D3 and the content represents the values that were delimited by the new-line. Inside PowerBI the new-line delimiter is "#(lf)".
  • Table.UnpivotOtherColumns that it is used to do a rotation of column to rows.
Table.UnpivotOtherColumns(#"Changed Type D", {"Column that I want to Keep 1","Column that I want to Keep 2" }, "Attribute", "Cloud Service", "Name of the new column") 
Important here to remember that you add an extra column that you will not need "Attribute"
  • Table.RemoveColumns removes the newly added column that it is not required - "Attribute"
Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})

If you have multiple columns where you need to apply this, take into consideration that you might end up with a high number of tables (one for each case). 
You can find below the full query:
let
    Source = Excel.Workbook(File.Contents("C:\Users\blabla\file.xlsx"), null, true),
    Projects_Sheet = ...,
    #"Renamed Columns" = ...,
    #"Replaced Value" = ...,
    #"Replaced Value1" = ...,
    #"Replaced Value2" = ...,
    #"Filtered Rows" = ...,
    #"Split Column by Delimiter" = Table.SplitColumn(#"Filtered Rows", "BigColumnWithNewLineAsDelimiterName", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"}),
    #"Changed Type D" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"D.1", type text}, {"D.2", type text}, {"D.3", type text}, {"D.4", type text}, {"D.5", type text}, {"D.6", type text}, {"D.7", type text}, {"D.8", type text}, {"D.9", type text}, {"D.10", type text} , {"D.11", type text}, {"D.12", type text}, {"D.13", type text}, {"D.14", type text}, {"D.15", type text}, {"D.16", type text}, {"D.17", type text}, {"D.18", type text}, {"D.19", type text}, {"D.20", type text}, {"D.21", type text}, {"D.22", type text}, {"D.23", type text}, {"D.24", type text}, {"D.25", type text}, {"D.26", type text}, {"D.27", type text}, {"D.28", type text}, {"D.29", type text}, {"D.30", type text}, {"D.31", type text}, {"D.32", type text}, {"D.33", type text}, {"D.34", type text}, {"D.35", type text}, {"D.36", type text}, {"D.37", type text}, {"D.38", type text}, {"D.39", type text}, {"D.40", type text}, {"D.41", type text}, {"D.42", type text}, {"D.43", type text}, {"D.44", type text}, {"D.45", type text}, {"D.46", type text}, {"D.47", type text}, {"D.48", type text}, {"D.49", type text}, {"D.50", type text}}),
    #"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Changed Type D", {"C","P" }, "Attribute", "SplitValuesOutput"),
    #"Removed Columns" = Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})
in
    #"Removed Columns"

In this way, I was able to remove the Azure SQL DB and have a more simple data update flow.







Comments

Post a Comment

Popular posts from this blog

Windows Docker Containers can make WIN32 API calls, use COM and ASP.NET WebForms

By
After the last post , I received two interesting questions related to Docker and Windows. People were interested if we do Win32 API calls from a Docker container and if there is support for COM. WIN32 Support To test calls to WIN32 API, let’s try to populate SYSTEM_INFO class. [StructLayout(LayoutKind.Sequential)] public struct SYSTEM_INFO { public uint dwOemId; public uint dwPageSize; public uint lpMinimumApplicationAddress; public uint lpMaximumApplicationAddress; public uint dwActiveProcessorMask; public uint dwNumberOfProcessors; public uint dwProcessorType; public uint dwAllocationGranularity; public uint dwProcessorLevel; public uint dwProcessorRevision; } ... [DllImport("kernel32")] static extern void GetSystemInfo(ref SYSTEM_INFO pSI); ... SYSTEM_INFO pSI = new SYSTEM_INFO(
Post a Comment
Read more

ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded

By
Today blog post will be started with the following error when running DB tests on the CI machine: threw exception: System.InvalidOperationException: The Entity Framework provider type 'System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer' registered in the application config file for the ADO.NET provider with invariant name 'System.Data.SqlClient' could not be loaded. Make sure that the assembly-qualified name is used and that the assembly is available to the running application. See http://go.microsoft.com/fwlink/?LinkId=260882 for more information. at System.Data.Entity.Infrastructure.DependencyResolution.ProviderServicesFactory.GetInstance(String providerTypeName, String providerInvariantName) This error happened only on the Continuous Integration machine. On the devs machines, everything has fine. The classic problem – on my machine it’s working. The CI has the following configuration: TeamCity .NET 4.51 EF 6.0.2 VS2013 It see
8 comments
Read more

Azure AD and AWS Cognito side-by-side

By
In the last few weeks, I was involved in multiple opportunities on Microsoft Azure and Amazon, where we had to analyse AWS Cognito, Azure AD and other solutions that are available on the market. I decided to consolidate in one post all features and differences that I identified for both of them that we should need to take into account. Take into account that Azure AD is an identity and access management services well integrated with Microsoft stack. In comparison, AWS Cognito is just a user sign-up, sign-in and access control and nothing more. The focus is not on the main features, is more on small things that can make a difference when you want to decide where we want to store and manage our users.  This information might be useful in the future when we need to decide where we want to keep and manage our users.  Feature Azure AD (B2C, B2C) AWS Cognito Access token lifetime Default 1h – the value is configurable 1h – cannot be modified
Post a Comment
Read more