Skip to main content

PowerBI - Split a column in multiple rows when newline is the value delimiter

By

I started to use PowerBI for reporting for more than 1 year. Because of my lack of deep knowledge of PowerBI and time limitations I preferred to have the following approach for data consolidation.


What is important to mention is:

  • Azure SQL DB was not running all the time, the DB was started only when I wanted to import a new version of data
  • The stored procedures were doing a lot of reformating, cleanup, transformation and data rotation
The real blocker, that forced me to use Azure SQL DB was of the columns where the data were a list of items that were separated by the new-line. For each item on the list, I had to expend the table with an additional row.  

A few months ago I had some time and I decided to look at how:
  • Split a column by a specific delimiter
  • Do a column rotation (unpivotal).
The solution that I found is based on three specific actions:
  • Table.SplitColumn that enables us to split a column based on a delimiter
Table.SplitColumn(#"Filtered Rows", "Column name", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"})
In the above example, I add 50 new columns with the value that I get from split action. The output of this action is 50 new columns added with name D1, D2, ... D3 and the content represents the values that were delimited by the new-line. Inside PowerBI the new-line delimiter is "#(lf)".
  • Table.UnpivotOtherColumns that it is used to do a rotation of column to rows.
Table.UnpivotOtherColumns(#"Changed Type D", {"Column that I want to Keep 1","Column that I want to Keep 2" }, "Attribute", "Cloud Service", "Name of the new column") 
Important here to remember that you add an extra column that you will not need "Attribute"
  • Table.RemoveColumns removes the newly added column that it is not required - "Attribute"
Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})

If you have multiple columns where you need to apply this, take into consideration that you might end up with a high number of tables (one for each case). 
You can find below the full query:
let
    Source = Excel.Workbook(File.Contents("C:\Users\blabla\file.xlsx"), null, true),
    Projects_Sheet = ...,
    #"Renamed Columns" = ...,
    #"Replaced Value" = ...,
    #"Replaced Value1" = ...,
    #"Replaced Value2" = ...,
    #"Filtered Rows" = ...,
    #"Split Column by Delimiter" = Table.SplitColumn(#"Filtered Rows", "BigColumnWithNewLineAsDelimiterName", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"}),
    #"Changed Type D" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"D.1", type text}, {"D.2", type text}, {"D.3", type text}, {"D.4", type text}, {"D.5", type text}, {"D.6", type text}, {"D.7", type text}, {"D.8", type text}, {"D.9", type text}, {"D.10", type text} , {"D.11", type text}, {"D.12", type text}, {"D.13", type text}, {"D.14", type text}, {"D.15", type text}, {"D.16", type text}, {"D.17", type text}, {"D.18", type text}, {"D.19", type text}, {"D.20", type text}, {"D.21", type text}, {"D.22", type text}, {"D.23", type text}, {"D.24", type text}, {"D.25", type text}, {"D.26", type text}, {"D.27", type text}, {"D.28", type text}, {"D.29", type text}, {"D.30", type text}, {"D.31", type text}, {"D.32", type text}, {"D.33", type text}, {"D.34", type text}, {"D.35", type text}, {"D.36", type text}, {"D.37", type text}, {"D.38", type text}, {"D.39", type text}, {"D.40", type text}, {"D.41", type text}, {"D.42", type text}, {"D.43", type text}, {"D.44", type text}, {"D.45", type text}, {"D.46", type text}, {"D.47", type text}, {"D.48", type text}, {"D.49", type text}, {"D.50", type text}}),
    #"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Changed Type D", {"C","P" }, "Attribute", "SplitValuesOutput"),
    #"Removed Columns" = Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})
in
    #"Removed Columns"

In this way, I was able to remove the Azure SQL DB and have a more simple data update flow.







Comments

Post a Comment

Popular posts from this blog

Windows Docker Containers can make WIN32 API calls, use COM and ASP.NET WebForms

By
After the last post , I received two interesting questions related to Docker and Windows. People were interested if we do Win32 API calls from a Docker container and if there is support for COM. WIN32 Support To test calls to WIN32 API, let’s try to populate SYSTEM_INFO class. [StructLayout(LayoutKind.Sequential)] public struct SYSTEM_INFO { public uint dwOemId; public uint dwPageSize; public uint lpMinimumApplicationAddress; public uint lpMaximumApplicationAddress; public uint dwActiveProcessorMask; public uint dwNumberOfProcessors; public uint dwProcessorType; public uint dwAllocationGranularity; public uint dwProcessorLevel; public uint dwProcessorRevision; } ... [DllImport("kernel32")] static extern void GetSystemInfo(ref SYSTEM_INFO pSI); ... SYSTEM_INFO pSI = new SYSTEM_INFO(...
Post a Comment
Read more

How to audit an Azure Cosmos DB

By
In this post, we will talk about how we can audit an Azure Cosmos DB database. Before jumping into the problem let us define the business requirement: As an Administrator I want to be able to audit all changes that were done to specific collection inside my Azure Cosmos DB. The requirement is simple, but can be a little tricky to implement fully. First of all when you are using Azure Cosmos DB or any other storage solution there are 99% odds that you’ll have more than one system that writes data to it. This means that you have or not have control on the systems that are doing any create/update/delete operations. Solution 1: Diagnostic Logs Cosmos DB allows us activate diagnostics logs and stream the output a storage account for achieving to other systems like Event Hub or Log Analytics. This would allow us to have information related to who, when, what, response code and how the access operation to our Cosmos DB was done. Beside this there is a field that specifies what was th...
Post a Comment
Read more

Cloud Myths: Cloud is Cheaper (Pill 1 of 5 / Cloud Pills)

By
Cloud Myths: Cloud is Cheaper (Pill 1 of 5 / Cloud Pills) The idea that moving to the cloud reduces the costs is a common misconception. The cloud infrastructure provides flexibility, scalability, and better CAPEX, but it does not guarantee lower costs without proper optimisation and management of the cloud services and infrastructure. Idle and unused resources, overprovisioning, oversize databases, and unnecessary data transfer can increase running costs. The regional pricing mode, multi-cloud complexity, and cost variety add extra complexity to the cost function. Cloud adoption without a cost governance strategy can result in unexpected expenses. Improper usage, combined with a pay-as-you-go model, can result in a nightmare for business stakeholders who cannot track and manage the monthly costs. Cloud-native services such as AI services, managed databases, and analytics platforms are powerful, provide out-of-the-shelve capabilities, and increase business agility and innovation. H...
Post a Comment
Read more