Skip to main content

PowerBI - Split a column in multiple rows when newline is the value delimiter

By

I started to use PowerBI for reporting for more than 1 year. Because of my lack of deep knowledge of PowerBI and time limitations I preferred to have the following approach for data consolidation.


What is important to mention is:

  • Azure SQL DB was not running all the time, the DB was started only when I wanted to import a new version of data
  • The stored procedures were doing a lot of reformating, cleanup, transformation and data rotation
The real blocker, that forced me to use Azure SQL DB was of the columns where the data were a list of items that were separated by the new-line. For each item on the list, I had to expend the table with an additional row.  

A few months ago I had some time and I decided to look at how:
  • Split a column by a specific delimiter
  • Do a column rotation (unpivotal).
The solution that I found is based on three specific actions:
  • Table.SplitColumn that enables us to split a column based on a delimiter
Table.SplitColumn(#"Filtered Rows", "Column name", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"})
In the above example, I add 50 new columns with the value that I get from split action. The output of this action is 50 new columns added with name D1, D2, ... D3 and the content represents the values that were delimited by the new-line. Inside PowerBI the new-line delimiter is "#(lf)".
  • Table.UnpivotOtherColumns that it is used to do a rotation of column to rows.
Table.UnpivotOtherColumns(#"Changed Type D", {"Column that I want to Keep 1","Column that I want to Keep 2" }, "Attribute", "Cloud Service", "Name of the new column") 
Important here to remember that you add an extra column that you will not need "Attribute"
  • Table.RemoveColumns removes the newly added column that it is not required - "Attribute"
Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})

If you have multiple columns where you need to apply this, take into consideration that you might end up with a high number of tables (one for each case). 
You can find below the full query:
let
    Source = Excel.Workbook(File.Contents("C:\Users\blabla\file.xlsx"), null, true),
    Projects_Sheet = ...,
    #"Renamed Columns" = ...,
    #"Replaced Value" = ...,
    #"Replaced Value1" = ...,
    #"Replaced Value2" = ...,
    #"Filtered Rows" = ...,
    #"Split Column by Delimiter" = Table.SplitColumn(#"Filtered Rows", "BigColumnWithNewLineAsDelimiterName", Splitter.SplitTextByDelimiter("#(lf)",QuoteStyle.Csv), {"D.1", "D.2", "D.3", "D.4", "D.5", "D.6", "D.7", "D.8", "D.9", "D.10","D.11", "D.12", "D.13", "D.14", "D.15", "D.16", "D.17"`, "D.18", "D.19", "D.20","D.21", "D.22", "D.23", "D.24", "D.25", "D.26", "D.27", "D.28", "D.29", "D.30","D.31", "D.32", "D.33", "D.34", "D.35", "D.36", "D.37", "D.38", "D.39","D.40","D.41", "D.42", "D.43", "D.44", "D.45", "D.46", "D.47", "D.48", "D.49","D.50"}),
    #"Changed Type D" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"D.1", type text}, {"D.2", type text}, {"D.3", type text}, {"D.4", type text}, {"D.5", type text}, {"D.6", type text}, {"D.7", type text}, {"D.8", type text}, {"D.9", type text}, {"D.10", type text} , {"D.11", type text}, {"D.12", type text}, {"D.13", type text}, {"D.14", type text}, {"D.15", type text}, {"D.16", type text}, {"D.17", type text}, {"D.18", type text}, {"D.19", type text}, {"D.20", type text}, {"D.21", type text}, {"D.22", type text}, {"D.23", type text}, {"D.24", type text}, {"D.25", type text}, {"D.26", type text}, {"D.27", type text}, {"D.28", type text}, {"D.29", type text}, {"D.30", type text}, {"D.31", type text}, {"D.32", type text}, {"D.33", type text}, {"D.34", type text}, {"D.35", type text}, {"D.36", type text}, {"D.37", type text}, {"D.38", type text}, {"D.39", type text}, {"D.40", type text}, {"D.41", type text}, {"D.42", type text}, {"D.43", type text}, {"D.44", type text}, {"D.45", type text}, {"D.46", type text}, {"D.47", type text}, {"D.48", type text}, {"D.49", type text}, {"D.50", type text}}),
    #"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Changed Type D", {"C","P" }, "Attribute", "SplitValuesOutput"),
    #"Removed Columns" = Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"})
in
    #"Removed Columns"

In this way, I was able to remove the Azure SQL DB and have a more simple data update flow.







Comments

Post a Comment