Wsrtjtyk

Is there a way to optimize this query to not use withColumn multiple times. My biggest problem is that I hit this issue: https://issues.apache.org/jira/browse/SPARK-18532

The query is something like this. I have a dataframe with 10 boolean columns.
I have some modifiers like:

val smallIncrease = 5

val smallDecrease = -5

val bigIncrease = 10

val bigDecrease = -10

Based on each of the boolean column I would like to calculate a final score by adding small/big increase/decrease base on values in different columns.
So now my query looks something like this:

df.withColumn("result", when(col("col1"), col("result") + lit(smallIncrease)).otherwise(col("result") + lit(smallDecrease)))

.withColumn("result", when(col("col2"), col("result") + lit(bigIncrease)).otherwise(col("result") + lit(bigDecrease)))

.withColumn("result", when(col("col3"), col("result") + lit(smallIncrease)).otherwise(col("result") + lit(smallDecrease)))

.withColumn("result", when(col("col4"), col("result") + lit(smallIncrease)).otherwise(col("result") + lit(smallDecrease)))

.withColumn("result", when(col("col5"), col("result") + lit(smallIncrease)).otherwise(col("result") + lit(bigDecrease)))

.withColumn("result", when(col("col6"), col("result") + lit(bigIncrease)).otherwise(col("result") + lit(smallDecrease)))

.withColumn("result", when(col("col7"), col("result") + lit(smallIncrease)).otherwise(col("result") + lit(smallDecrease)))

Is there a way to compact this query and avoid multiple withColumns.
Unfortunaltey UDF is not choise as there are more than 10 boolean column to take into account and UDFs are limited to 10 column. Maybe I can split it into 2 UDFs but this looks very ugly to me...

How about something like this?

def myFun(b: Seq[Boolean], result: Int): Int = {



  val conversions: Seq[(Boolean, Int) => Int] = ??? // Functions to apply increase/decrease for each boolean value col1, col2 etc.



  b.zip(conversions).foldLeft(result){

    case (acc, (nextBool, nextFun)) => nextFun(nextBool, acc) 

  }

}



val myUdf = udf(myFun(_: Seq[Boolean], _: Int))



df.select(myUdf(array($"col1", $"col2", $"col3"...), $"result").as("result"))