Wsrtjtyk

Here is the schema of the incomming data stream. Im using spark 2.3.2 streaming to process the data.

val schema = StructType(Seq(

            StructField("status", StringType),

            StructField("data", StructType(Seq(

                StructField("resultType", StringType),

                StructField("result", ArrayType(StructType(Array(

                    StructField("metric", StructType(Seq(StructField("application", StringType),

                                                         StructField("component", StringType),

                                                         StructField("instance", StringType)))), 

                    StructField("value", ArrayType(StringType))

                ))))

             )

         )))) 

Here is how i've applied the schema to the dstream's rdd.

  val df = rdd.toDS()                        

                    .selectExpr("cast (value as string) as myData") 

                    .select(from_json($"myData", schema).as("myData"))               

                    .select($"myData.data.*")

                    .select("result")

The above code yields the following output

{"result":[{"metric":{"application":"A","component":"S","instance":"tp01.net:9072"},"value":["1.542972576979E9","237006995456"]},

       {"metric":{"application":"A","component":"S","instance":"tp02.net:9072"},"value":["1.542972576979E9","237006995456"]},

       {"metric":{"application":"A","component":"S","instance":"tp03.net:9072"},"value":["1.542972576979E9","237006995456"]},

       {"metric":{"application":"B","component":"S","instance":"bp03.net:9072"},"value":["1.542972576979E9","270860144640"]},

       {"metric":{"application":"B","component":"S","instance":"bp04.net:9072"},"value":["1.542972576979E9","270860144640"]},

       {"metric":{"application":"B","component":"S","instance":"ps01.net:9072"},"value":["1.542972576979E9","135177400320"]},

 ]}

But in order to extract features, i need to convert the above to the following data frame

application     component       instance            value1              value2

A               S               tp01.net:9072       1.542972576979E9    237006995456

A               S               tp02.net:9072       1.542972576979E9    237006995456

A               S               tp03.net:9072       1.542972576979E9    237006995456

B               S               bp03.net:9072       1.542972576979E9    270860144640

B               S               bp04.net:9072       1.542972576979E9    270860144640

B               S               ps01.net:9072       1.542972576979E9    135177400320

As you see the each row is already an exploded row. Any ideas on how to select the array values and the struct into a single dataframe please?

Thanks