基于Scala的Spark数据帧多重滤波

你可以用多种方法来做。。下面是一个例子。

import org.apache.spark.sql.SparkSession

object DataFrameFromCSVFile2 {

  def main(args:Array[String]):Unit= {

    val spark: SparkSession = SparkSession.builder()
      .master("local[1]")
      .appName("SparkByExample")
      .getOrCreate()

    val filePath="src/main/resources/demodata.tx"

    val df = spark.read.options(Map("inferSchema"->"true","delimiter"->"|","header"->"true")).csv(filePath)

    val df2 = df.select("Gender", "BirthDate", "TotalCost", "TotalChildren", "ProductCategoryName")
      .filter("Gender is not null")
      .filter("BirthDate is not null")
      .filter("TotalChildren is not null")
      .filter("ProductCategoryName is not null")
    df2.show()

  }
}

输出:

+------+-------------------+---------+-------------+-------------------+
|Gender|          BirthDate|TotalCost|TotalChildren|ProductCategoryName|
+------+-------------------+---------+-------------+-------------------+
|Female|1957-03-06 00:00:00|     5000|            3|             Beauty|
|  Male|1959-03-06 00:00:00|     6000|            4|                Car|
+------+-------------------+---------+-------------+-------------------+

谢谢 纳文

您可以从数据帧中筛选它,如下所示, df.where(!$“性别”。isNull和!$“TotalChildren”。isNull)。显示