如何在Scala/Spark中忽略Groupby中的第一个元素?

删除数据帧中的第一行并非易事(另请参见 Drop first row of Spark DataFrame ). 但您可以使用窗口函数来实现:

val df = Seq(
  "cat", "cat", "cat", "hat", "hat", "bat"
).toDF("value")


val dfGrouped = df
  .groupBy($"value").count()
  .sort($"count".desc)

dfGrouped.show()

+-----+-----+
|value|count|
+-----+-----+
|  cat|    3|
|  hat|    2|
|  bat|    1|
+-----+-----+

val dfWithoutFirstRow = dfGrouped
  .withColumn("rank", dense_rank().over(Window.partitionBy().orderBy($"count".desc)))
  .where($"rank" =!= 1).drop($"rank") // this filters "cat"
  .sort($"count".desc)


dfWithoutFirstRow
  .show()

+-----+-----+
|value|count|
+-----+-----+
|  hat|    2|
|  bat|    1|
+-----+-----+

第一行可以这样删除:

val filteredValue = dfGrouped.first.get(0)
val result = dfGrouped.filter(s"value!='$filteredValue'")