数据块中pandas.dataframe.tail的等效值是多少[关闭]

数据块显然正在使用 pyspark.sql 数据帧,而不是 pandas .

# Index the df if you haven't already
# Note that monotonically increasing id has size limits
from pyspark.sql.functions import monotonically_increasing_id
df = df.withColumn("index", monotonically_increasing_id())

# Query with the index
tail = sqlContext.sql("""SELECT * FROM df ORDER BY index DESC limit 5""")
tail.show()

请注意,这是昂贵的,不发挥的优势 Spark .

参见:

https://medium.com/@chris_bour/6-differences-between-pandas-and-spark-dataframes-1380cec394d2

pyspark,spark: how to select last row and also how to access pyspark dataframe by index