基于列减去2个pyspark数据帧

我有2个pyspark数据帧,

i
+---+-----+
| ID|COL_A|
+---+-----+
|  1|  123|
|  2|  456|
|  3|  111|
|  4|  678|
+---+-----+
j
+----+-----+
|ID_B|COL_B|
+----+-----+
|   2|  456|
|   3|  111|
|   4|  876|
+----+-----+

我想减去 i 从…起 j 基于特定列的值,即 COL_A 我 不应该出现在 COL_B 属于 J .

diff
+---+-----+
| ID|COL_A|
+---+-----+
|  1|  123|
|  4|  678|
+---+-----+

这是我的密码,

common = i.join(j.withColumnRenamed('COL_B', 'COL_A'), ['COL_A'], 'leftsemi')
diff = i.subtract(common)
diff.show()

但产出出现了问题,

diff
+---+-----+
| ID|COL_A|
+---+-----+
|  2|  456|
|  1|  123|
|  4|  678|
|  3|  111|
+---+-----+