我正在做一个简单的二进制分类,使用来自pyspark.ml的RandomForestClassifier。在将数据输入训练之前,我设法使用VectorIndexer通过提供参数maxCategories来决定特征是数值的还是分类的。
问题:
即使我使用了maxCategories设置为30的VectorIndexer,在训练管道期间仍然会出现错误:
An error occurred while calling o15371.fit.
: java.lang.IllegalArgumentException: requirement failed: DecisionTree requires maxBins (= 32) to be at least as large as the number of values in each categorical feature, but categorical feature 0 has 10765 values. Considering remove this and other categorical features with a large number of values, or add more training examples.
我的代码很简单,col_idx是我生成的列字符串列表,它将传递给stringindexer,col_all是列字符串列表,它将传递给stringindexer和onehotecoder,col_num是数值列名。
from pyspark.ml.feature import OneHotEncoderEstimator, StringIndexer, VectorAssembler, IndexToString, VectorIndexer
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
my_data.cache()
# stringindexers and encoders
stIndexers = [StringIndexer(inputCol = Col, outputCol = Col + 'Index').setHandleInvalid('keep') for Col in col_idx]
encoder = OneHotEncoderEstimator(inputCols = [Col + 'Index' for Col in col_all], outputCols = [Col + 'ClassVec' for Col in col_all]).setHandleInvalid('keep')
# vector assemblor
col_into_assembler = [cols + 'Index' for cols in col_idx] + [cols + 'ClassVec' for cols in col_all] + col_num
assembler = VectorAssembler(inputCols = col_into_assembler, outputCol = "features")
# featureIndexer, labelIndexer, rf classifier and labelConverter
featureIndexer = VectorIndexer(inputCol = "features", outputCol = "indexedFeatures", maxCategories = 30)
# columns smaller than maxCategories => categorical features, columns larger than maxCategories => numerical / continuous features, smaller value => less categorical features, larger value => more categorical features.
labelIndexer = StringIndexer(inputCol = "label", outputCol = "indexedLabel").fit(my_data)
rf = RandomForestClassifier(featuresCol = "indexedFeatures", labelCol = "indexedLabel")
labelConverter = IndexToString(inputCol = "prediction", outputCol = "predictedLabel", labels=labelIndexer.labels)
# chain all the estimators and transformers stages into a Pipeline estimator
rfPipeline = Pipeline(stages = stIndexers + [encoder, assembler, featureIndexer, labelIndexer, rf, labelConverter])
# split data, cache them
training, test = my_data.randomSplit([0.7, 0.3], seed = 100)
training.cache()
test.cache()
# fit the estimator with training dataset to get a compiled pipeline with transformers and fitted models.
ModelRF = rfPipeline.fit(training)
# make predictions
predictions = ModelRF.transform(test)
predictions.printSchema()
predictions.show(5)
