当我尝试将tf-idf应用于测试集时，维度不匹配
我正在尝试将一种新的预处理算法应用于我的数据集，答案如下：Encoding text in ML classifier
我现在尝试的内容如下：
def test_tfidf(data, ngrams = 1):
df_temp = data.copy(deep = True)
df_temp = basic_preprocessing(df_temp)
tfidf_vectorizer = TfidfVectorizer(ngram_range=(1, ngrams))
tfidf_vectorizer.fit(df_temp['Text'])
list_corpus = df_temp["Text"].tolist()
list_labels = df_temp["Label"].tolist()
X = tfidf_vectorizer.transform(list_corpus)
return X, list_labels(我建议参考上面提到的所有代码的链接)。当我尝试将后两个函数应用于我的数据集时：
train_x, train_y, count_vectorizer = tfidf(undersample_train, ngrams = 1)
testing_set = pd.concat([X_test, y_test], axis=1)
test_x, test_y = test_tfidf(testing_set, ngrams = 1)
full_result = full_result.append(training_naive(train_x, test_x, train_y, test_y), ignore_index = True)我得到了这个错误：
---> 12 full_result = full_result.append(training_naive(train_x, test_x, train_y, test_y, ), ignore_index = True)
---> 14 y_pred = clf.predict(X_test_naive)
ValueError: dimension mismatch错误中提到的函数是：
def training_naive(X_train_naive, X_test_naive, y_train_naive, y_test_naive, preproc):
clf = MultinomialNB()
clf.fit(X_train_naive, y_train_naive)
y_pred = clf.predict(X_test_naive)
return 如果对理解我的新定义和/或将tf-idf应用于我的数据集有任何帮助(请参阅此处的相关部分：Encoding text in ML classifier)，将不胜感激。
更新:我认为这个问题/答案也可以帮助我解决这个问题：scikit-learn ValueError: dimension mismatch
如果我用test_x, test_y = test_tfidf(undersample_train, ngrams = 1)替换test_x, test_y = test_tfidf(testing_set, ngrams = 1)，它不会返回任何错误。然而，我不认为这是正确的，因为我得到了非常非常高的值(所有统计数据中99%)
回答 1
Stack Overflow用户
发布于 2020-12-13 20:25:12
当使用转换(在本例中为TfidfVectorizer)时，您必须使用相同的对象来转换训练和测试数据。转换器通常仅使用训练数据进行拟合，然后重新用于转换测试数据。
在您的情况下，正确的方法是：
def tfidf(data, ngrams = 1):
df_temp = data.copy(deep = True)
df_temp = basic_preprocessing(df_temp)
tfidf_vectorizer = TfidfVectorizer(ngram_range=(1, ngrams))
tfidf_vectorizer.fit(df_temp['Text'])
list_corpus = df_temp["Text"].tolist()
list_labels = df_temp["Label"].tolist()
X = tfidf_vectorizer.transform(list_corpus)
return X, list_labels, tfidf_vectorizer
def test_tfidf(data, vectorizer, ngrams = 1):
df_temp = data.copy(deep = True)
df_temp = basic_preprocessing(df_temp)
# No need to create a new TfidfVectorizer here!
list_corpus = df_temp["Text"].tolist()
list_labels = df_temp["Label"].tolist()
X = vectorizer.transform(list_corpus)
return X, list_labels
# this method is copied from the other SO question
def training_naive(X_train_naive, X_test_naive, y_train_naive, y_test_naive, preproc):
clf = MultinomialNB() # Gaussian Naive Bayes
clf.fit(X_train_naive, y_train_naive)
res = pd.DataFrame(columns = ['Preprocessing', 'Model', 'Precision', 'Recall', 'F1-score', 'Accuracy'])
y_pred = clf.predict(X_test_naive)
f1 = f1_score(y_pred, y_test_naive, average = 'weighted')
pres = precision_score(y_pred, y_test_naive, average = 'weighted')
rec = recall_score(y_pred, y_test_naive, average = 'weighted')
acc = accuracy_score(y_pred, y_test_naive)
res = res.append({'Preprocessing': preproc, 'Model': 'Naive Bayes', 'Precision': pres,
'Recall': rec, 'F1-score': f1, 'Accuracy': acc}, ignore_index = True)
return res
train_x, train_y, count_vectorizer = tfidf(undersample_train, ngrams = 1)
testing_set = pd.concat([X_test, y_test], axis=1)
test_x, test_y = test_tfidf(testing_set, count_vectorizer, ngrams = 1)
full_result = full_result.append(training_naive(train_x, test_x, train_y, test_y, count_vectorizer), ignore_index = True)https://stackoverflow.com/questions/65270921
复制相似问题
腾讯云开发者
Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有
深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 
粤公网安备44030502008569号
腾讯云计算（北京）有限责任公司 京ICP证150476号 | 京ICP备11018762号