我试图循环大量的试验,并计算一些子集的加权平均值。目前,数据格式为长格式,包含试验列、区域分数。
trial area score
0 T106 0 0.0035435
1 T106 1 0.0015967
2 T106 4 0.0003191
3 T106 4 0.1272919
4 T288 0 0.1272883
我有大约120000个试验,有4个区域,每个试验可能有10到100个分数,总共约700万行。我的第一个想法是在4个区域的循环中循环所有试验,构建一个临时数据框来计算分数,并将分数添加到外部数据框中:
for area in range(4):
for trial in trial_names.iloc[:,0]:
Tscore = 0
temp_trial = pd.DataFrame(trials_long.loc[(trials_long['tname'] == trial) & (trials_long['area'] == int(area))])
#match score in tria
temp_trial = temp_trial.merge(scores_df, how='left')
#sum score for all matching 'trial' +'area' #this will be weigted avrg, with >0.5 *2 and >0.9* 3
temp_trial.loc[temp_trial['score'] > 0.9, ['score']] *= 3 #weight 3x for >0.9
temp_trial.loc[temp_trial['score'] > 0.5, ['score']] *= 2 #weight 2x for >0.5
Tscore = temp_trial['score'].sum() / int(len(temp_trial.index))
trial_names.loc[trial,area] = Tscore #store Tscore somewhere
Tscore = 0
print('done')
