贝尔斯调参及代码实现

一、贝尔斯（Bayesian）优化调参

​ 贝尔斯优化用于机器学习调参，由J.Snoek(2012)提出，其主要思想及技术过程是，给定优化的特定目标函数空间（广义的函数），通过假定一个先验函数（Prior Function, PF）形式（如高斯过程回归形式），每一轮迭代不断添加样本点实现样本点上的先验函数阶段回归，然后再此阶段先验函数回归的基础上使用采集函数（Acquisition Function , AC）计算出一个最优值的新样本点，如果满足度量准则则作为候选输出，否则反馈给先验函数阶段回归，重新计算高斯过程回归，如此不断迭代，直到达到迭代次数或者目标函数空间（就是由产生的不同样本点所代表的目标函数值所组成的空间）的后验分布不断收敛至真实分布（在计算过程中表现为目标函数值收敛至特定误差限）。

二、实现代码

​ 贝尔斯优化可直接借用Python库的BayesianOptmization来实现。

实现代码：

import lightgbm as lgb
import numpy as np
import pandas as pd
import time
from sklearn.model_selection import train_test_split

data = pd.read_csv("flights.csv")
data = data.sample(frac=0.01, random_state=200)  #这里采样可能不均衡 , y_true 有时只有一个标签；需要检查

data = data[["MONTH","DAY","DAY_OF_WEEK","AIRLINE","FLIGHT_NUMBER","DESTINATION_AIRPORT",
                 "ORIGIN_AIRPORT","AIR_TIME", "DEPARTURE_TIME","DISTANCE","ARRIVAL_DELAY"]]
data.shape

data = data.reset_index(drop=True)
data.head()

data["ARRIVAL_DELAY"] = (data["ARRIVAL_DELAY"]>10)*1

cols = ["AIRLINE","FLIGHT_NUMBER","DESTINATION_AIRPORT","ORIGIN_AIRPORT"]
for item in cols:
    data[item] = data[item].astype("category").cat.codes + 1
    
X_train, X_test, y_train, y_test = train_test_split(data.drop(["ARRIVAL_DELAY"], axis=1), data["ARRIVAL_DELAY"],
                                                random_state=10, test_size=0.3)

print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)

y_train.value_counts()

y_test.value_counts()

from bayes_opt import BayesianOptimization
from tqdm import tqdm

def xgb_evaluate(min_child_weight, colsample_bytree,max_depth,subsample, gamma,alpha):
    params['min_child_weight'] = int(min_child_weight)
    params['cosample_bytree'] = max(min(colsample_bytree, 1), 0)
    params['max_depth'] = int(max_depth)
    params['subsample'] = max(min(subsample, 1), 0)
    params['gamma'] = max(gamma, 0)
    params['alpha'] = max(alpha, 0)
    cv_result = xgb.cv(params, dtrain, num_boost_round=num_rounds, nfold=5,
             seed=2021,
             callbacks=[xgb.callback.EarlyStopping(50)]) # 最新用法  EarlyStopping ;
    return cv_result['test-auc-mean'].values[-1]

num_rounds = 3000
random_state = 2021
num_iter = 25
init_points = 5
params = {
    'eta': 0.1,
    'silent': 1,
    'eval_metric': 'auc',
    'verbose_eval': True,
    'seed': random_state
}

xgbBO = BayesianOptimization(xgb_evaluate, {'min_child_weight': (1, 20),
                                            'colsample_bytree': (0.1, 1),
                                            'max_depth': (5, 15),
                                            'subsample': (0.5, 1),
                                            'gamma': (0, 10),
                                            'alpha': (0, 10),
                                            })

xgbBO.maximize(init_points=init_points, n_iter=num_iter,acq="ucb", kappa=1)

结果：

​ 可见，贝尔斯优化在第28次迭代中达到最优，此时alpha参数取3.647，列抽样比例0.919，gamma参数为0.1535，树最大平均深度为5.127，最小子树权重取11.24以及子抽样比例为0.8673时，测试集ROC-AUC达到最优的0.7277。