目录

1. 简介
2. 框架特点
3. XGB演示
4. pytorch演示

1简介

optuna是一个为机器学习，深度学习特别设计的自动超参数优化框架，具有脚本语言特性的用户API。因此，optuna的代码具有高度的模块特性，并且用户可以根据自己的希望动态构造超参数的搜索空间。

官方地址

2.框架特点

• 急切搜索空间

  • 使用 Python 条件、循环和语法自动搜索最佳超参数

• 最先进的算法

  • 有效搜索大型空间，并剪除不具潜力的试验，以更快地获得结果

• 轻松并行化

  • 在多个线程或进程上并行搜索超参数，无需修改代码

3 XGB演示

1. 使用目标函数对模型训练进行打包，并返回准确率
2. 使用试验对象建议超参数 trial object
3. 创建研究对象并执行优化 study object

• 1 创造目标函数 run(trial),然后在开头写好需要调的参数

def run(trial):
    learning_rate = trial.suggest_float("learning_rate", 1e-2, 0.25, log=True)
    reg_lambda = trial.suggest_loguniform("reg_lambda", 1e-8, 100.0)
    reg_alpha = trial.suggest_loguniform("reg_alpha", 1e-8, 100.0)
    subsample = trial.suggest_float("subsample", 0.1, 1.0)
    colsample_bytree = trial.suggest_float("colsample_bytree", 0.1, 1.0)
    max_depth = trial.suggest_int("max_depth", 1, 7)

    model = XGBRegressor(
        random_state=42,
        tree_method="gpu_hist",
        gpu_id=1,
        predictor="gpu_predictor",
        n_estimators=7000,
        learning_rate=learning_rate,
        reg_lambda=reg_lambda,
        reg_alpha=reg_alpha,
        subsample=subsample,
        colsample_bytree=colsample_bytree,
        max_depth=max_depth,
    )
    model.fit(xtrain, ytrain, early_stopping_rounds=300, eval_set=[(xvalid, yvalid)], verbose=1000)
    preds_valid = model.predict(xvalid)
    rmse = mean_squared_error(yvalid, preds_valid, squared=False)
    return rmse

• 2创建study

#因为rmse越小越好，使用minimize策略。具体的可以去官方文档查看 
study = optuna.create_study(direction="minimize")
study.optimize(run,n_trials= 5)#进行五次测试

运行过程：

Trial 1 finished with value: 0.7205222356228401 and parameters: {'learning_rate': 0.012638734258461837, 'reg_lambda': 0.1654682444195007, 'reg_alpha': 8.464204988794859e-07, 'subsample': 0.2814943186672778, 'colsample_bytree': 0.7410401444936071, 'max_depth': 6}. Best is trial 1 with value: 0.7205222356228401.
Trial 2 finished with value: 0.7216804907513846 and parameters: {'learning_rate': 0.12133025733597931, 'reg_lambda': 0.014541905001821521, 'reg_alpha': 3.158537981108844e-05, 'subsample': 0.5599409012134213, 'colsample_bytree': 0.40823839586330135, 'max_depth': 5}. Best is trial 1 with value: 0.7205222356228401.
Trial 4 finished with value: 0.7235507300754 and parameters: {'learning_rate': 0.05472940162092398, 'reg_lambda': 3.979142205866442e-06, 'reg_alpha': 0.0004941142482206257, 'subsample': 0.5413367489580537, 'colsample_bytree': 0.8220457595772027, 'max_depth': 1}. Best is trial 3 with value: 0.7187393044726648.

• 3 查看最佳参数

study.best_params

运行结果

{'learning_rate': 0.040075760420698166,
 'reg_lambda': 0.0004699745046141965,
 'reg_alpha': 1.3655450307414496,
 'subsample': 0.7222794496202382,
 'colsample_bytree': 0.5074652573342793,
 'max_depth': 4}

4 pytorch 演示

1. 使用目标函数对模型训练进行打包，并返回准确率
2. 使用试验对象建议超参数 trial object
3. 创建研究对象并执行优化 study object

import torch
import optuna

# 1. Define an objective function to be maximized.
def objective(trial):

    # 2. Suggest values of the hyperparameters using a trial object.
    n_layers = trial.suggest_int('n_layers', 1, 3)
    layers = []

    in_features = 28 * 28
    for i in range(n_layers):
        out_features = trial.suggest_int(f'n_units_l{i}', 4, 128)
        layers.append(torch.nn.Linear(in_features, out_features))
        layers.append(torch.nn.ReLU())
        in_features = out_features
    layers.append(torch.nn.Linear(in_features, 10))
    layers.append(torch.nn.LogSoftmax(dim=1))
    model = torch.nn.Sequential(*layers).to(torch.device('cpu'))
    ...
    return accuracy

# 3. Create a study object and optimize the objective function.
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)