Automatic Threshold Tuning optimizes your binary classification models for reliable "real world" performance.
What is Automatic Threshold Tuning?
It’s an effective way to ensure that AutoML selects the correct threshold value to optimize the F1 scores of binary classification models. That optimization enables you to get a more realistic sense of how your binary models may perform in “the real world”.
How does it work?
AutoML computes the precision and recall for hundreds of different possible thresholds from 0 to 1, then the threshold that achieves the maximum F1 score is chosen. By selecting the threshold rather than relying on the default 0.5 threshold, AutoML produces predictions that are more robust for datasets that have an imbalance in the number of positive and negative cases.
Do I need to do anything to use Automatic Threshold Tuning?
Nope! It’s enabled for all new binary models that you build in AutoML and everything happens, well, automatically.
What about models I built before Automatic Threshold Tuning was available in AutoML?
Prior to Automatic Threshold Tuning, AutoML used a default threshold value of .5 for all binary classification models. Unless you retrain your “legacy” binary model in AutoML, it will continue to use the .5 threshold value. However, if you retrain a model, AutoML will apply Automatic Threshold Tuning. In virtually all cases, you will notice an improvement in your model’s F1 score, since AutoML is now more effectively optimizing model performance. Because AutoML automatically selects the algorithm with the highest F1 score, you may notice after retraining a binary model that AutoML has chosen a different algorithm than it did when you originally trained the model. As always, you have the option to override the automatic algorithm selection with the one of your choice.
How do I retrain an existing model to use Automatic Threshold Tuning?
| Step 1: From the analysis overview screen, click the "Retrain" option on the right.. |  |
| Step 2: (Optional, but recommended) Verify that your model is now using Automatic Threshold Tuning by clicking into Model Metrics for your analysis, then clicking on an algorithm/model name to see the details. |  |
How do I get AutoML to apply predictions using my retrained model with Automatic Threshold Tuning enabled?
| Step 1: From the analysis overview screen, click the "Deploy" option on the right. |  |
| Step 2: Select the "Publish" action for the new version of your analysis. |  |
Does Automatic Threshold Tuning work with automatic Hyperparameter Optimization (HPO) option?
Absolutely! If your company has purchased automatic Hyperparameter Optimization (HPO) and you create a model with HPO enabled, AutoML will use both the optimization of hyperparameter values and Automatic Threshold Tuning for the most effective, accurate model results available from AutoML. Note that, like all models built prior to Automatic Threshold Tuning, your “legacy” HPO-enabled models were trained with a default .5 threshold value. If you retrain your HPO-enabled models, they will now use Automatic Threshold Tuning.
It's worth noting that the Automatic Threshold value will only be displayed for HPO-enabled models if you click on one of the underlined top-performing model/algorithm names to access Model Metrics. If you expand the list of models for a given algorithm and then drill into the Model Metrics, the Automatic Threshold value will not be displayed.
How do I know what threshold value AutoML has selected for the algorithms in my model?
AutoML displays the threshold value on the left-hand side of the algorithm screen within Model Metrics. After your model is trained, you can click the “Model Metrics” button (the yellow submarine) to access Model Metrics. From there, click on any of the algorithms to access the threshold value, along with the model metrics and confusion matrix you’re used to seeing. It’s worth noting that the threshold value is determined on a per-algorithm basis, so you will likely see a different threshold value for each algorithm in your model.
