Automatic Threshold Tuning optimizes your binary classification models for reliable "real world" performance.
What is Automatic Threshold Tuning?
It’s an effective way to ensure that Kraken selects the correct threshold value to optimize the F1 scores of binary classification models. That optimization enables you to get a more realistic sense of how your binary models may perform in “the real world”.
How does it work?
Kraken computes the precision and recall for hundreds of different possible thresholds from 0 to 1, then the threshold that achieves the maximum F1 score is chosen. By selecting the threshold rather than relying on the default 0.5 threshold, Kraken produces predictions that are more robust for datasets that have an imbalance in the number of positive and negative cases.
Do I need to do anything to use Automatic Threshold Tuning?
Nope! It’s enabled for all new binary models that you build in Kraken and everything happens, well, automatically.
What about models I built before Automatic Threshold Tuning was available in Kraken?
Prior to Automatic Threshold Tuning, Kraken used a default threshold value of .5 for all binary classification models. Unless you retrain your “legacy” binary model in Kraken, it will continue to use the .5 threshold value. However, if you retrain a model, Kraken will apply Automatic Threshold Tuning. In virtually all cases, you will notice an improvement in your model’s F1 score, since Kraken is now more effectively optimizing model performance. Because Kraken automatically selects the algorithm with the highest F1 score, you may notice after retraining a binary model that Kraken has chosen a different algorithm than it did when you originally trained the model. As always, you have the option to override Kraken’s algorithm selection with the one of your choice.
How do I retrain an existing model to use Automatic Threshold Tuning?
| Step 1: From the analysis overview screen, click the "REFINE" option to the right of your analysis. |  |
| Step 2: Click the pink "ANALYZE" button in the upper right. (You do not need to add or modify any Data Pipeline steps before you analyze to retrain, but you're free to do so.) |  |
| Step 3: (Optional, but recommended) Verify that your model is now using Automatic Threshold Tuning by clicking into Model Metrics for your analysis, then clicking on an algorithm/model name to see the details. |  |
How do I get Kraken to apply predictions using my retrained model with Automatic Threshold Tuning enabled?
| Step 1: From the analysis overview screen, click the "DEPLOY" option to the right of your analysis. |  |
| Step 2: Select the "Publish" action for the new version of your analysis. |  |
Does Automatic Threshold Tuning work with Kraken’s automatic Hyperparameter Optimization (HPO) option?
Absolutely! If your company has purchased Kraken’s automatic Hyperparameter Optimization (HPO) option and you create a model with HPO enabled, Kraken will use both the optimization of hyperparameter values and Automatic Threshold Tuning for the most effective, accurate model results available from Kraken. Note that, like all models built prior to Automatic Threshold Tuning, your “legacy” HPO-enabled models were trained with a default .5 threshold value. If you retrain your HPO-enabled models, they will now use Automatic Threshold Tuning.
It's worth noting that the Automatic Threshold value will only be displayed for HPO-enabled models if you click on one of the underlined top-performing model/algorithm names to access Model Metrics. If you expand the list of models for a given algorithm and then drill into the Model Metrics, the Automatic Threshold value will not be displayed.
How do I know what threshold value Kraken has selected for the algorithms in my model?
Kraken displays the threshold value on the left-hand side of the algorithm screen within Model Metrics. After your model is trained, you can click the “Model Metrics” button (the yellow submarine) to access Model Metrics. From there, click on any of the algorithms to access the threshold value, along with the model metrics and confusion matrix you’re used to seeing. It’s worth noting that the threshold value is determined on a per-algorithm basis, so you will likely see a different threshold value for each algorithm in your model.
