We will cover the following tasks in 58 minutes:
Introduction and Project Overview
In this task, we will create a new experiment from the Azure ML Studio dashboard. Next, we will import and take a look at the Adult Data Set before we move on to pre-processing.
Now that we have some idea about the properties of the data, we can start to get it ready for our model. First, we will clean the missing data. We are going to substitute all missing values by 0 using the Clean Missing Data module.
Our next step is to use the Select Columns in Dataset module to exclude irrelevant and redundant columns from the data. We do this to reduce the clutter doing analysis.
Once we have our final features, we use the Edit Metatdata module to convert the following columns from String types to Categorical Feature types:
Accounting for Class Imbalance
Before we create our training and test sets, we have one last pre-processing step. And that’s dealing with class imbalance in our dataset. The number of people earning less than $50K/yr is more than twice of the people earning greater than $50K/yr. What we want to do is upsample the minority class. At this point, it’s very easy to fall into the trap of applying upsampling to your entire dataset. I strongly caution against doing that. The timing of upsampling can affect the generalization ability of a model. Since one of the primary goals of model validation is to estimate how it will perform on unseen data, upsampling correctly is critical. The right way is to only do it on the training data.
By upsampling only on the training data, none of the information in the validation data is being used to create synthetic observations. So these results should be generalizable. Let’s see if that’s true.
What we’re going to do now is to train two models. One model will be trained on the upsampled data, and the other with just the original pre-processed data. We can compare how both models perform and make our conclusions about the efficacy of creating synthetic observations by upsampling the minority class.
Training a Two-Class Boosted Decision Tree Model and Hyperparameter Tuning
For this experiment we will train a two-class boosted decision tree model to predict the
Income class label. Hyperparameter tuning is done using the Tune Model Hyperparameters module. The parameter sweeps and training will take only 5 minutes to complete.
Scoring and Evaluating the Models
AUC stands for “area under curve”, and as it’s name implies, it refers to the amount of area under the ROC curve, which theoretically is a value between 0 and 1. As I explained, the worst possible curve in practice is a diagonal line, hence the AUC should never be lower than 0.5 (for large data sets) . Using the AUC metric you can quickly compare multiple learning models. Remember that the ROC curves of two models usually don’t cross each other, hence when comparing two models, the one with a higher AUC will be the better one regardless of the threshold setting. Compared to the statistical measures of accuracy, precision, recall and F1 score, AUC’s independence of threshold makes it uniquely qualified for model selection. On the other hand, unlike accuracy, precision, recall and F1 score, AUC does not tell us what performance to expect from the model for a given threshold setting, nor can it be used to determine the optimal value for threshold. In that regard it doesn’t take away the need for the other statistical measures.
So in short The ROC plot and the AUC are very useful for comparing and selecting the best machine learning model for a given data set. A model with an AUC score near 1, and where the ROC curve comes close to the upper left corner, has a very good performance. A model with a score near 0.5 will have a curve near the diagonal and its performance is hardly better than a random predictor.
Publishing the Trained Model as a Web Service for Inference
We are now ready to create a web service from an Azure ML prediction model. When the experiment run completes successfully, you will be guided to create a Scoring or Prediction Experiment.
Preparation for deployment is a three-step process:
- Remove one of the models
- Convert the training experiment you’ve created into a predictive experiment
- Deploy the predictive experiment as a web service
The prediction experiment will automatically be created for you with a click. In the prediction experiment, the learner will be replaced with a trained model that has been automatically saved for you from your training experiment.
Once your scoring experiment runs successfully, you will be guided to publish your trained model as a web service.
About the Host (Snehan Kekre)
Snehan hosts Machine Learning and Data Sciences projects at Rhyme. He is in his senior year of university at the Minerva Schools at KGI, studying Computer Science and Artificial Intelligence. When not applying computational and quantitative methods to identify the structures shaping the world around him, he can sometimes be seen trekking in the mountains of Nepal.