Predicting bank loan defaults using the ID3 (Iterative Dichotomiser 3) algorithm
1.0 Dataset Understanding
First, we create a hypothetical dataset of ten records with the following features that might influence loan defaulting: Age (Young, Middle-aged, Old), Income (Low, Medium, High), Credit Score (Low, High), and the target variable Loan Default (Yes, No).
| ID | AGE | INCOME | CREDIT SCORE | LOAN DEFAULT |
| 1 | Young | Low | Low | Yes |
| 2 | Young | Low | High | No |
| 3 | Middle-aged | Medium | Low | No |
| 4 | Old | Medium | Low | No |
| 5 | Old | High | High | No |
| 6 | Old | High | High | No |
| 7 | Middle-aged | High | High | No |
| 8 | Young | Medium | Low | Yes |
| 9 | Young | High | High | Yes |
| 10 | Old | Medium | High | No |
2.0 Computation of Entropy for the Whole Dataset
Entropy measures the impurity or uncertainty in the dataset. The formula for entropy, given a dataset D, is:
where pi is the proportion of the class i in the dataset, and m is the number of classes. In our case, we have two classes for the target variable (Loan Default: Yes, No).
3.0 Computation of Entropy for Each Feature
For each feature, we calculate the entropy to understand how well the feature separates the data. The lower the entropy, the better the feature is at splitting the data into homogeneous groups (in terms of the target variable).
4.0 Computation of Information Gain for Each Feature
Information Gain is calculated as the difference between the dataset’s original entropy and the entropy after splitting the dataset based on a feature. It measures how much information a feature gives us about the class. The formula is:
Where Dv is the subset of D for which feature A has value v, and Values(A) are all possible values of feature A.
5.0 Building the Decision Tree
Start with the entire dataset as the root.
Select the feature with the highest information gain as the root node. Divide the dataset based on the values of this feature.
Repeat for each branch, using only the data that reaches the branch. If a branch has pure data (all Yes or No), stop dividing.
Repeat the process for each feature until all data is classified or no further information gain is possible.
let’s illustrate computing the overall entropy and the information gain for one feature, “Credit Score,” to determine if it should be the root node.
Computing Overall Entropy:
Given our dataset has 6 No’s and 4 Yes’s for Loan Default:
Computing Entropy for “Credit Score”:
Credit Score = High: 5 No, 2 Yes
Credit Score = Low: 1 No, 2 Yes
Computing Information Gain for “Credit Score”:
Let’s perform the calculations for the overall entropy and the information gain for the “Credit Score” feature to start building our decision tree.
The overall entropy of the dataset, representing the impurity before any splits, is approximately 0.971. This value indicates a mix of positive and negative outcomes in the dataset, leading to uncertainty.
After computing the information gain for the “Credit Score” feature, we find it to be approximately 0.091. This value indicates how much uncertainty in the dataset would be reduced after splitting on this feature. The higher the information gain, the more effective the feature is at reducing uncertainty about the target variable (Loan Default).
The next steps in building the decision tree using the ID3 algorithm would involve calculating the entropy and information gain for the remaining features (“Age” and “Income”) in a similar manner. Then, you’d compare the information gain values to choose the feature with the highest information gain as the root node or the next node in the tree for each split. We continue this process recursively, splitting the dataset based on the chosen features and their values, until we either achieve pure nodes (where all instances belong to a single class) or no further information gain is possible.
6.0 Making Prediction Based on the Model
This process involves significant computation, especially as the number of features and the size of the dataset increase. For the sake of this example, let’s assume the decision tree we ended up with has the following simple structure based on our initial dataset:
Root Node: Credit Score
If High, follow right branch.
If Low, follow left branch.
Right Branch (Credit Score = High): Majority of instances are “No” (loan will not default), so we’ll predict No.
Left Branch (Credit Score = Low): This branch is more mixed, but for simplicity, let’s say further splits led us to predict Yes for the majority (loan will default).
Now, let’s create a hypothetical test dataset with three new records to predict if these loans will default based on our simple decision tree.
| ID | AGE | INCOME | CREDIT SCORE |
| 1 | Middle-aged | High | High |
| 2 | Young | Low | Low |
| 3 | Old | Medium | High |
Making Predictions
Now, we’ll use the decision tree logic to make predictions on the test dataset:
Record 1: Credit Score = High. According to our tree, we predict No (the loan will not default).
Record 2: Credit Score = Low. Based on our tree, we predict Yes (the loan will default).
Record 3: Credit Score = High. Following the tree, we again predict No (the loan will not default).
These predictions are based on the simplification of our initial decision tree and demonstrate how a decision tree can be used to make predictions. In a real-world scenario, the decision tree have more levels and consider more features to make a prediction, thus providing a more detailed understanding of the factors leading to loan defaults.