What is Support Vector Machine?

 What is Support Vector Machine?

Support Vector Machine (SVM) is a supervised machine learning algorithm that is widely used in classification, regression, and outlier detection problems. SVM is based on the concept of finding the optimal hyperplane that separates different classes in the feature space.

In detail, the SVM algorithm works by mapping the input data into a high-dimensional feature space using a non-linear mapping function. It then finds the optimal hyperplane that maximizes the margin between the two classes. The margin is the distance between the hyperplane and the nearest data points from each class. The hyperplane that has the maximum margin is the one that is chosen as the optimal hyperplane. SVM is capable of handling both linear and non-linear classification problems by using different kernel functions.

How the algorithm works:

• First, the algorithm takes the input data and maps it into a higher-dimensional space. This mapping is done using a kernel function, which transforms the input data into a new space where it is easier to separate the classes using a hyperplane.
• Next, the algorithm finds the hyperplane that maximizes the margin between the two classes. The margin is the distance between the hyperplane and the closest data points from each class.
• The algorithm then predicts the class of new data points by determining which side of the hyperplane they fall on.

Advantages of SVM include:

• SVM can handle both linear and non-linear classification problems by using different kernel functions such as linear, polynomial, radial basis function (RBF), and sigmoid.
• SVM can handle high-dimensional data and can perform well even when the number of features is greater than the number of samples.
• SVM has a regularization parameter that helps to avoid overfitting and improve the generalization performance of the model.
• SVM can handle both binary and multi-class classification problems by using different strategies such as one-vs-one and one-vs-all.

Disadvantages of SVM include:

• SVM can be sensitive to the choice of kernel function and its parameters. Choosing the right kernel function and its parameters can be a challenging task.
• SVM can be computationally expensive, especially for large datasets with a large number of features.
• SVM can be sensitive to outliers in the data and may result in a suboptimal solution.

An example of building a simple SVM model using Python's scikit-learn library:

First, let's load the dataset and split it into training and testing sets:

from sklearn.datasets import load_breast_cancer

from sklearn.model_selection import train_test_split

# Load data

cancer = load_breast_cancer()

# Split data into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target, test_size=0.3, random_state=42)

Next, let's create an SVM model with a radial basis function (RBF) kernel and a regularization parameter of 1.0:

from sklearn.svm import SVC

# Create SVM model

svc = SVC(kernel='rbf', C=1.0)

We can train the model on the training data using the fit method:

# Train SVM model on training data

svc.fit(X_train, y_train)

We can then use the model to make predictions on the testing data using the predict method:

# Make predictions on testing data

y_pred = svc.predict(X_test)

Finally, we can evaluate the performance of the model using metrics such as accuracy, precision, recall, and F1-score:

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Calculate evaluation metrics

accuracy = accuracy_score(y_test, y_pred)

precision = precision_score(y_test, y_pred)

recall = recall_score(y_test, y_pred)

f1 = f1_score(y_test, y_pred)

# Print evaluation metrics

print('Accuracy: {:.2f}'.format(accuracy))

print('Precision: {:.2f}'.format(precision))

print('Recall: {:.2f}'.format(recall))

print('F1-score: {:.2f}'.format(f1))

This will output the evaluation metrics for the SVM model on the testing data. The exact values may vary each time the code is run due to the random splitting of the data into training and testing sets.

In this example, we first load the iris dataset from Scikit-learn's built-in datasets. We split the data into training and testing sets using the train_test_split function. We create an SVM model with a linear kernel and a regularization parameter of 1.0. We train the SVM model on the training data using the fit function. We then use the trained model to predict the classes of the testing data using the predict function. Finally, we calculate the accuracy score of the model on the testing data using the accuracy_score function and print the result.

What is Random Forests?

What is Random Forests? 

Random Forests is a popular machine learning algorithm used for both regression and classification tasks. It is an ensemble method that combines multiple decision trees to make more accurate predictions.

How the algorithm works:

1. Data Preparation: Random Forests can handle both categorical and continuous data. It requires a labeled dataset with both input features and output labels.
2. Feature Selection: Random Forests randomly select a subset of features from the dataset to build each decision tree. This helps to avoid overfitting and improves the performance of the algorithm.
3. Build Decision Trees: Random Forests builds multiple decision trees using the subset of features selected in step 2. Each decision tree is built by selecting a random sample of the data and a random subset of features.
4. Voting: When making a prediction, Random Forests takes the input features and runs them through each decision tree in the forest. Each tree returns a prediction, and the final prediction is made by taking a majority vote of all the individual tree predictions.
5. Evaluation: Random Forests performance is evaluated by using a metric that is appropriate for the problem at hand. For example, for a regression problem, one could use mean squared error (MSE), while for a classification problem, one could use accuracy or F1 score.

Advantages of Random Forests:

1. Random Forests can handle both categorical and continuous data.
2. It can handle missing data.
3. Random Forests are resistant to overfitting because of feature selection and bagging.
4. It can be used for both classification and regression tasks.
5. It can handle high dimensional data with a large number of features.
6. It provides an estimate of feature importance.

Disadvantages of Random Forests:

1. Random Forests can be slow to train on large datasets with a large number of trees.
2. The model can be difficult to interpret because of the large number of decision trees.
3. Random Forests can be biased towards features with many categories.

Random Forests is a powerful machine learning algorithm that is widely used for both classification and regression tasks. It combines multiple decision trees to make more accurate predictions and is resistant to overfitting. However, it can be slow to train on large datasets, and the model can be difficult to interpret.

An example of building a simple random forest model using Python's scikit-learn library:

1. First, let's import the necessary libraries:

from sklearn.ensemble import RandomForestClassifier

from sklearn.datasets import make_classification

from sklearn.model_selection import train_test_split

2. Next, let's generate a sample dataset using make_classification:

X, y = make_classification(n_samples=1000, n_features=4, n_informative=2, n_redundant=0, random_state=42)

3. Here, we generate a dataset with 1000 samples, 4 features, 2 informative features, and 0 redundant features. Now, let's split the dataset into training and testing sets:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

4. Here, we use 20% of the dataset for testing. Now, let's create a random forest classifier and fit it to the training data:

rf = RandomForestClassifier(n_estimators=100, random_state=42)

rf.fit(X_train, y_train)

5. Here, we create a random forest classifier with 100 trees and fit it to the training data. Finally, let's evaluate the performance of the model on the testing data:

print("Accuracy:", rf.score(X_test, y_test)) 

This will print the accuracy of the model on the testing data.

And that's it! You've built a simple random forest model using scikit-learn. Of course, you can modify the parameters of the random forest classifier to improve its performance or adapt it to your specific needs.

Confusion Matrix

What is Confusion Matrix and Advanced Classification Metrics?

 

After data preparation and model training, there is model evaluation phase which I mentioned in my earlier article Simple Picture of Machine Learning Modelling Process. 

Once model is developed, the next phase is to calculate the performance of the developed model using some evaluation metrics. In this article, you will just discover about confusion matrix though there are many classification metrics out there.

Mainly, it focuses on below points:

• What is confusion matrix? 
• Four outputs in confusion matrix
• Advanced classification metrics 
  
  

  

  
  

  Table 1. Confusion matrix with advanced classification metrics
  

Confusion Matrix is a tool to determine the performance of classifier. It contains information about actual and predicted classifications. The below table shows confusion matrix of two-class, spam and non-spam classifier.
 

 Table 2. Confusion matrix of email classification 

Let’s understand four outputs in confusion matrix.

1. True Positive (TP) is the number of correct predictions that an example is positive which means positive class correctly identified as positive.

Example: Given class is spam and the classifier has been correctly predicted it as spam.      

2. False Negative (FN) is the number of incorrect predictions that an example is negative which means positive class incorrectly identified as negative.
Example: Given class is spam however, the classifier has been incorrectly predicted it as non-spam.  

3. False positive (FP) is the number of incorrect predictions that an example is positive which means negative class incorrectly identified as positive.
Example: Given class is non-spam however, the classifier has been incorrectly predicted it as spam. 

4. True Negative (TN) is the number of correct predictions that an example is negative which means negative class correctly identified as negative.
Example: Given class is spam and the classifier has been correctly predicted it as negative. 

Now, let’s see some advanced classification metrics based on confusion matrix. These metrics are mathematically expressed in Table 1 with example of email classification, shown in Table 2. Classification problem has spam and non-spam classes and dataset contains 100 examples, 65 are Spams and 35 are non-spams.

Sensitivity is also referred as True Positive Rate or Recall. It is measure of positive examples labeled as positive by classifier. It should be higher. For instance, proportion of emails which are spam among all spam emails.  

Table 3. Sensitivity in confusion matrix

Sensitivity = 45/(45+20) = 69.23% .  

The 69.23% spam emails are correctly classified and excluded from all non-spam emails. 

Specificity is also know as True Negative Rate. It is measure of negative examples labeled as negative by classifier. There should be high specificity. For instance, proportion of emails which are non-spam among all non-spam emails. 

Table 4. Specificity in confusion matrix

specificity = 30/(30+5) = 85.71% .

The 85.71% non-spam emails are accurately classified and excluded from all spam emails.

Precision is ratio of total number of correctly classified positive examples and the total number of predicted positive examples. It shows correctness achieved in positive prediction. 

Table 5. Precision in confusion matrix

Precision = 45/(45+5)= 90% 

The 90% of examples are classified as spam are actually spam.

Accuracy is the proportion of the total number of predictions that are correct.

 Table 6. Accuracy in confusion matrix

Accuracy = (45+30)/(45+20+5+30) = 75%

The 75% of examples are correctly classified by the classifier. 

F1 score is a weighted average of the recall (sensitivity) and precision. F1 score might be good choice when you seek to balance between Precision and Recall. 

It helps to compute recall and precision in one equation so that the problem to distinguish the models with low recall and high precision or vice versa could be solved.  

Kindly follow my blog by email and stay tuned for more advanced post on regression measures. 

Thank you!

---