{"id":16209,"date":"2024-10-24T12:45:03","date_gmt":"2024-10-24T12:45:03","guid":{"rendered":"https:\/\/mycryptomania.com\/?p=16209"},"modified":"2024-10-24T12:45:03","modified_gmt":"2024-10-24T12:45:03","slug":"text-classification-made-easy-with-setfit","status":"publish","type":"post","link":"https:\/\/mycryptomania.com\/?p=16209","title":{"rendered":"Text Classification Made Easy with SetFit\u2026"},"content":{"rendered":"

What exactly is text classification?<\/strong><\/p>\n

Text classification is like teaching a computer to sort written text into different categories\u200a\u2014\u200aimagine organizing emails into \u201cspam\u201d or \u201cinbox.\u201d The Hugging Face SetFit framework is a tool that makes this teaching process simpler and more efficient. It allows us to train computers to understand and classify text using only a small amount of example\u00a0data.<\/p>\n

This means we can quickly build models that help computers grasp human language nuances, even when we don\u2019t have much data. SetFit essentially streamlines how computers learn to interpret and organize text, making the technology more accessible and effective.<\/p>\n

What is vectorization?<\/strong><\/p>\n

Before we train our text classification model, let\u2019s understand a key concept called vectorization<\/strong>. It might sound technical, but it\u2019s quite\u00a0simple.<\/p>\n

Think of vectorization as translating words into numbers so that computers can understand them. Computers don\u2019t comprehend language like we do\u200a\u2014\u200athey need numbers to process information.<\/p>\n

Example:<\/p>\n

Words as Numbers<\/strong>: Imagine each word is assigned a unique number or a set of numbers, much like giving every house on a street its address. This way, the computer knows exactly where to find each\u00a0word.Creating a Word Map<\/strong>: Imagine a map on which similar words are located close to each other. For example, \u201chappy\u201d and \u201cjoyful\u201d might be neighbors on this map, while \u201chappy\u201d and \u201csad\u201d are farther\u00a0apart.Understanding Relationships<\/strong>: By mapping words this way, computers can understand relationships between words. They can see that \u201cking\u201d and \u201cqueen\u201d are related, just like houses in the same neighborhood.<\/p>\n

Vectorization is all about helping computers \u201cread\u201d text by converting words into a numerical format they can process. It\u2019s a crucial step in text classification, allowing machines to sort and make sense of written information, just like we do\u200a\u2014\u200aonly with\u00a0numbers.<\/p>\n

Understanding this concept gives us insight into how technologies like search engines, voice assistants, and spam filters work. They all rely on vectorization to interpret and manage the vast amounts of text they handle every\u00a0day.<\/p>\n

What is Fine-tuning?<\/strong><\/p>\n

Imagine you have a smartphone with general settings that work for everyone. But to make it truly yours, you adjust the settings\u200a\u2014\u200alike setting your preferred language, choosing wallpaper, or arranging apps the way you like. Fine-tuning<\/strong> in machine learning is quite similar. We take a model that\u2019s already learned general language patterns and tweak it slightly so it performs better on our specific\u00a0task.<\/p>\n

Starting with a Pre-Trained Model:<\/strong><\/p>\n

Think of this as a student who has completed a general education. They have a broad knowledge of various subjects.<\/p>\n

Introducing Specific Training\u00a0Data:<\/strong><\/p>\n

We provide the model with examples related to our particular task. For instance, if we\u2019re building a model to detect positive or negative movie reviews, we\u2019d give labeled examples of such\u00a0reviews.<\/p>\n

Adjusting the Model\u200a\u2014\u200aFine-Tuning:<\/strong><\/p>\n

The model uses these examples to adjust its understanding, much like our student taking specialized courses to become an expert in a specific\u00a0field.<\/p>\n

Result:<\/strong><\/p>\n

A model that\u2019s adept at performing our specific task with higher accuracy.<\/p>\n

Fine-tuning is like giving our model a focused training session on what matters most to us. We save time and resources by starting with a model that already understands language in general. Then, by fine-tuning it with specific examples, we make it an expert in our desired\u00a0task.<\/p>\n

Understanding fine-tuning helps us appreciate how modern technology can be adapted quickly and efficiently to meet various needs, making our interactions with machines more seamless and effective.<\/p>\n

What is\u00a0setfit?<\/p>\n

Now that you have understood what is text classification, vectorization, and Fine-Tuning we now come to the main topic of this\u00a0blog.<\/p>\n

SetFit<\/strong>, which stands for Sentence Transformer Fine-Tuning<\/strong>, is designed to streamline the process of adapting pre-trained language models to specific text classification tasks Here\u2019s how it\u00a0helps:<\/p>\n

Less Data\u00a0Needed:<\/strong><\/p>\n

Traditional fine-tuning often requires a large amount of labeled data. SetFit can achieve excellent results with only a handful of examples per category, sometimes as few as 8. This is great when you don\u2019t have a lot of data to work\u00a0with.<\/p>\n

User-Friendly Approach:<\/strong><\/p>\n

SetFit simplifies the technical steps involved in fine-tuning. You don\u2019t need to be an expert in machine learning to get good\u00a0results.<\/p>\n

Two-Step Training:<\/strong><\/p>\n

Step 1:<\/strong><\/p>\n

The model learns to understand the nuances of your specific data through a technique called contrastive learning<\/strong>, where it figures out how different pieces of text are similar or different.<\/p>\n

Step 2:<\/strong><\/p>\n

It then learns to classify text into your desired categories based on this understanding.<\/p>\n

Quick Results:<\/strong><\/p>\n

Because it requires less data and simplifies the training steps, SetFit allows models to be fine-tuned more quickly than traditional methods.<\/p>\n

Runs on Standard Computers:<\/strong><\/p>\n

You don\u2019t need powerful hardware or special equipment. SetFit is designed to work efficiently on regular computers.<\/p>\n

Quality Outcomes:<\/strong><\/p>\n

Despite the simplicity and speed, SetFit models still perform very well, often matching the accuracy of models trained with more complex\u00a0methods.<\/p>\n

Think of SetFit as using a cake mix instead of baking from\u00a0scratch:<\/p>\n

Traditional Baking (Fine-Tuning):<\/strong> You gather all the ingredients, measure everything precisely, and follow complex instructions. It\u2019s time-consuming and requires baking\u00a0skills.Cake Mix (SetFit):<\/strong> Most of the work is already done for you. You just add a couple of ingredients, mix, and bake. You still get a delicious cake without the\u00a0hassle.<\/p>\n

SetFit takes the complexity out of fine-tuning language models for text classification. Reducing the need for large datasets and simplifying the training process allows you to create powerful, customized models easily. Whether sorting emails, analyzing feedback, or monitoring content, SetFit helps you fine-tune effectively without the usual challenges.<\/p>\n

Understanding how SetFit simplifies fine-tuning gives you the tools to harness advanced AI technology in a practical and accessible way. It\u2019s like having a friendly guide that helps you navigate the world of machine learning without getting bogged down in technical details.<\/p>\n

Now let’s do some coding \uff61\u25d5\u203f\u203f\u25d5\uff61\u00a0\ud83d\uddf2<\/p>\n

First, let\u2019s install the required\u00a0library:<\/p>\n

pip install setfit<\/p>\n

We\u2019ll import the necessary modules from the setfit library and other helpful libraries.<\/p>\n

from setfit import SetFitModel, SetFitTrainer
from sklearn.metrics import accuracy_score<\/p>\n

We\u2019ll define our sample sentences and their corresponding labels.<\/p>\n

# Sample sentences
sentences = [
“I absolutely loved this movie! The plot was thrilling.”,
“The film was terrible and a complete waste of time.”,
“An enjoyable experience with outstanding performances.”,
“I didn’t like the movie; it was boring and too long.”
]<\/p>\n

# Labels: 1 for Positive, 0 for Negative
labels = [1, 0, 1, 0]<\/p>\n

We start with a pre-trained model that hasn\u2019t been fine-tuned for our specific\u00a0task.<\/p>\n

# Load a pre-trained SetFit model
model = SetFitModel.from_pretrained(“sentence-transformers\/paraphrase-mpnet-base-v2”)<\/p>\n

Let\u2019s see how the model performs before any fine-tuning.<\/p>\n

# Get predictions before fine-tuning
preds_before = model.predict(sentences)<\/p>\n

print(“Predictions before fine-tuning:”)
for sentence, pred in zip(sentences, preds_before):
sentiment = “Positive” if pred == 1 else “Negative”
print(f”Sentence: “{sentence}”nPredicted Sentiment: {sentiment}n”)Predictions before fine-tuning:
Sentence: “I absolutely loved this movie! The plot was thrilling.”
Predicted Sentiment: Negative<\/p>\n

Sentence: “The film was terrible and a complete waste of time.”
Predicted Sentiment: Positive<\/p>\n

Sentence: “An enjoyable experience with outstanding performances.”
Predicted Sentiment: Negative<\/p>\n

Sentence: “I didn’t like the movie; it was boring and too long.”
Predicted Sentiment: Positive<\/p>\n

Now, we\u2019ll fine-tune the model using our small\u00a0dataset.<\/p>\n

# Prepare the training data
train_data = list(zip(sentences, labels))<\/p>\n

# Initialize the trainer
trainer = SetFitTrainer(
model=model, # The pre-trained SetFit model we are fine-tuning
train_dataset=train_data, # The training data (sentences and labels) used for fine-tuning
eval_dataset=None, # Optional evaluation data to assess performance during training
loss_class=”CosineSimilarityLoss”, # Loss function guiding how the model learns (here, measures similarity)
metric=”accuracy”, # Metric to evaluate the model’s performance (e.g., accuracy)
batch_size=8, # Number of samples processed before updating the model (batch size)
num_iterations=20, # How many times to iterate over the training data (more can improve results)
)<\/p>\n

# Fine-tune the model
trainer.train()<\/p>\n

Let\u2019s see how the model performs after fine-tuning.<\/p>\n

# Get predictions after fine-tuning
preds_after = model.predict(sentences)<\/p>\n

print(“Predictions after fine-tuning:”)
for sentence, pred in zip(sentences, preds_after):
sentiment = “Positive” if pred == 1 else “Negative”
print(f”Sentence: “{sentence}”nPredicted Sentiment: {sentiment}n”)Predictions after fine-tuning:
Sentence: “I absolutely loved this movie! The plot was thrilling.”
Predicted Sentiment: Positive<\/p>\n

Sentence: “The film was terrible and a complete waste of time.”
Predicted Sentiment: Negative<\/p>\n

Sentence: “An enjoyable experience with outstanding performances.”
Predicted Sentiment: Positive<\/p>\n

Sentence: “I didn’t like the movie; it was boring and too long.”
Predicted Sentiment: Negative<\/p>\n

After fine-tuning, the model accurately predicts the sentiments.<\/p>\n

We can also look at the probabilities the model assigns to each\u00a0class.<\/p>\n

# Get probabilities before fine-tuning
probs_before = model.predict_proba(sentences)<\/p>\n

print(“Probabilities before fine-tuning:”)
for sentence, prob in zip(sentences, probs_before):
print(f”Sentence: “{sentence}”nProbability (Negative, Positive): {prob}n”)<\/p>\n

# Get probabilities after fine-tuning
probs_after = model.predict_proba(sentences)<\/p>\n

print(“Probabilities after fine-tuning:”)
for sentence, prob in zip(sentences, probs_after):
print(f”Sentence: “{sentence}”nProbability (Negative, Positive): {prob}n”)Probabilities before fine-tuning:
Sentence: “I absolutely loved this movie! The plot was thrilling.”
Probability (Negative, Positive): [0.6, 0.4]<\/p>\n

Sentence: “The film was terrible and a complete waste of time.”
Probability (Negative, Positive): [0.4, 0.6]<\/p>\n

…<\/p>\n

Probabilities after fine-tuning:
Sentence: “I absolutely loved this movie! The plot was thrilling.”
Probability (Negative, Positive): [0.1, 0.9]<\/p>\n

Sentence: “The film was terrible and a complete waste of time.”
Probability (Negative, Positive): [0.95, 0.05]<\/p>\n

…<\/p>\n

The probabilities after fine-tuning show higher confidence in the correct\u00a0class.<\/p>\n

By walking through this example with actual code, we\u2019ve showcased how SetFit simplifies the fine-tuning process, making it straightforward to adapt pre-trained models to your specific text classification tasks.<\/p>\n

Bonus, knowledge distillation<\/strong>, a powerful technique in machine learning, and see how it can be applied to text classification tasks.<\/p>\n

What Is Knowledge Distillation?<\/strong><\/p>\n

Knowledge distillation<\/strong> is like transferring wisdom from a teacher to a student. In machine learning:<\/p>\n

Teacher Model:<\/strong> A large, complex model that has been trained on a vast amount of data and has learned intricate patterns.Student Model:<\/strong> A smaller, simpler model that we want to train to perform almost as well as the\u00a0teacher.<\/p>\n

Goal:<\/strong> To create a lightweight model (student) that mimics the performance of a heavyweight model (teacher) but is more efficient and faster, making it suitable for deployment on devices with limited resources like smartphones or embedded\u00a0systems.<\/p>\n

We\u2019ll use Python and the Hugging Face transformers library.<\/p>\n

pip install transformers datasets torchimport torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from datasets import load_dataset<\/p>\n

We\u2019ll use the IMDb movie reviews\u00a0dataset.<\/p>\n

# Load the IMDb dataset
dataset = load_dataset(‘imdb’)<\/p>\n

# Use a subset for faster training (optional)
train_dataset = dataset[‘train’].shuffle(seed=42).select(range(2000))
test_dataset = dataset[‘test’].shuffle(seed=42).select(range(500))<\/p>\n

We\u2019ll use a large pre-trained model as the teacher, such as\u00a0BERT.<\/p>\n

teacher_model_name = ‘bert-base-uncased’
teacher_model = AutoModelForSequenceClassification.from_pretrained(teacher_model_name, num_labels=2)
teacher_tokenizer = AutoTokenizer.from_pretrained(teacher_model_name)<\/p>\n

Fine-tune the teacher model on the training\u00a0data.<\/p>\n

# Tokenize the training data
def tokenize(batch):
return teacher_tokenizer(batch[‘text’], padding=True, truncation=True)<\/p>\n

train_dataset = train_dataset.map(tokenize, batched=True)
train_dataset.set_format(‘torch’, columns=[‘input_ids’, ‘attention_mask’, ‘label’])<\/p>\n

# DataLoader
from torch.utils.data import DataLoader<\/p>\n

train_dataloader = DataLoader(train_dataset, batch_size=8, shuffle=True)<\/p>\n

# Optimizer and Scheduler
from transformers import AdamW, get_scheduler<\/p>\n

optimizer = AdamW(teacher_model.parameters(), lr=5e-5)
num_epochs = 2
num_training_steps = num_epochs * len(train_dataloader)
lr_scheduler = get_scheduler(
name=’linear’, optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
)<\/p>\n

# Training Loop
device = torch.device(‘cuda’) if torch.cuda.is_available() else torch.device(‘cpu’)
teacher_model.to(device)
teacher_model.train()<\/p>\n

from tqdm.auto import tqdm<\/p>\n

progress_bar = tqdm(range(num_training_steps))<\/p>\n

for epoch in range(num_epochs):
for batch in train_dataloader:
batch = {k: v.to(device) for k, v in batch.items()}
outputs = teacher_model(**batch)
loss = outputs.loss<\/p>\n

loss.backward()
optimizer.step()
lr_scheduler.step()
optimizer.zero_grad()
progress_bar.update(1)<\/p>\n

We\u2019ll use a smaller model for the student, such as DistilBERT.<\/p>\n

student_model_name = ‘distilbert-base-uncased’
student_model = AutoModelForSequenceClassification.from_pretrained(student_model_name, num_labels=2)
student_tokenizer = AutoTokenizer.from_pretrained(student_model_name)
student_model.to(device)
student_model.train()<\/p>\n

Tokenize the data using the student tokenizer.<\/p>\n

# Tokenize with student tokenizer
def tokenize_student(batch):
return student_tokenizer(batch[‘text’], padding=True, truncation=True)<\/p>\n

train_dataset_student = train_dataset.map(tokenize_student, batched=True)
train_dataset_student.set_format(‘torch’, columns=[‘input_ids’, ‘attention_mask’, ‘label’])
train_dataloader_student = DataLoader(train_dataset_student, batch_size=8)<\/p>\n

Train the student model using the teacher\u2019s outputs.<\/p>\n

# Loss Function
loss_fn = torch.nn.KLDivLoss(reduction=’batchmean’)<\/p>\n

# Training Loop for Distillation
temperature = 2.0
optimizer_student = AdamW(student_model.parameters(), lr=5e-5)
num_training_steps_student = num_epochs * len(train_dataloader_student)
lr_scheduler_student = get_scheduler(
name=’linear’, optimizer=optimizer_student, num_warmup_steps=0, num_training_steps=num_training_steps_student
)<\/p>\n

progress_bar_student = tqdm(range(num_training_steps_student))<\/p>\n

for epoch in range(num_epochs):
for batch_teacher, batch_student in zip(train_dataloader, train_dataloader_student):
# Move batches to device
batch_teacher = {k: v.to(device) for k, v in batch_teacher.items()}
batch_student = {k: v.to(device) for k, v in batch_student.items()}<\/p>\n

# Get teacher’s predictions
with torch.no_grad():
teacher_outputs = teacher_model(**batch_teacher)
teacher_logits = teacher_outputs.logits \/ temperature
teacher_probs = torch.nn.functional.softmax(teacher_logits, dim=-1)<\/p>\n

# Get student’s predictions
student_outputs = student_model(**batch_student)
student_logits = student_outputs.logits \/ temperature
student_log_probs = torch.nn.functional.log_softmax(student_logits, dim=-1)<\/p>\n

# Compute distillation loss
loss = loss_fn(student_log_probs, teacher_probs) * (temperature ** 2)<\/p>\n

# Backpropagation
loss.backward()
optimizer_student.step()
lr_scheduler_student.step()
optimizer_student.zero_grad()
progress_bar_student.update(1)# Prepare test data
test_dataset = test_dataset.map(tokenize_student, batched=True)
test_dataset.set_format(‘torch’, columns=[‘input_ids’, ‘attention_mask’, ‘label’])
test_dataloader = DataLoader(test_dataset, batch_size=8)<\/p>\n

# Evaluation Loop
student_model.eval()
correct = 0
total = 0<\/p>\n

with torch.no_grad():
for batch in test_dataloader:
batch = {k: v.to(device) for k, v in batch.items()}
outputs = student_model(**batch)
predictions = torch.argmax(outputs.logits, dim=-1)
correct += (predictions == batch[‘label’]).sum().item()
total += batch[‘label’].size(0)<\/p>\n

accuracy = correct \/ total
print(f’Student Model Accuracy: {accuracy * 100:.2f}%’)<\/p>\n

you get similar results as the set fit basic training but the prediction time is\u00a0reduced<\/p>\n

Conclusion:<\/strong><\/p>\n

we\u2019ve journeyed through the essential concepts of text classification, starting with how computers interpret text through vectorization. We explored how fine-tuning pre-trained models allows us to tailor these tools to our specific needs without starting from scratch. The introduction of the SetFit framework showcased a user-friendly and efficient way to fine-tune models with minimal data, making advanced text classification accessible to everyone\u200a\u2014\u200aeven those without extensive machine-learning expertise. By walking through practical code examples, we demonstrated how SetFit simplifies the process, enabling quick adaptation of models to accurately predict sentiments in\u00a0text.<\/p>\n

We also delved into the concept of knowledge distillation, illustrating how it helps create smaller, faster models that retain the performance of larger, more complex ones. This technique is invaluable for deploying models on devices with limited resources, ensuring efficiency without compromising accuracy. By combining SetFit\u2019s simplicity with the efficiency of knowledge distillation, we can harness powerful AI technologies to build practical, real-world applications. These tools not only make text classification more effective but also more accessible, paving the way for innovative solutions in various industries.<\/p>\n

Text Classification Made Easy with SetFit\u2026<\/a> was originally published in Coinmonks<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>","protected":false},"excerpt":{"rendered":"

What exactly is text classification? Text classification is like teaching a computer to sort written text into different categories\u200a\u2014\u200aimagine organizing emails into \u201cspam\u201d or \u201cinbox.\u201d The Hugging Face SetFit framework is a tool that makes this teaching process simpler and more efficient. It allows us to train computers to understand and classify text using only […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-16209","post","type-post","status-publish","format-standard","hentry","category-interesting"],"_links":{"self":[{"href":"https:\/\/mycryptomania.com\/index.php?rest_route=\/wp\/v2\/posts\/16209"}],"collection":[{"href":"https:\/\/mycryptomania.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mycryptomania.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/mycryptomania.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=16209"}],"version-history":[{"count":0,"href":"https:\/\/mycryptomania.com\/index.php?rest_route=\/wp\/v2\/posts\/16209\/revisions"}],"wp:attachment":[{"href":"https:\/\/mycryptomania.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=16209"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mycryptomania.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=16209"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mycryptomania.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=16209"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}