Scikit-LLM vs. Conventional Textual content Classifiers: When Ought to You Use an LLM?

On this article, you’ll learn to benchmark three textual content classification approaches — from a classical TF-IDF pipeline to a zero-shot giant language mannequin — to grasp when every is most applicable.

Subjects we are going to cowl embrace:

The best way to implement and consider a classical TF-IDF and logistic regression textual content classification pipeline.
The best way to apply zero-shot classification utilizing a transformer-based mannequin (BART) and evaluate it in opposition to the classical baseline.
The best way to use scikit-LLM with a Groq-hosted giant language mannequin for production-ready zero-shot classification with minimal code adjustments.

Scikit-LLM vs. Traditional Text Classifiers: When Should You Use an LLM?

Scikit-LLM vs. Conventional Textual content Classifiers: When Ought to You Use an LLM?

Introduction

In recent times, generative AI fashions like LLMs (giant language fashions) have steadily taken over classical machine studying ones for addressing sure duties, as an example, textual content classification. However the fact is: somewhat than having a one-beats-all resolution, there are important trade-offs builders have to face — ought to we persist with quick, battle-tested typical fashions, spend money on fine-tuning a transformer-based LLM, or maybe leverage LLMs’ zero-shot reasoning potential?

On this article, we are going to implement a benchmarking between three distinct approaches for textual content classification:

TF-IDF and logistic regression (traditional baseline).
Zero-shot classification with BART: a deep studying, transformer-based normal structure.
Scikit-LLM with zero-shot classification: probably the most trendy, prompt-based strategy.

The tutorial under is saved fully free for everybody to attempt, with no prices or API fee limits. To take action, we are going to use scikit-LLM alongside a mannequin obtainable from Groq. You will want to register at Groq and procure an API key for evaluating the third resolution under.

Implementing the Benchmarking

First, we set up all of the core libraries we are going to want.

!pip set up scikit-learn transformers scikit-llm scikit-ollama pandas torch

!pip set up scikit–be taught transformers scikit–llm scikit–ollama pandas torch

For enabling reproducibility, we create a small, artificial dataset containing buyer assist messages. The tickets are categorized into 5 courses. As soon as created, we retailer it in a DataFrame object and cut up it into coaching and check units.

import pandas as pd from sklearn.model_selection import train_test_split knowledge = { “textual content”: [ # Technical “My screen is completely black and won’t turn on.”, “The app keeps crashing every time I click save.”, “The Wi-Fi module is failing to connect to the router.”, “Data sync isn’t working across my devices.”, “My bluetooth headphones won’t pair with the app.”, “I keep getting an Error 404 on the login screen.”, “The database connection timed out during the export.”, “API rate limit exceeded even though I haven’t used it.”, “Profile images won’t load on the dashboard.”, “The software installation failed at 99%.”, # Billing “I was charged twice this month, please fix this.”, “How do I update my credit card information?”, “My invoice for last month is missing from the portal.”, “The VAT calculation on my receipt is wrong.”, “My transaction was declined but I have funds.”, “Can I change my billing cycle from monthly to annual?”, “Where can I find my official receipt?”, “My saved credit card expired and I need to swap it.”, “I was overcharged on my last statement.”, “Please remove my saved payment method.”, # Account “My account is locked and I forgot my password.”, “How do I change the email address on my profile?”, “Please delete my account and all associated data.”, “I want to update my profile picture.”, “How do I enable two-factor authentication (2FA)?”, “I didn’t receive the email verification link.”, “Can I merge two different accounts into one?”, “Is there a way to change my username?”, “I need to transfer account ownership to my manager.”, “I am locked out because I lost my 2FA phone.”, # Sales “Do you offer enterprise discounts for large teams?”, “Do you have an annual plan with a discount?”, “Can you compare the pro and basic tiers for me?”, “What is the pricing for a 50-user bulk license?”, “Is there a student discount available?”, “Can I schedule a demo with your sales team?”, “Do you sell and ship to customers in Europe?”, “How does your partner and reseller program work?”, “What are the usage limits on the free tier?”, “I need a custom quote for a government contract.”, # Refund “Can I get a refund for my last purchase? It was a mistake.”, “I want my money back for the subscription.”, “Accidental purchase, please reverse the charge.”, “I am not satisfied with the product, need a refund.”, “Cancel my subscription immediately and refund me.”, “I was charged after my free trial ended.”, “I need a prorated refund for the remaining months.”, “What is your official refund policy?”, “I was promised a refund last week but haven’t received it.”, “The item arrived broken, I want a full refund.” ], “label”: [ “Technical”] * 10 + [“Billing”] * 10 + [“Account”] * 10 + [“Sales”] * 10 + [“Refund”] * 10 } df = pd.DataFrame(knowledge) # Stratified train-test splitting ensures all 5 classes are proportionally represented in each subsets when the dataset is small X_train, X_test, y_train, y_test = train_test_split( df[“text”], df[“label”], test_size=0.3, random_state=42, stratify=df[“label”] ) print(f”Coaching rows: {len(X_train)} | Testing rows: {len(X_test)}”)

import pandas as pd

from sklearn.model_selection import train_test_split

knowledge = {

“textual content”: [

# Technical

“My screen is completely black and won’t turn on.”, “The app keeps crashing every time I click save.”,

“The Wi-Fi module is failing to connect to the router.”, “Data sync isn’t working across my devices.”,

“My bluetooth headphones won’t pair with the app.”, “I keep getting an Error 404 on the login screen.”,

“The database connection timed out during the export.”, “API rate limit exceeded even though I haven’t used it.”,

“Profile images won’t load on the dashboard.”, “The software installation failed at 99%.”,

# Billing

“I was charged twice this month, please fix this.”, “How do I update my credit card information?”,

“My invoice for last month is missing from the portal.”, “The VAT calculation on my receipt is wrong.”,

“My transaction was declined but I have funds.”, “Can I change my billing cycle from monthly to annual?”,

“Where can I find my official receipt?”, “My saved credit card expired and I need to swap it.”,

“I was overcharged on my last statement.”, “Please remove my saved payment method.”,

# Account

“My account is locked and I forgot my password.”, “How do I change the email address on my profile?”,

“Please delete my account and all associated data.”, “I want to update my profile picture.”,

“How do I enable two-factor authentication (2FA)?”, “I didn’t receive the email verification link.”,

“Can I merge two different accounts into one?”, “Is there a way to change my username?”,

“I need to transfer account ownership to my manager.”, “I am locked out because I lost my 2FA phone.”,

# Sales

“Do you offer enterprise discounts for large teams?”, “Do you have an annual plan with a discount?”,

“Can you compare the pro and basic tiers for me?”, “What is the pricing for a 50-user bulk license?”,

“Is there a student discount available?”, “Can I schedule a demo with your sales team?”,

“Do you sell and ship to customers in Europe?”, “How does your partner and reseller program work?”,

“What are the usage limits on the free tier?”, “I need a custom quote for a government contract.”,

# Refund

“Can I get a refund for my last purchase? It was a mistake.”, “I want my money back for the subscription.”,

“Accidental purchase, please reverse the charge.”, “I am not satisfied with the product, need a refund.”,

“Cancel my subscription immediately and refund me.”, “I was charged after my free trial ended.”,

“I need a prorated refund for the remaining months.”, “What is your official refund policy?”,

“I was promised a refund last week but haven’t received it.”, “The item arrived broken, I want a full refund.”

“label”: [

“Technical”] * 10 + [“Billing”] * 10 + [“Account”] * 10 + [“Sales”] * 10 + [“Refund”] * 10

}

df = pd.DataFrame(knowledge)

# Stratified train-test splitting ensures all 5 classes are proportionally represented in each subsets when the dataset is small

X_train, X_test, y_train, y_test = train_test_split(

df[“text”], df[“label”], test_size=0.3, random_state=42, stratify=df[“label”]

)

print(f“Coaching rows: {len(X_train)} | Testing rows: {len(X_test)}”)

We first implement and consider probably the most classical strategy: TF-IDF mixed with a logistic regression classifier. The method is proven under:

import time from sklearn.feature_extraction.textual content import TfidfVectorizer from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline from sklearn.metrics import classification_report start_time = time.time() # Creating and coaching the classical pipeline logreg_clf = make_pipeline(TfidfVectorizer(), LogisticRegression()) logreg_clf.match(X_train, y_train) # Inference: predictions on the check examples y_pred_logreg = logreg_clf.predict(X_test) logreg_latency = time.time() – start_time # Latency can be measured to evaluate the mannequin’s effectivity print(f”Logistic Regression Latency: {logreg_latency:.4f} seconds”) print(classification_report(y_test, y_pred_logreg, zero_division=0))

import time

from sklearn.feature_extraction.textual content import TfidfVectorizer

from sklearn.linear_model import LogisticRegression

from sklearn.pipeline import make_pipeline

from sklearn.metrics import classification_report

start_time = time.time()

# Creating and coaching the classical pipeline

logreg_clf = make_pipeline(TfidfVectorizer(), LogisticRegression())

logreg_clf.match(X_train, y_train)

# Inference: predictions on the check examples

y_pred_logreg = logreg_clf.predict(X_test)

logreg_latency = time.time() – begin_time

# Latency can be measured to evaluate the mannequin’s effectivity

print(f“Logistic Regression Latency: {logreg_latency:.4f} seconds”)

print(classification_report(y_test, y_pred_logreg, zero_division=0))

Output:

Logistic Regression Latency: 0.0615 seconds precision recall f1-score assist Account 0.25 0.33 0.29 3 Billing 1.00 1.00 1.00 3 Refund 0.67 0.67 0.67 3 Gross sales 0.25 0.33 0.29 3 Technical 1.00 0.33 0.50 3 accuracy 0.53 15 macro avg 0.63 0.53 0.55 15 weighted avg 0.63 0.53 0.55 15

Logistic Regression Latency: 0.0615 seconds

precision recall f1–rating assist

Account 0.25 0.33 0.29 3

Billing 1.00 1.00 1.00 3

Refund 0.67 0.67 0.67 3

Gross sales 0.25 0.33 0.29 3

Technical 1.00 0.33 0.50 3

accuracy 0.53 15

macro avg 0.63 0.53 0.55 15

weighted avg 0.63 0.53 0.55 15

The classifier exhibits a combined conduct: it performs nicely on classes like Billing and, to some extent, Refund, however struggles with the remaining. That is the quickest strategy by far; nonetheless, its classification efficiency is restricted by its incapacity to seize the advanced linguistic nuances that extra trendy language fashions can successfully deal with. Sticking to aggregated outcomes, we get accuracies ranging between 0.53 and 0.55 general.

Let’s see what our second strategy — zero-shot classification with fb/bart-large-mnli — has to supply:

from transformers import pipeline import time # Utilizing a HuggingFace zero-shot classification pipeline as our transformer consultant # We have to overload the default classifier to specify our personal label set classifier = pipeline(“zero-shot-classification”, mannequin=”fb/bart-large-mnli”) candidate_labels = [“Technical”, “Billing”, “Account”, “Sales”, “Refund”] start_time = time.time() # Inference time! bert_preds = [] for textual content in X_test: end result = classifier(textual content, candidate_labels) bert_preds.append(end result[‘labels’][0]) # Get the very best scoring label bert_latency = time.time() – start_time print(f”Transformer Inference Latency: {bert_latency:.4f} seconds”) print(classification_report(y_test, bert_preds, zero_division=0))

from transformers import pipeline

import time

# Utilizing a HuggingFace zero-shot classification pipeline as our transformer consultant

# We have to overload the default classifier to specify our personal label set

classifier = pipeline(“zero-shot-classification”, mannequin=“fb/bart-large-mnli”)

candidate_labels = [“Technical”, “Billing”, “Account”, “Sales”, “Refund”]

start_time = time.time()

# Inference time!

bert_preds = []

for textual content in X_test:

end result = classifier(textual content, candidate_labels)

bert_preds.append(end result[‘labels’][0]) # Get the very best scoring label

bert_latency = time.time() – start_time

print(f“Transformer Inference Latency: {bert_latency:.4f} seconds”)

print(classification_report(y_test, bert_preds, zero_division=0))

These are the outcomes:

Transformer Inference Latency: 32.2503 seconds precision recall f1-score assist Account 0.40 0.67 0.50 3 Billing 1.00 0.33 0.50 3 Refund 0.75 1.00 0.86 3 Gross sales 1.00 0.33 0.50 3 Technical 0.75 1.00 0.86 3 accuracy 0.67 15 macro avg 0.78 0.67 0.64 15 weighted avg 0.78 0.67 0.64 15

Transformer Inference Latency: 32.2503 seconds

precision recall f1–rating assist

Account 0.40 0.67 0.50 3

Billing 1.00 0.33 0.50 3

Refund 0.75 1.00 0.86 3

Gross sales 1.00 0.33 0.50 3

Technical 0.75 1.00 0.86 3

accuracy 0.67 15

macro avg 0.78 0.67 0.64 15

weighted avg 0.78 0.67 0.64 15

A lot greater latency, and solely a modest enchancment in accuracy: 0.64–0.67 in broad phrases.

Lastly, the zero-shot LLM classifier with a scikit-LLM pipeline and a Groq mannequin:

from skllm.config import SKLLMConfig from skllm.fashions.gpt.classification.zero_shot import ZeroShotGPTClassifier import getpass import time from sklearn.metrics import classification_report # 1. Securely asking for the important thing in a non-public enter field: # GET YOURS AT https://console.groq.com/keys print(“Get your free Groq API key right here: https://console.groq.com/keys”) api_key = getpass.getpass(“Paste your API Key right here: “) # 2. Configuring Scikit-LLM SKLLMConfig.set_openai_key(api_key) SKLLMConfig.set_gpt_url(“https://api.groq.com/openai/v1/”) # 3. Initializing with the most recent lively mannequin for zero-shot classification # ‘llama-3.3-70b-versatile’ is supported by Groq on the time of writing llm_clf = ZeroShotGPTClassifier(mannequin=”custom_url::llama-3.3-70b-versatile”) start_time = time.time() # 4. Operating the classification activity llm_clf.match(X_train, y_train) y_pred_llm = llm_clf.predict(X_test) llm_latency = time.time() – start_time print(f”nScikit-LLM Latency: {llm_latency:.4f} seconds”) print(classification_report(y_test, y_pred_llm, zero_division=0))

from skllm.config import SKLLMConfig

from skllm.fashions.gpt.classification.zero_shot import ZeroShotGPTClassifier

import getpass

import time

from sklearn.metrics import classification_report

# 1. Securely asking for the important thing in a non-public enter field:

# GET YOURS AT https://console.groq.com/keys

print(“Get your free Groq API key right here: https://console.groq.com/keys”)

api_key = getpass.getpass(“Paste your API Key right here: “)

# 2. Configuring Scikit-LLM

SKLLMConfig.set_openai_key(api_key)

SKLLMConfig.set_gpt_url(“https://api.groq.com/openai/v1/”)

# 3. Initializing with the most recent lively mannequin for zero-shot classification

# ‘llama-3.3-70b-versatile’ is supported by Groq on the time of writing

llm_clf = ZeroShotGPTClassifier(mannequin=“custom_url::llama-3.3-70b-versatile”)

start_time = time.time()

# 4. Operating the classification activity

llm_clf.match(X_train, y_train)

y_pred_llm = llm_clf.predict(X_test)

llm_latency = time.time() – start_time

print(f“nScikit-LLM Latency: {llm_latency:.4f} seconds”)

print(classification_report(y_test, y_pred_llm, zero_division=0))

Remaining outcomes:

Scikit-LLM Latency: 2.5905 seconds precision recall f1-score assist Account 0.67 0.67 0.67 3 Billing 1.00 0.67 0.80 3 Refund 1.00 1.00 1.00 3 Gross sales 1.00 1.00 1.00 3 Technical 0.75 1.00 0.86 3 accuracy 0.87 15 macro avg 0.88 0.87 0.86 15 weighted avg 0.88 0.87 0.86 15

Scikit–LLM Latency: 2.5905 seconds

precision recall f1–rating assist

Account 0.67 0.67 0.67 3

Billing 1.00 0.67 0.80 3

Refund 1.00 1.00 1.00 3

Gross sales 1.00 1.00 1.00 3

Technical 0.75 1.00 0.86 3

accuracy 0.87 15

macro avg 0.88 0.87 0.86 15

weighted avg 0.88 0.87 0.86 15

That is by far the very best end result by way of classification accuracy (0.86–0.87). And surprisingly, it’s also significantly sooner than the BART-based zero-shot mannequin. This isn’t all that shocking: the Groq-hosted mannequin was educated on an enormous, broad dataset. It doesn’t have to be taught what a given sort of buyer assist ticket means — it already is aware of, not like the zero-shot BART mannequin used earlier.

So, we now have a transparent winner!

On a last notice: that is the place the worth of scikit-LLM lies. It bridges the hole between classical and trendy AI by means of a standardized, production-ready interface, utilizing scikit-learn-like syntax all through. With this in hand, you may swap between a classical logistic regressor and a contemporary Groq LLM with minimal effort.

Wrapping Up

This text benchmarked, on a toy dataset, scikit-LLM’s zero-shot classification in opposition to extra classical approaches — logistic regression with TF-IDF, and a zero-shot transformer mannequin (BART) sitting someplace in between. As for the query posed within the title, when do you have to use an LLM for textual content classification? The selection of a small, toy dataset right here was deliberate. When the quantity of obtainable knowledge is restricted and the duty requires deep linguistic reasoning and contextual understanding, scikit-LLM is a compelling asset: it makes it attainable to immediately deploy a mannequin’s pre-trained world data right into a pipeline like ours, eliminating each the time and infrastructure prices of coaching a mannequin of this magnitude from scratch.