Project Report

Goal

Our goal is to make a model that can categorize websites based on their scraped data.

Challenges

The first problem was that we had non-labelled data. To train our model, we needed a labelled dataset. We initially tried clustering websites based on extracted keywords, but the clusters had too much noise.

Approaches

Pre-trained Models for Categorization

We tested two types of models:

• Fine-tuned models from Hugging Face: These models are faster, smaller, and require fewer resources. However, they are often trained on specific categories only, limiting their ability to categorize all websites accurately.
• Open-source LLM models from Ollama: These models (e.g., Meta Llama, Deepseek V3) are highly accurate but require substantial computational resources.

Ollama is a platform where quantized versions of LLM models are published.

Models Tested from Hugging Face

We tested the following text classification models:

• Ali Mazhar Text Classification (only 15 categories available)
• Tiabet 2 (only 5 categories available)
• Fine-Tuned Bart Model (overfitted on training model)
• Fine-Tuned Zero Shot (overfitted on training model)

These models fall into two types:

1. Pre-defined Category Models: These models directly classify text into categories but have limited flexibility as they are trained on specific datasets.
2. Zero-Shot Models: We define a list of categories, and the model evaluates the text to determine the most probable category.

Example:

sequence_to_classify = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(sequence_to_classify, candidate_labels)
# Output:
# {'labels': ['travel', 'dancing', 'cooking'],
#  'scores': [0.9938651323318481, 0.0032737774308770895, 0.002861034357920289],
#  'sequence': 'one day I will see the world'}

Final Decision

We decided to use the Facebook Zero-Shot Model.

Challenges with the Facebook Zero-Shot Model

• It requires clean text, but scraped website data contains irrelevant information (e.g., cookie notices).

Solutions

1. Facebook Bart Large CNN Model: Used for text summarization before classification. However, this increased processing time significantly.
2. Keyword Extraction: More time-efficient than summarization.

Testing & Results

We tested 1000 websites using the following categories:

['Adult And Dating', 'Advertisements', 'Agriculture', 'Alcohol And Tobacco', 'Articles', 'Astrology', 'Automobiles And Transportation', 'Beauty', 'Biography', 'Blogs And Forums', 'Business', 'Cloudflare', 'Construction', 'Drugs', 'Ecommerce And Shopping', 'Education', 'Entertainment', 'Environment', 'Fashion', 'Finance', 'Food And Beverages', 'Gambling', 'Gaming', 'Government', 'Hacking', 'Health', 'Healthcare', 'Hobbies And Interests', 'Home And Garden', 'Intolerance And Hate', 'Jobs Search', 'Life', 'Logistics', 'Medicine', 'News', 'Non Governmental Organization', 'Parked Domain', 'Personal And Portfolio Website', 'Pets And Animals', 'Politics', 'Pornography', 'Real Estate And Property', 'Religious', 'Research', 'Restaurants And Dining', 'Search Engines', 'Security And Defense', 'Services and Repair', 'Social Media Networking', 'Society And Culture', 'Sports', 'Stock Market', 'Technology', 'Tourism', 'Unknown']

First Iteration Results

• We manually categorized the 1000 websites and found 67.5% accuracy.
• We discovered additional categories missing from our initial list:

['Adult Entertainment', 'Architecture', 'Association', 'Auditons', 'AutoMobile', 'Automation', 'Automobiles', 'Business/Technology', 'Bussiness/Nonprofit', 'Coding', 'Constuction', 'Craked APPS', 'Crypto', 'Cybersecurity', 'Data Security', 'Dating', 'E- Commerce', 'E-Commerce', 'Ecommerce', 'Entertainment/Online Video Streaming', 'Escort Services', 'Food and Beverages', 'Hospitality', 'House Interior', 'INNOVATION/Reseach', 'Insurance', 'Jewellery', 'Legal', 'Lighting/Innovation', 'MARKETING', 'MARKETING/AUTOMATION', 'Manufacor', 'Manufactror', 'Manufactur', 'Manufacture', 'Manufacture and supplier', 'Marketing', 'Media/News', 'Music', 'NGO/Charity', 'News/ Blogs', 'ONLINE TOOLS STORE', 'Online Interview Preparation', 'Online Pdf Convertor', 'Online Store', 'Payments', 'Printing', 'Real Estate', 'Recycleing', 'Redirect To Google', 'Rewards And Coupens', 'SUPPLEMENTS', 'Social Media', 'Startup', 'Stationery Items', 'Techonology', 'Trust & Safety', 'UNCATEGORIZED', 'Visualization/Technology', 'WOOD', 'malicious', 'pharmaceutical']

Second Iteration Results

• After updating the category list and retesting, the model’s accuracy improved to ~90%.
• Note: Multilingual websites require translation for better classification.

AI Models for Improved Accuracy

For better accuracy, we need at least two predictions:

1. Facebook Zero-Shot Model
2. An LLM Model

Benefits of LLMs

• Can ignore irrelevant text with better prompting.
• Some are multilingual, eliminating the need for translation.
• High accuracy, even with smaller models.

Best LLM Model Identified

• Llama3.1-instruct-405b was found to be the most capable LLM.

*Note: This research was conducted before DeepseekV3 was released. A detailed comparison between Llama3.1 405B and Deepseek V3 is included at the end of the document.

Open Source Models and Quantization

The open-source models available to use locally on our systems are always in quantized form.

Explanation of Quantization Types

Quantization is a technique used to reduce the memory and computational requirements of AI models by representing weights with fewer bits. The different quantization types listed here refer to how the weights are stored and processed. Here’s what they mean:

1. fp16 (Full-Precision 16-bit Floating Point)

• Type: Floating point
• Description: Uses 16-bit floating-point (FP16) numbers to represent weights.
• Pros: Higher precision, better accuracy.
• Cons: Requires more memory compared to lower-bit quantized models.
• Use Case: Best for high-performance inference with minimal accuracy loss.

2. q2_K, q3_K_L, q3_K_M, q3_K_S (2-bit & 3-bit Quantization)

• Type: Low-bit integer quantization
• Description:
  • q2_K → 2-bit quantization (very aggressive compression).
  • q3_K_L → 3-bit quantization (Large variant).
  • q3_K_M → 3-bit quantization (Medium variant).
  • q3_K_S → 3-bit quantization (Small variant).
• Pros: Very small model size, extremely fast inference.
• Cons: High accuracy loss compared to higher-bit quantizations.
• Use Case: Suitable for edge devices or applications needing extreme memory efficiency.

3. q4_0, q4_1, q4_K_M, q4_K_S (4-bit Quantization)

• Type: Moderate quantization
• Description:
  • q4_0 → 4-bit quantization (basic method, less accurate).
  • q4_1 → 4-bit quantization (slightly better precision than q4_0).
  • q4_K_M → 4-bit quantization (Medium variant, optimized for speed/accuracy balance).
  • q4_K_S → 4-bit quantization (Small variant, highly compressed).
• Pros: Good balance of speed, compression, and accuracy.
• Cons: Slight performance drop compared to FP16.
• Use Case: Ideal for cloud inference or situations where model size matters but accuracy is still important.

4. q5_0, q5_1, q5_K_M, q5_K_S (5-bit Quantization)

• Type: Higher precision quantization
• Description:
  • q5_0 → 5-bit quantization (basic method).
  • q5_1 → 5-bit quantization (slightly better accuracy than q5_0).
  • q5_K_M → 5-bit quantization (Medium variant, accuracy-focused).
  • q5_K_S → 5-bit quantization (Small variant, optimized for storage).
• Pros: Higher accuracy than 4-bit quantization while keeping good compression.
• Cons: Slightly more memory usage than 4-bit.
• Use Case: Useful when better accuracy is needed but still with compression benefits.

5. q6_K (6-bit Quantization)

• Type: High precision quantization
• Description:
  • Uses 6-bit integer weights, maintaining a strong balance between compression and accuracy.
• Pros: Very close to FP16 performance while reducing memory usage significantly.
• Cons: Slightly larger than lower-bit quantizations but much more efficient than FP16.
• Use Case: Best for deployment where accuracy is critical but memory is still a concern.

6. q8_0 (8-bit Quantization)

• Type: Near full-precision quantization
• Description:
  • Uses 8-bit integers instead of floating-point numbers.
• Pros:
  • Minimal accuracy loss compared to FP16.
  • Faster inference speed.
  • Lower memory footprint than FP16 but higher than lower-bit quantizations.
• Cons: Requires more memory than 4-bit or 5-bit quantizations.
• Use Case: Best for latency-sensitive applications where accuracy still matters.

Summary of Quantization Types

| Quantization Type | Bit Size | Pros | Cons | Best Use Case | |———————-|————|———-|———-|—————-| | fp16 | 16-bit | High precision, best accuracy | High memory usage | Best for performance-critical tasks | | q2_K | 2-bit | Extremely small model size | High accuracy loss | Extreme memory-constrained environments | | q3_K_L/M/S | 3-bit | Very efficient | Significant accuracy loss | Edge devices, ultra-low memory use | | q4_0, q4_1, q4_K_M/S | 4-bit | Good compression-accuracy tradeoff | Some performance drop | Cloud inference, mobile AI | | q5_0, q5_1, q5_K_M/S | 5-bit | Higher accuracy than 4-bit | Slightly larger size | General-purpose AI tasks | | q6_K | 6-bit | Almost FP16 accuracy | Slightly higher memory usage | Balanced accuracy vs efficiency | | q8_0 | 8-bit | Near-FP16 performance, fast | Uses more memory than 4-bit | Latency-sensitive applications |

Q4 is the standard quantisation for all models. We used the q4 quantised version of Llama3 (not 3.1) for testing.

First we downloaded a dataset of top 1000 websites from github. We were able to scrape 363 websites out of them. These websites were of following domains.

[‘Energy’ ‘Motor Vehicles & Parts’ ‘Wholesalers’ ‘Technology’ ‘Financials’ ‘Retailing’ ‘Food & Drug Stores’ ‘Aerospace & Defense’ ‘Household Products’ ‘Health Care’ ‘Food, Beverages & Tobacco’ ‘Chemicals’ ‘Industrials’ ‘Materials’ ‘Transportation’ ‘Business Services’ ‘Media’ ‘Hotels, Resturants & Leisure’ ‘Engineering & Construction’ ‘Apparel’ ‘Telecommunications’]

We were able to achieve an accuracy of 90 percent. Following were the websites that were categorized incorrectly.

*Note: All the financial/crypto/insurance sites were categorised with 100 percent accuracy.

Then we scraped the same 1000 websites that we did with Facebook Zero Shot Model.

Again the model gave an accuracy of nearly 90 percent accuracy. But the major problem was coming in categorizing pornographic websites.

To make this project completely foolproof, we will apply 3 layers of predictions. First we will categorize a large number of websites using above mentioned models. Then we will make our own model that will be trained on that labelled dataset. Finally with these 3 layers of categorization we will be able to make the whole project extremely accurate.

Detailed Hardware & CPU Offloading Table For LLaMA3.1:405B

We were not able to run Deekseek V3 locally due to hardware limitations and there are not many reliable sources to compare the performance of V3 with Llama3.1. The below mentioned table is taken from official deekseek website.

According to this, deepseek is better than Llama in almost all categories. If we want to run deepseek locally then here are the hardware requirements for it.