LLM2CLIP-Llama3.2-1B-EVA02-L-14-336
10
1.0B
license:apache-2.0
by
microsoft
Image Model
OTHER
1B params
New
0 downloads
Early-stage
Edge AI:
Mobile
Laptop
Server
3GB+ RAM
Mobile
Laptop
Server
Quick Summary
LLM2CLIP: Extending the Capability Boundaries of CLIP through Large Language Models Weiquan Huang 1 , Aoqi Wu 1 , Yifan Yang 2† , Xufang Luo 2 , Yuqing Yang 2 ,...
Device Compatibility
Mobile
4-6GB RAM
Laptop
16GB RAM
Server
GPU
Minimum Recommended
1GB+ RAM
Training Data Analysis
🟡 Average (4.8/10)
Researched training datasets used by LLM2CLIP-Llama3.2-1B-EVA02-L-14-336 with quality assessment
Specialized For
general
science
multilingual
reasoning
Training Datasets (4)
common crawl
🔴 2.5/10
general
science
Key Strengths
- •Scale and Accessibility: At 9.5+ petabytes, Common Crawl provides unprecedented scale for training d...
- •Diversity: The dataset captures billions of web pages across multiple domains and content types, ena...
- •Comprehensive Coverage: Despite limitations, Common Crawl attempts to represent the broader web acro...
Considerations
- •Biased Coverage: The crawling process prioritizes frequently linked domains, making content from dig...
- •Large-Scale Problematic Content: Contains significant amounts of hate speech, pornography, violent c...
c4
🔵 6/10
general
multilingual
Key Strengths
- •Scale and Accessibility: 750GB of publicly available, filtered text
- •Systematic Filtering: Documented heuristics enable reproducibility
- •Language Diversity: Despite English-only, captures diverse writing styles
Considerations
- •English-Only: Limits multilingual applications
- •Filtering Limitations: Offensive content and low-quality text remain despite filtering
wikipedia
🟡 5/10
science
multilingual
Key Strengths
- •High-Quality Content: Wikipedia articles are subject to community review, fact-checking, and citatio...
- •Multilingual Coverage: Available in 300+ languages, enabling training of models that understand and ...
- •Structured Knowledge: Articles follow consistent formatting with clear sections, allowing models to ...
Considerations
- •Language Inequality: Low-resource language editions have significantly lower quality, fewer articles...
- •Biased Coverage: Reflects biases in contributor demographics; topics related to Western culture and ...
arxiv
🟡 5.5/10
science
reasoning
Key Strengths
- •Scientific Authority: Peer-reviewed content from established repository
- •Domain-Specific: Specialized vocabulary and concepts
- •Mathematical Content: Includes complex equations and notation
Considerations
- •Specialized: Primarily technical and mathematical content
- •English-Heavy: Predominantly English-language papers
Explore our comprehensive training dataset analysis
View All DatasetsCode Examples
Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Usagepythontransformers
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoModel, AutoConfig, AutoTokenizer
from eva_clip import create_model_and_transforms
from llm2vec import LLM2Vec
from PIL import Image
import torch.nn as nn
import torch
class LLM2VecWrapper(LLM2Vec):
def prepare_for_tokenization(self, text):
text = (
"<|start_header_id|>user<|end_header_id|>\n\n"
+ text.strip()
+ "<|eot_id|>"
)
return text
class LlamaVec_1B_FeatureExtractor(nn.Module):
def __init__(self):
super().__init__()
model_path = 'microsoft/LLM2CLIP-Llama-3.2-1B-Instruct-CC-Finetuned'
config = AutoConfig.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
self.l2v = LLM2VecWrapper(model, tokenizer, pooling_mode="mean", max_length=512, skip_instruction=True)
def extract_features(self, text):
with torch.amp.autocast('cuda'):
reps_norm = self.l2v.encode(text, convert_to_tensor=True)
reps_norm = torch.nn.functional.normalize(reps_norm, p=2, dim=1)
return reps_norm
model, _, preprocess_val = create_model_and_transforms('Llama3.2-1B-EVA02-L-14-336', force_custom_clip=True)
ckpt = torch.load('LLM2CLIP-Llama3.2-1B-EVA02-L-14-336.pt')
model.load_state_dict(ckpt)
model = model.cuda().eval()
text_model = LlamaVec_1B_FeatureExtractor()
image_path = "CLIP.png"
captions = ["a diagram", "a dog", "a cat"]
image = preprocess_val(Image.open(image_path)).cuda().unsqueeze(dim=0)
text_features = text_model.extract_features(captions).to('cuda')
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text_features)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)Deploy This Model
Production-ready deployment in minutes
Together.ai
Instant API access to this model
Production-ready inference API. Start free, scale to millions.
Try Free APIReplicate
One-click model deployment
Run models in the cloud with simple API. No DevOps required.
Deploy NowDisclosure: We may earn a commission from these partners. This helps keep LLMYourWay free.