txtai-arxiv

from txtai.embeddings import Embeddings

# Load the index from the HF Hub
embeddings = Embeddings()
embeddings.load(provider="huggingface-hub", container="neuml/txtai-arxiv")

# Search for papers matching a query
embeddings.search("Survey of vector databases")

# Search for papers matching an abstract
embeddings.search("""
Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral
has the same architecture as Mistral 7B, with the difference that each
layer is composed of 8 feedforward  blocks (i.e. experts). For every
token, at each layer, a router network selects two experts to process
the current state and combine their outputs.
""")

embeddings.search("""
Humanity has wondered whether we are alone for millennia. The discovery
of life elsewhere in the Universe, particularly intelligent life, would
have profound effects, comparable to those of recognizing that the Earth
is not the center of the Universe and that humans evolved from previous
species.
""")

embeddings.search("""
The main objective of this paper is to investigate the extent to which
the margin of victory can be predicted solely by the rankings of the
opposing teams in NCAA Division I men's basketball games.
""")

pip install ragdata

python -m ragdata.arxiv.index \
       -d <path to directory with file downloaded in previous step> \
       -o txtai-arxiv