rasyosef

--- tags: - sentence-transformers - sparse-encoder - sparse - splade - generated_from_trainer - dataset_size:1200000 - loss:SpladeLoss - loss:SparseMarginMSELoss - loss:FlopsLoss base_model: - praj...

This is a smaller version of the Meta's Llama-3.2-1B decoder transformer model pretrained from scratch for 26 hours using a single A100 40GB GPU 274 million tokens of Amharic text. - It has 180 Million parameters - The context size of this model is 1024 tokens. - It has the same tokenizer as Llama-3.2-1B, trained from scratch using the same Amharic dataset as the model with a vocabulary size of 32k. - This is a base model and hasn't undergone any supervised finetuing yet. The isntruction following version of this model is Llama-3.2-180M-Amharic-Instruct How to use First, you need to install the latest version of transformers You can use this model directly with a pipeline for text generation:

This is a SPLADE sparse retrieval model based on BERT-Small (29M) that was trained by distilling a Cross-Encoder on the MSMARCO dataset. The cross-encoder used was ms-marco-MiniLM-L6-v2. This SPLADE model is `2x` smaller than Naver's official `splade-v3-distilbert` while having `91%` of it's performance on the MSMARCO benchmark. This model is small enough to be used without a GPU on a dataset of a few thousand documents. - `Collection:` https://huggingface.co/collections/rasyosef/splade-tiny-msmarco-687c548c0691d95babf65b70 - `Distillation Dataset:` https://huggingface.co/datasets/yosefw/msmarco-train-distil-v2 - `Code:` https://github.com/rasyosef/splade-tiny-msmarco The splade models were evaluated on 55 thousand queries and 8.84 million documents from the MSMARCO dataset. ||Size (# Params)|MRR@10 (MS MARCO dev)| |:---|:----|:-------------------| |`BM25`|-|18.0|-|-| |`rasyosef/splade-tiny`|4.4M|30.9| |`rasyosef/splade-mini`|11.2M|34.1| |`rasyosef/splade-small`|28.8M|35.4| |`naver/splade-v3-distilbert`|67.0M|38.7| Model Description - Model Type: SPLADE Sparse Encoder - Base model: prajjwal1/bert-small - Maximum Sequence Length: 512 tokens - Output Dimensionality: 30522 dimensions - Similarity Function: Dot Product - Documentation: Sentence Transformers Documentation - Documentation: Sparse Encoder Documentation - Repository: Sentence Transformers on GitHub - Hugging Face: Sparse Encoders on Hugging Face | Metric | Value | |:----------------------|:-----------| | dotaccuracy@1 | 0.4547 | | dotaccuracy@3 | 0.7685 | | dotaccuracy@5 | 0.8786 | | dotaccuracy@10 | 0.9484 | | dotprecision@1 | 0.4547 | | dotprecision@3 | 0.2634 | | dotprecision@5 | 0.1828 | | dotprecision@10 | 0.0998 | | dotrecall@1 | 0.44 | | dotrecall@3 | 0.7543 | | dotrecall@5 | 0.8678 | | dotrecall@10 | 0.9424 | | dotndcg@10 | 0.7031 | | dotmrr@10 | 0.6288 | | dotmap@100 | 0.6253 | | queryactivedims | 24.9142 | | querysparsityratio | 0.9992 | | corpusactivedims | 171.8592 | | corpussparsityratio | 0.9944 | Size: 800,000 training samples Columns: query , positive , negative1 , negative2 , negative3 , negative4 , and label Approximate statistics based on the first 1000 samples: | | query | positive | negative1 | negative2 | negative3 | negative4 | label | |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:-----------------------------------| | type | string | string | string | string | string | string | list | | details | min: 4 tokens mean: 8.96 tokens max: 37 tokens | min: 19 tokens mean: 79.51 tokens max: 230 tokens | min: 22 tokens mean: 78.09 tokens max: 203 tokens | min: 14 tokens mean: 77.84 tokens max: 215 tokens | min: 15 tokens mean: 76.65 tokens max: 249 tokens | min: 19 tokens mean: 74.67 tokens max: 227 tokens | size: 4 elements | Samples: | query | positive | negative1 | negative2 | negative3 | negative4 | label | |:----------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------| | who was president during detente | Détente ended after the Soviet intervention in Afghanistan, which led to the United States boycott of the 1980 Olympics in Moscow. Ronald Reagan's election as president in 1980, based in large part on an anti-détente campaign, marked the close of détente and a return to Cold War tensions. | Soviet Premier Alexei Kosygin (front) next to U.S. President Lyndon B. Johnson (behind) during the Glassboro Summit Conference The most obvious manifestation of détente was the series of summits held between the leaders of the two superpowers and the treaties that resulted from these meetings. | The activities of President Ronald Reagan returned tensions to a fever pitch. Soviet relations with the People's Republic of China Détente could probably not have taken place, and certainly wouldn't have assumed the form that it did, without the rift that developed between the world's two primary communist regimes, the Soviet Union and the People's Republic of China (PRC). | Détente is the easing of strained relations, especially in a political situation. The term originates in the time of the Triple Entente and Entente cordiale in reference to an easing of tensions between England and France who, subsequent to being commingled polities under Norman rule, were warring rivals for the better part of a millennium but pursuant to a policy of détente became enduring allies. In the context of the Cold War, the lessening of tensions between the East and West, along ... | Détente (French pronunciation: ​[detɑ̃t], meaning relaxation)[1] is the easing of strained relations, especially in political situation. | [1.0, 2.1879749298095703, 8.371654510498047, 10.16702938079834] | | what is an ftp file | File Transfer Protocol (FTP) is a standard Internet protocol for transmitting files between computers on the Internet over TCP/IP connections. FTP is a client-server protocol that relies on two communications channels between client and server: a command channel for controlling the conversation and a data channel for transmitting file content. Clients initiate conversations with servers by requesting to download a file. | The FTP (File Transfer Protocol) utility program is commonly used for copying files to and from other computers. These computers may be at the same site or at different sites thousands of miles apart. FTP is a general protocol that works on UNIX systems as well as a variety of other (non-UNIX) systems. | To transfer files via File Transfer Protocol (FTP), you need to establish an FTP connection. To make an FTP connection you can use a standard Web browser (Internet Explorer, Mozilla Firefox, etc.) or an FTP Client. To transfer a file with FTP you need to have an FTP accounts for the web space you are going to transfer the file to. FTP hosting account where you plan to upload your files. | The Difference Between FTP Servers and File Servers. When two terms are similar and even describe a similar concept, people have a tendency to start using them interchangeably. This is definitely true in the case of FTP servers and file servers, which sound like they accomplish the same goal but in reality are two very different animals altogether. | The command-line secure file transfer program (sftp) and graphical SFTP clients, such as WinSCP and Fetch, use SSH2 encryption to authenticate and establish secure channels between networked hosts. | [1.0, 5.810799598693848, 7.961757183074951, 18.629709243774414] | | what causes a t wave abnormality | T –wave abnormalities may not necessarily indicate the presence of a severe heart condition. There are non-specific wave changes that result from common, non-specific causes of T-wave abnormality which includes the following: 1 No obvious causes, which are usually associated with women. Fever. | 5 Causes Of T-Wave Abnormality. T wave is basically the diagrammatically representation of ventricular polarization called electrocardiography. The structure of a T wave is like slight inverted upward stroke that follow a peak generated by R and S waves. One heart beat is represented in form of Q, R, S and T wave. The abnormality in T waves may be indicated by longer, flatter or higher peaks in the diagram. The measurement of heart beat in such a way makes it possible to diagnose heart related problems easily. If you are suffering from any heart related disease, the electrocardiogram will show an abnormality in the T wave. In this write up we will discuss various causes of abnormality in T wave measurement. | Specific states or conditions that cause T-wave abnormality. Complete inversions can signify the presence of cardiovascular diseases and other serious complications, which include the following: Ischemia is a condition in which oxygenated blood becomes constrained in a certain body part. | T Wave Abnormalities. christine m. smith. I just received a copy of an ECG I had done in Sept 1998. The preliminary report was borderline ECG and the final interpretation was Abnormal ECG. There was normal sinus rhythm and Nonspecific Anterior T abnormalities. When compared to an ECG taken in 1986 there was minimal T wave change. I have been told by many that abnormalities like this are usually no problem. | Prolonged Q-T interval. Long QT syndrome is a heart rhythm disorder that can cause serious irregular heart rhythms (arrhythmias). In a normal heart, your heart circulates blood throughout your body during each heartbeat. Your heart's chambers contract and relax to pump blood. | [1.0, 1.0, 5.3564229011535645, 11.585516929626465] | Loss: SpladeLoss with these parameters: Training Hyperparameters Non-Default Hyperparameters - `evalstrategy`: epoch - `perdevicetrainbatchsize`: 48 - `perdeviceevalbatchsize`: 48 - `learningrate`: 4e-05 - `numtrainepochs`: 4 - `lrschedulertype`: cosine - `warmupratio`: 0.025 - `fp16`: True - `loadbestmodelatend`: True - `pushtohub`: True - `overwriteoutputdir`: False - `dopredict`: False - `evalstrategy`: epoch - `predictionlossonly`: True - `perdevicetrainbatchsize`: 48 - `perdeviceevalbatchsize`: 48 - `pergputrainbatchsize`: None - `pergpuevalbatchsize`: None - `gradientaccumulationsteps`: 1 - `evalaccumulationsteps`: None - `torchemptycachesteps`: None - `learningrate`: 4e-05 - `weightdecay`: 0.0 - `adambeta1`: 0.9 - `adambeta2`: 0.999 - `adamepsilon`: 1e-08 - `maxgradnorm`: 1.0 - `numtrainepochs`: 4 - `maxsteps`: -1 - `lrschedulertype`: cosine - `lrschedulerkwargs`: {} - `warmupratio`: 0.025 - `warmupsteps`: 0 - `loglevel`: passive - `loglevelreplica`: warning - `logoneachnode`: True - `loggingnaninffilter`: True - `savesafetensors`: True - `saveoneachnode`: False - `saveonlymodel`: False - `restorecallbackstatesfromcheckpoint`: False - `nocuda`: False - `usecpu`: False - `usempsdevice`: False - `seed`: 42 - `dataseed`: None - `jitmodeeval`: False - `useipex`: False - `bf16`: False - `fp16`: True - `fp16optlevel`: O1 - `halfprecisionbackend`: auto - `bf16fulleval`: False - `fp16fulleval`: False - `tf32`: None - `localrank`: 0 - `ddpbackend`: None - `tpunumcores`: None - `tpumetricsdebug`: False - `debug`: [] - `dataloaderdroplast`: False - `dataloadernumworkers`: 0 - `dataloaderprefetchfactor`: None - `pastindex`: -1 - `disabletqdm`: False - `removeunusedcolumns`: True - `labelnames`: None - `loadbestmodelatend`: True - `ignoredataskip`: False - `fsdp`: [] - `fsdpminnumparams`: 0 - `fsdpconfig`: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgradckpt': False} - `fsdptransformerlayerclstowrap`: None - `acceleratorconfig`: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None} - `deepspeed`: None - `labelsmoothingfactor`: 0.0 - `optim`: adamwtorchfused - `optimargs`: None - `adafactor`: False - `groupbylength`: False - `lengthcolumnname`: length - `ddpfindunusedparameters`: None - `ddpbucketcapmb`: None - `ddpbroadcastbuffers`: False - `dataloaderpinmemory`: True - `dataloaderpersistentworkers`: False - `skipmemorymetrics`: True - `uselegacypredictionloop`: False - `pushtohub`: True - `resumefromcheckpoint`: None - `hubmodelid`: None - `hubstrategy`: everysave - `hubprivaterepo`: None - `hubalwayspush`: False - `hubrevision`: None - `gradientcheckpointing`: False - `gradientcheckpointingkwargs`: None - `includeinputsformetrics`: False - `includeformetrics`: [] - `evaldoconcatbatches`: True - `fp16backend`: auto - `pushtohubmodelid`: None - `pushtohuborganization`: None - `mpparameters`: - `autofindbatchsize`: False - `fulldeterminism`: False - `torchdynamo`: None - `rayscope`: last - `ddptimeout`: 1800 - `torchcompile`: False - `torchcompilebackend`: None - `torchcompilemode`: None - `includetokenspersecond`: False - `includenuminputtokensseen`: False - `neftunenoisealpha`: None - `optimtargetmodules`: None - `batchevalmetrics`: False - `evalonstart`: False - `useligerkernel`: False - `ligerkernelconfig`: None - `evalusegatherobject`: False - `averagetokensacrossdevices`: False - `prompts`: None - `batchsampler`: batchsampler - `multidatasetbatchsampler`: proportional - `routermapping`: {} - `learningratemapping`: {} Training Logs | Epoch | Step | Training Loss | dotndcg@10 | |:-----:|:-----:|:-------------:|:-----------:| | 1.0 | 16667 | 8.363 | 0.6961 | | 2.0 | 33334 | 6.5021 | 0.7031 | | 3.0 | 50001 | 5.2209 | 0.7031 | Framework Versions - Python: 3.12.11 - Sentence Transformers: 5.1.0 - Transformers: 4.55.3 - PyTorch: 2.8.0+cu126 - Accelerate: 1.10.0 - Datasets: 4.0.0 - Tokenizers: 0.21.4

This model is an Instruction Tuned version of Llama 3.2 1B Amharic. How to use First, you need to install the latest version of transformers You can use this model directly with a pipeline for text generation: