KBLab

--- language: sv datasets: - common_voice - KTH/nst metrics: - wer - cer tags: - audio - automatic-speech-recognition - speech - xlsr-fine-tuning-week license: apache-2.0 model-index: - name: XLSR Wav2Vec2 Swedish by KBLab results: - task: name: Speech Recognition type: automatic-speech-recognition dataset: name: Common Voice sv-SE type: common_voice args: sv-SE metrics: - name: Test WER type: wer value: 14.298610 - name: Test CER type: cer value: 4.925294 ---

Finetuned version of KBs VoxRex large model using Swedish radio broadcasts, NST and Common Voice data. Evalutation without a language model gives the following: WER for NST + Common Voice test set (2% of total sentences) is 2.5%. WER for Common Voice test set is 8.49% directly and 7.37% with a 4-gram language model. When using this model, make sure that your speech input is sampled at 16kHz. Chart shows performance without the additional 20k steps of Common Voice fine-tuning Training This model has been fine-tuned for 120000 updates on NST + CommonVoice and then for an additional 20000 updates on CommonVoice only. The additional fine-tuning on CommonVoice hurts performance on the NST+CommonVoice test set somewhat and, unsurprisingly, improves it on the CommonVoice test set. It seems to perform generally better though [citation needed] . Usage The model can be used directly (without a language model) as follows:

This is a sentence-transformers model: It maps Swedish sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. This model is a bilingual Swedish-English model trained according to instructions in the paper Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation and the documentation accompanying its companion python package. We have used the strongest available pretrained English Bi-Encoder (all-mpnet-base-v2) as a teacher model, and the pretrained Swedish KB-BERT as the student model. A more detailed description of the model can be found in an article we published on the KBLab blog here and for the updated model here. Update: We have released updated versions of the model since the initial release. The original model described in the blog post is v1.0. The current version is v2.0. The newer versions are trained on longer paragraphs, and have a longer max sequence length. v2.0 is trained with a stronger teacher model and is the current default. | Model version | Teacher Model | Max Sequence Length | |---------------|---------|----------| | v1.0 | paraphrase-mpnet-base-v2 | 256 | | v1.1 | paraphrase-mpnet-base-v2 | 384 | | v2.0 | all-mpnet-base-v2 | 384 | Using this model becomes easy when you have sentence-transformers installed: Loading an older model version (Sentence-Transformers) Currently, the easiest way to load an older model version is to clone the model repository and load it from disk. For example, to clone the v1.0 model: Then you can load the model by pointing to the local folder where you cloned the model: Usage (HuggingFace Transformers) Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings. To load an older model specify the version tag with the `revision` arg. For example, to load the v1.0 model, use the following code: The model was evaluated on SweParaphrase v1.0 and SweParaphrase v2.0. This test set is part of SuperLim -- a Swedish evaluation suite for natural langage understanding tasks. We calculated Pearson and Spearman correlation between predicted model similarity scores and the human similarity score labels. Results from SweParaphrase v1.0 are displayed below. | Model version | Pearson | Spearman | |---------------|---------|----------| | v1.0 | 0.9183 | 0.9114 | | v1.1 | 0.9183 | 0.9114 | | v2.0 | 0.9283 | 0.9130 | The following code snippet can be used to reproduce the above results: In general, v1.1 correlates the most with human assessment of text similarity on SweParaphrase v2.0. Below, we present zero-shot evaluation results on all data splits. They display the model's performance out of the box, without any fine-tuning. | Model version | Data split | Pearson | Spearman | |---------------|------------|------------|------------| | v1.0 | train | 0.8355 | 0.8256 | | v1.1 | train | 0.8383 | 0.8302 | | v2.0 | train | 0.8209 | 0.8059 | | v1.0 | dev | 0.8682 | 0.8774 | | v1.1 | dev | 0.8739 | 0.8833 | | v2.0 | dev | 0.8638 | 0.8668 | | v1.0 | test | 0.8356 | 0.8476 | | v1.1 | test | 0.8393 | 0.8550 | | v2.0 | test | 0.8232 | 0.8213 | When it comes to retrieval tasks, v2.0 performs the best by quite a substantial margin. It is better at matching the correct answer to a question compared to v1.1 and v1.0. | Model version | Data split | Accuracy | |---------------|------------|------------| | v1.0 | train | 0.5262 | | v1.1 | train | 0.6236 | | v2.0 | train | 0.7106 | | v1.0 | dev | 0.4636 | | v1.1 | dev | 0.5818 | | v2.0 | dev | 0.6727 | | v1.0 | test | 0.4495 | | v1.1 | test | 0.5229 | | v2.0 | test | 0.5871 | Examples how to evaluate the models on some of the test sets of the SuperLim suites can be found on the following links: evaluatefaq.py (Swedish FAQ), evaluateswesat.py (SweSAT synonyms), evaluatesupersim.py (SuperSim). An article with more details on data and v1.0 of the model can be found on the KBLab blog. Around 14.6 million sentences from English-Swedish parallel corpuses were used to train the model. Data was sourced from the Open Parallel Corpus (OPUS) and downloaded via the python package opustools. Datasets used were: JW300, Europarl, DGT-TM, EMEA, ELITR-ECA, TED2020, Tatoeba and OpenSubtitles. `torch.utils.data.dataloader.DataLoader` of length 180513 with parameters: This model was trained by KBLab, a data lab at the National Library of Sweden. You can cite the article on our blog: https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/ . We gratefully acknowledge the HPC RIVR consortium (www.hpc-rivr.si) and EuroHPC JU (eurohpc-ju.europa.eu/) for funding this research by providing computing resources of the HPC system Vega at the Institute of Information Science (www.izum.si).

The National Library of Sweden/KBLab releases a robust, multi-label sentiment classifier finetuned on Megatron-BERT-large-165K. The model was trained on approximately 75K Swedish texts from multiple linguistic domains and datasets. There is a post on the KBLab blog describing the model in further detail.