litert-community

This model provides a few variants of deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B that are ready for deployment on Android using the LiteRT (fka TFLite) stack, MediaPipe LLM Inference API and LiteRt-LM. Disclaimer: The target deployment surface for the LiteRT models is Android/iOS/Web and the stack has been optimized for performance on these targets. Trying out the system in Colab is an easier way to familiarize yourself with the LiteRT stack, with the caveat that the performance (memory and latency) on Colab could be much worse than on a local device. [](https://colab.research.google.com/#fileId=https://huggingface.co/litert-community/DeepSeek-R1-Distill-Qwen-1.5B/blob/main/notebook.ipynb) Download and install the apk. Follow the instructions in the app. To build the demo app from source, please follow the instructions from the GitHub repository. Note that all benchmark stats are from a Samsung S24 Ultra with 1280 KV cache size with multiple prefill signatures enabled. Backend Quantization Context Length Prefill (tokens/sec) Decode (tokens/sec) Time-to-first-token (sec) Model size (MB) Peak RSS Memory (MB) GPU Memory (MB) CPU dynamicint8 4096 166.50 tk/s 26.35 tk/s 6.41 s 1831.43 MB 2221 MB N/A GPU dynamicint8 4096 927.54 tk/s 26.98 tk/s 5.46 s 1831.43 MB 2096 MB 1659 MB Model Size: measured by the size of the .tflite flatbuffer (serialization format for LiteRT models) Memory: indicator of peak RAM usage The inference on CPU is accelerated via the LiteRT XNNPACK delegate with 4 threads Benchmark is done assuming XNNPACK cache is enabled Benchmark is run with cache enabled and initialized. During the first run, the time to first token may differ. dynamicint8: quantized model with int8 weights and float activations.

This model provides HuggingFaceTB/SmolVLM-256M-Instruct model in TFLite format. You can use this model with Custom Cpp Pipiline or run with python pipeline (see COLAB example below). Please note that, at the moment, AI Edge Torch VLMS not supported on MediaPipe LLM Inference API, for example qwenvl model, that was used as reference to write SmolVLM-256M-Instruct convertation scripts. [](https://colab.research.google.com/#fileId=https://huggingface.co/litert-community/SmolVLM-256M-Instruct/blob/main/smalvlmnotebook.ipynb ) To fine-tune SmolVLM on a specific task, you can follow the fine-tuning tutorial. Than, you can convert model to TFlite using custom smalvlm scripts (see Readme.md). You can also check the official documentation ai-edge-torch generative. The model was converted with the following parameters: