LLMEasyQuant -- An Easy to Use Toolkit for LLM Quantization
Abstract
Currently, there are many quantization methods appeared for LLM quantization, yet few are user-friendly and easy to be deployed locally. Packages like TensorRT and Quantohave many underlying structures and self-invoking internal functions, which are not conducive to developers' personalized development and learning for deployment. Therefore, we develop LLMEasyQuant, it is a package aiming to for easy quantization deployment which is user-friendly and suitable for beginners' learning.
- Publication:
-
arXiv e-prints
- Pub Date:
- June 2024
- DOI:
- 10.48550/arXiv.2406.19657
- arXiv:
- arXiv:2406.19657
- Bibcode:
- 2024arXiv240619657L
- Keywords:
-
- Computer Science - Machine Learning