| --- |
| license: mit |
| tags: |
| - text-generation-inference |
| - Transformer |
| - large-language-model |
| - generative AI |
| - on-device-computing |
| - edge-computing |
| --- |
| **QuicktypeGPT is an on-device C-written large language model (LLM) to assist you typing quicker and carrying out meaningful conversations.** |
|
|
| This model only has 15M parameters (dim = 288, 6 layers, 6 heads and 6 kv heads) and 27MB. The model is pre-trained on a single A40 GPU and can be inferenced through a pure C program on a laptop CPU (e.g. AMD, Intel) with decent quality and speed. This project is to demonstrate that: |
| - We do not need to train a very sophisticated LLM but can still achieve santisfactory performance if the LLM is only focused on a small and dedicated domain or task. |
| - We can deploy small LLMs on edge devices (e.g. desktop, laptop, tablet or phone) to perform inference tasks without relying on the servers in the cloud. |
|
|
| For more details, please refer to [quicktypeGPT](https://github.com/chaoluond/quicktypeGPT) github project. |
