Whileone Tech Space

Introduction For this project, I have used llama.cpp as the local inference engine and TinyLlama-1.1B-Chat-v1.0 as the language model. llama.cpp is a lightweight C/C++ inference framework designed to run LLMs locally with minimal setup across CPUs and GPUs. It is well suited for embedded and edge-oriented workflows because it supports efficient local execution without depending on cloud APIs. The TinyLlama model used here is the chat-tuned 1.1B parameter variant published on

LLMs are Getting Better and Smaller Let’s look at Llama as an example. The rapid evolution of these models highlights a key trend in AI: prioritizing efficiency and performance. When Llama 2 70B launched in August 2023, it was considered a top-tier foundational model. However, its massive size demanded powerful hardware like the NVIDIA H100 accelerator. Less than nine months later, Meta introduced Llama 3 8B, shrinking the model by almost 9x. This enabled it to run on smaller