Whileone Tech Space

We changed one setting, the floating-point format a detection model runs in; on a service already deployed to a Jetson board, and cut its latency in half. Here's the full accounting: what FP16 actually costs, what it doesn't, and exactly where the savings run out. The service in question detects a set of product-condition classes on a live line, using a YOLOv8-style model exported to ONNX and served behind a FastAPI endpoint. It runs on an NVIDIA Jetson (JetPack 6.2, Ampere-c

Introduction For this project, I have used llama.cpp as the local inference engine and TinyLlama-1.1B-Chat-v1.0 as the language model. llama.cpp is a lightweight C/C++ inference framework designed to run LLMs locally with minimal setup across CPUs and GPUs. It is well suited for embedded and edge-oriented workflows because it supports efficient local execution without depending on cloud APIs. The TinyLlama model used here is the chat-tuned 1.1B parameter variant published on

Inspiration In high-throughput production environments, every sensor reading tells a story. Test systems continuously record Pressure, Temperature, and Differential Loss (DL) across thousands of cycles, but much of this data remains passive, observed but not interpreted. We set out to change that by deploying machine learning directly at the edge on a BeagleBone Black board. The goal was not anomaly detection, but live inference: to compute what the ideal DL should be (DL_pre