On-Device AI & Private LLM Architecture
EdgeTal runs a private LLM entirely on-device — no API keys, no inference server, no cloud calls. This page documents how a quantised Gemma-2B model and 512-dimensional MediaPipe embeddings run locally on Android, iOS and macOS hardware, and how that makes the app work fully offline. Candidate résumés, search queries, notes and AI evaluations never leave your device.
0 Bytes Cloud Data Transmission
No third-party LLM APIs, no analytics telemetry, no remote databases. All embedding generation, vector indexing, and generative AI reasoning execute locally.
How to run a local LLM on phone and laptop hardware
Running an LLM on a phone is a hardware-delegate problem, not a model-size problem. EdgeTal binds inference directly to each platform's GPU and neural accelerator, which is what makes on-device generation viable on commodity devices — including a 2019 budget Android phone with 4 GB of RAM.
- ▪Embedding Engine: Swift
MediaPipeTasksTextCocoaPod creating 512d dense vector representations. - ▪LLM Hardware Acceleration: Metal Performance Shaders binding directly to GPU & Apple Neural Engine.
- ▪Local Security: Protected with iOS Keychain and
AppLockServiceFace ID / Touch ID vault locks.
- ▪Embedding Engine: Native Kotlin
EmbedderChannel.ktwith Google TFLite text embedder bindings. - ▪LLM Hardware Acceleration:
LlmChannel.ktovercom.google.mediapipe:tasks-genai(GPU & NNAPI delegates). - ▪Local Database: High-performance ObjectBox HNSW C++ vector engine storing local candidate embeddings.
Core Architectural Highlights
1. 512-Dimensional MediaPipe Vector Search
EdgeTal maps candidate resumes into dense 512-dimensional vector spaces. Natural language queries (such as “Python developer with high-throughput API experience”) match candidate resumes based on conceptual skill relationships, even if exact keywords vary (e.g. matching FastAPI or Django).
2. Quantized Gemma-2B IT On-Device GenAI
Google Gemma-2B IT is loaded into mobile memory using Int4 quantization. The model reads target job descriptions and candidate resumes locally, outputting structured thought reasoning, evidence quotes, and fit recommendations without paying per-token API charges.
3. Passcode Vault Lock & Encrypted Peer-to-Peer Archives
EdgeTal protects your local app with biometric Face ID / Touch ID or PIN passcode locks via AppLockService. When sharing candidate pools with recruiting teammates, EdgeTal exports password-encrypted .edgetal archives over AirDrop, email, or USB.
On-device AI & private LLM questions
Can you actually run an LLM on a phone?
Yes. EdgeTal runs Google’s Gemma-2B instruction-tuned model quantised to Int4, bound directly to the device GPU and neural accelerator rather than the CPU. In peer-reviewed benchmarks it ran on both a Pixel 7 Pro (Tensor G2) and a Redmi Note 7 — a 2019 budget phone with 4GB of RAM — with no cloud inference at any point.
What is on-device AI?
On-device AI means the model weights live on your hardware and inference runs locally, so no input is transmitted to a remote server. The practical consequence is that there are no per-token API fees, no network latency, no vendor retention of your data, and the feature keeps working with no connection.
Which model does EdgeTal run locally?
Two. A MediaPipe Text Embedder produces 512-dimensional L2-normalised vectors for semantic retrieval, and a quantised Google Gemma-2B IT model performs the generative fit analysis. On iOS and macOS these bind to Metal Performance Shaders and the Apple Neural Engine; on Android they use GPU and NNAPI delegates via com.google.mediapipe:tasks-genai.
Does EdgeTal work offline?
Yes, completely. After a one-time model download the app needs no network connection. Résumé parsing, embedding generation, vector search and generative analysis all execute locally, which is why it works on a plane, in a secure facility, or anywhere with no reliable connectivity.
Is a local LLM accurate enough compared to a cloud model?
For this task, yes. In the EICON 2026 evaluation the on-device agentic pipeline reached Precision@10 of 0.47 overall versus 0.21 for keyword matching. On conceptual queries the gap is starker: keyword search scored 0.00 while the on-device pipeline recovered 0.48. Twenty generated candidate reports were rated 4.00/5 for accuracy by independent human raters, with Cohen’s Kappa of 0.833.
What is the performance cost of running locally?
Vector search is effectively hardware-agnostic at 18ms on a Pixel 7 Pro and 17ms on a Redmi Note 7. Semantic retrieval completes in under 200ms on both devices. Generative analysis is where hardware matters — the agentic step showed a 7.2× gap between the two devices, so LLM inference, not vector search, is the scalability bottleneck.
Benchmark figures are from EdgeTal's peer-reviewed EICON 2026 poster (paper FPC53), awarded Best Poster Presenter in the Computer Science, Artificial Intelligence and Information Systems track.