正文
- Tool calls
- Given the functions your app exposes, Needle picks the right ones and fills every argument from what the user said. Ask for two things and you get two calls in order; ask for something no tool covers and you get an empty list, not a guess.
- Structured extraction
- Declare a shape, hand over messy text, get typed fields back: an invoice, a booking, a notification, a form. The decode grammar guarantees the output parses. Extraction generalised well to classification problems too.
- Text embedding
- The same model returns a vector for a sentence, so an app can search, match and route locally: find the note you mean, pick the tool closest to a request, collapse duplicate alerts.
What that looks like in a product:
- Smart home
- Go from pressing buttons to talking to the house. "Dim the bedroom and lock up" becomes two calls, executed offline, with no hub round trip.
- Robots
- Give a vacuum or a small robot nuanced instructions: "clean the kitchen but leave the bedroom", "go back to the dock when you are done". Each becomes a sequence of moves it can execute.
- Phones
- An assistant that acts on the device instead of answering: make an album from last weekend's photos, open a site, dim the screen, find the lease in your files.
- Wearables
- Read a notification into structured data on the wrist: a card charge into merchant, amount and date; a message into a reply; a complaint into a sentiment flag.
- AR glasses
- Navigation and nearby search from a short request, with no phone or network in the loop.
- Automotive
- Climate, media, navigation and calls from requests in the cabin, with the tool set pinned so it survives a long drive's worth of conversation.
- Computers
- Plain-English control of the machine in front of you: draft the mail, start the timer, copy the address, open the tab.
- Search and matching
- Embeddings that never leave the device: semantic search over notes, messages and documents; a query matched to the closest of hundreds of tools; near-duplicate alerts merged on a watch.
Model
Intelligence laddering. Every layer of Needle 3 is a sub-network with monotonically increasing capacity. Developers can choose the right size from the 2-layer (2L) subnetwork to 20 layers (20L). Each subnetwork is amenable to fine-tuning, such that 4L can match DeepSeek V4 Flash when tuned on downstream tasks for one epoch. Intelligence laddering produces 9 to 29 MB CQ2-bit binaries and supports a wide range of tiny devices.
- Inputs
- Text prompts, plus tool definitions or an extraction schema
- Outputs
- Structured JSON with tool calls or extractions
- Model
- 29-121M Laddered Simple Attention Networks, CQ2 quantisation
- Training
- 360B tokens of proprietary structured dataset
- Speed
- 400-4k tokens/s decode and 1-10k tokens/s prefill on a Raspberry Pi 5
Laddered Simple Attention Networks
Get started
Install the Python package. The inference engine is fetched once from Hugging Face and cached; there is nothing else to build.
Needle reads your tool descriptions to decide what to call and how to fill arguments, so describing them well is the whole game.
Simple: decorate a function. The signature gives the argument types, the docstring is the tool description, and run() completes the loop: the model picks the call, Needle executes your function, feeds the result back, and returns the final response with the executed tool results attached as results.
import needle @needle.tool def get_weather(city: str): "Get the current weather for a city." return {"city": city, "temp_c": 27, "sky": "clear"} agent = needle.Needle(tools=[get_weather]) print(agent.run("what's it like in Lagos right now?")["results"]) # [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
Route by pattern: when a description cannot enumerate every phrasing, give a tool triggers, regular expressions matched against each request. A match restricts the decode to the matched tools and requires a call, so the request reaches the tool you named instead of being refused or misrouted, and the call ships even below the confidence floor. A match restricts the whole turn, so a catch-all should exclude the nouns other tools own, e.g. ^(?![\s\S]*\b(lights?|doors?)\b)[\s\S]*\b(turn|switch)\b[\s\S]*\b(on|off)\b; then "switch the fan on and dim the kitchen lights" still reaches both tools.
from typing import Literal @needle.tool(triggers=[r"\b(turn|switch|power|flip)\b.*\b(on|off)\b", r"\btoggle\b"]) def control_device(device: str, action: Literal["on", "off", "toggle"]): "Switch or toggle any named smart-home device." return {"device": device, "action": action} agent = needle.Needle(tools=[control_device, get_weather]) agent.complete("toggle the garage door") # function_calls [{"name": "control_device", "arguments": {"device": "garage door", "action": "toggle"}}]
Extraction: to pull structured data out of text, declare the shape and call extract(). Pass a Pydantic model and you get a typed object back.
from pydantic import BaseModel class Invoice(BaseModel): vendor: str total: float due_date: str invoice = needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice) print(invoice.vendor, invoice.total) # -> Acme Corp 1200.0
Every turn returns one JSON object:
{
"type": "call",
"success": true,
"error": null,
"error_code": null,
"function_calls": [ { "name": "set_lights", "arguments": { "room": "living room", "on": true, "brightness": 30 } } ],
"reasoning": "'living room' -> room; 'dim' -> on true, brightness 30",
"confidence": 0.94,
"prefill_tps": 4300.0,
"decode_tps": 850.0,
"peak_ram_mb": 28.5
}Confidence gating and routing: every response carries a confidence score from a calibrated head, and the engine already applies a floor of 0.1. Below it, the call is withheld into suppressed_calls and function_calls is empty. Above it, the score is yours to route on: act at once when it is high, show the call and ask when it is middling, and treat an empty result as a refusal. A tool with triggers always produces a call for a matching request, so the score is what tells you whether to run it or confirm it.
r = agent.complete(user_text) calls = r["function_calls"] held = r["suppressed_calls"] if calls and r["confidence"] >= 0.7: execute(calls) # sure: act elif calls or held: confirm(calls or held, r["reasoning"]) # unsure: show the call, ask else: say("I can't do that here") # nothing to do: refuse
Writing tools: the model reads a schema literally, so a narrow tool with a plain description beats a broad one. One tool per action, described by the actions it covers ("Turn a room's lights on or off") rather than a category. Name enum options after what a user says (action: ["increase", "decrease"]) and keep synonyms in the description. Give a required argument a default when a request may leave it out; a required argument with no default and no evidence in the request is withheld rather than guessed. Put value formats in descriptions ("City, ST", "e.g. T-1042"). Add triggers to intents that must always reach a tool, and keep the toolset per turn small, since every extra tool is a chance to misroute.
Fine-tune: the Python package is the quick path. LoRA on the frozen base at the full 20 layers, then a 4-bit .cact of any subnetwork that runs on the same engine.
needle finetune data.jsonl --epochs 10 --out adapter.safetensors needle build --lora adapter.safetensors --out tuned.cact needle build --lora adapter.safetensors --platform linux-arm64 --layers 2 --out ./device
The guides go deeper: designing tools, confidence, extraction, fine-tuning, the Python reference, supported devices, the .cact format and porting Needle. Source is on GitHub.
Cactus Platform
Needle was designed to be customised. Its capacity is a ladder, and a subnetwork as small as 2 layers, fine-tuned on one product's tools, runs optimally on devices far smaller than the full model needs. Constraining the capacity to a narrow, well-defined task is what lets it reach frontier-level accuracy there: fine-tuning on DroidCall lifts every subnetwork by 18 to 36 points, and from 4 layers up the tuned subnetwork passes DeepSeek V4 Flash, starting at 29M parameters (Figure 3).
Every subnetwork, fine-tuned on the platform
The Cactus Platform is the full path: Cactus datasets, the 2-bit quantisation behind the shipped model, evaluation design and tracking, full-depth fine-tunes and dataset management, all on our infrastructure and training pipeline, no need to build your own.
Deploy
Every deployment target ships a prebuilt engine under 1 MB that loads the needle3.cact weights at start. needle build fetches the engine for a platform and puts the weights beside it, at the full 20 layers or any smaller subnetwork:
| Target | Platform folder | Ships |
|---|---|---|
| macOS | macos-arm64 | needle CLI, libneedle.a, needle.h |
| Linux | linux-x86_64, linux-arm64, linux-armv7, linux-riscv64, linux-mipsel | needle CLI, libneedle.a, needle.h |
| Windows | windows-x86_64, windows-arm64 | needle.exe, libneedle.a, needle.h |
| Android | android-arm64, android-armv7, android-riscv64 | needle CLI, libneedle.a, needle.h |
| iOS | ios-arm64, ios-sim-arm64 | libneedle.a, needle.h |
| tvOS, watchOS | tvos-arm64, watchos-arm64 | libneedle.a, needle.h |
| Browser | wasm | needle.js, needle.wasm, needle.h |
| WASI component | wasm-component | needle.component.wasm, needle.wit |
needle build --platform downloads it and places needle3.cact beside the engine.# engine, header and weights for this Mac needle build --platform macos-arm64 # an 8-layer subnetwork for a Pi needle build --platform linux-arm64 --layers 8 --out ./pi # a tuned archive needle build --lora adapter.safetensors --out tuned.cact