Rendered at 23:24:16 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
nater5000 1 days ago [-]
This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack.
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).
derefr 3 hours ago [-]
> Like, what does a 28MB binary get you?
For one thing, on beefy-enough recent CPUs, you could keep the weights hot in the L2 cache of a single CPU core. (Which is clearly not the use-case, but might be interesting to those looking for extreme TPS numbers. Or perhaps for efficient training!)
HenryNdubuaku 1 days ago [-]
So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.
anon373839 18 hours ago [-]
It seems to me that the model struggles to have enough general intelligence, knowledge, or reasoning capacity for arbitrary prompted tool calling. At this size, not surprising.
I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.
londons_explore 10 hours ago [-]
It also seems to have far more tokens per second than needed for general "close the blinds" "tool_call(blinds, CLOSED)".
I do wonder if more smartness could be had by using sparser experts.... And possibly even having some kind of expert switching penalty to try to reduce the amount of data read from read only flash memory by encouraging subsequent tokens to use already loaded experts.
electroglyph 13 hours ago [-]
i would assume a model this size would require finetuning tbh. even functiongemma recommends that.
fwipsy 23 hours ago [-]
14mb? More like sub-$20 devices.
TomatoCo 23 hours ago [-]
Most pi pico's come with 16mb of flash. I wonder what kind of performance that can eek out.
SequoiaHope 21 hours ago [-]
Well running from QSPI flash (even the internal memory versions use SPI internally) so any inference would be very slow streaming from that compared to RAM. The featured article says: “With a peak session RAM around 28MB, Needle runs on newer microcontrollers like ESP32-S3.” So I don’t see this doing anything useful on a Pico. The Pico 2 (RP2350) for example has 520k of RAM.
Rohansi 20 hours ago [-]
An ESP32 has the same amount of SRAM as the Pi Pico. You can hook up PSRAM to the Pi Pico just like ESP32 to get 16MB more RAM.
silentbob7 18 hours ago [-]
I'm quite impressed by the results of the web demo, especially given its size and the precision with which it uses the three available tools (tested with German commands). I could imagine that this LLM would fit well into a setup with multiple micro-sized LLMs for different purposes; so 14 MB for precise tool invocation is a reasonable memory footprint when a number of other local models are running (e.g. STT, TTS + language models).
HenryNdubuaku 14 hours ago [-]
yes, that's what we had in mind while building
ProtoAES256 12 hours ago [-]
It's called a LM :). LLM stands for Large Language Model.
watwut 11 hours ago [-]
LLM as in Little Language Model
HenryNdubuaku 10 hours ago [-]
Ok we'd be adopting Little Language Model officially haha
SamPatt 3 hours ago [-]
lLM
jhgik798 11 hours ago [-]
TLM
11 hours ago [-]
CarpeNecopinus 16 hours ago [-]
It's definitely cool that you can get any reasoning whatsoever out of such a small model. That said, its reasoning is "interesting":
Query: "Make the living room dark"
Agent: "User wants lights on in living room. 'dark' implies dim. Room 'living room', action 'on'." (And on every test I did, it just completely ignored the "brightness" parameter)
It also appears to have no concept of what a door or light actually is, whenever the query diverges from "Lock door X" or "Turn on light X", it tries to shoehorn whatever additional context is given into the device name:
Query: "Lock out the vacuum salesman at the front door"
Agent tries to lock "front door vacuum salesman"
"The way you talk really makes me appreciate silence" is classified as "positive" with 82% confidence.
HenryNdubuaku 14 hours ago [-]
Ok, this is genuinely funny, we will fix these as we iterate, thanks lol.
AmazingTurtle 15 hours ago [-]
that model is 14MB large what do you expect. but I agree it's funny regardless
dbeardsl 23 hours ago [-]
My first query:
> Make it a little warmer in here.
The reply:
> "name": "set_thermostat",
> "arguments": {
> "temperature": 65,
> "mode": "cool",
> ...
> "reasoning": "'warmer' implies need for cooling; set_thermostat with temperature 65 (typical warmth) and mode 'cool'.",
Maybe I'm doing it wrong?
ocihangir 17 hours ago [-]
Tested your example, the confidence is 0. In smart home context, I can think of an application where the low confidence answers can be forwarded to cloud, whereas the vast majority generic queries solved locally, if the confidence is reliable enough. The response is quite fast by the way.
ComputerGuru 9 hours ago [-]
But this isn’t a query that should need to be forwarded to the cloud for acting on!
goodmythical 4 hours ago [-]
I don't know, the back of my napkin says you'll need consensus of at least three servers in order to complete the command.
dannyw 22 hours ago [-]
It's not a conversational model. It's meant as a local tool calling model.
derangedHorse 22 hours ago [-]
Yes, I think OP understands that. What he and many others in this thread are trying to understand is what makes this model useful.
ehnto 21 hours ago [-]
Turning a voice command into a tool call should be self evidently useful, being able to do that on a small embedded device is the novelty here. In this theoretical example, the thermostat is hosting the model on device and would use no external services.
tomrod 21 hours ago [-]
I confused by the dispatch model. Tool calls typically need some reasonability to be deterministic and, more importantly, predictable in response (o/w GIGO). Why would the thermostat need to interpret a voice command that the node capturing the voice command would not interpret instead?
ehnto 19 hours ago [-]
The node capturing the voice command could be the thermostat. From my understanding they are targeting very small devices.
So that could be a master home automation node, but why not also a single purpose device? I can think of more bad examples than I can good ones, but maybe I am doing some soldering and I need my soldering iron turned up a bit; my hands are full, so doing that by voice would be useful enough. Something I would never link up to a big AI model or home automation network, but could be useful to control by voice.
If it's something that can be burnt directly into a chip and shipped with the products for cheap, maybe that's a more pragmatic way to get AI into small devices (see taalas for a much bigger model doing that, althoug not yet cheap).
tomrod 19 hours ago [-]
Oh I agree that's not unreasonable. I wonder about the harnessing heft required to make it feasible though. If I say I'd like it a bit warmer, a tool can deterministically bump a few degrees while an LLM might bump it 10C. So do we limit the tool's range?
weird-eye-issue 15 hours ago [-]
That's exactly what they were testing and what it failed at
22 hours ago [-]
KPGv2 13 hours ago [-]
Yes, and it seems the tool called here was the thermostat, to set the mode literally the opposite of what was requested.
"Make it warmer"
tool that cools room called
victor_edka 8 hours ago [-]
but wouldn't it need to know the current temp for that to work? my understanding was that the model is best at tool calling
owebmaster 20 hours ago [-]
Try asking it to set the thermostat to a value. It's a very small model, there's not much reasoning capability
Tiberium 1 days ago [-]
Funny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so:
Query: HN
Result:
{
"function_calls": [
{
"name": "lock_door",
"arguments": {
"door": "front door"
}
}
],
"reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.",
"confidence": 0
}
I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.
yoavm 1 days ago [-]
The website says the model is for "tool calling, device use, and structured extraction". Your example just doesn't seem to be very relevant. FWIW, it did a pretty good job for tool calling when I tried it, and I think it could be pretty nice to have this running on locally and integrate with Home Assistant.
evmaki 1 days ago [-]
False positives are definitely relevant and worth measuring - natural language interfaces always have a discoverability problem, i.e., users not knowing what actions the system does and does not support. If the frontend of that system lacks the ability to reject unsupported commands, weird stuff happens.
Nonetheless, this is very cool work! If I can offer a small suggestion to the team at Cactus, it would be to evaluate your releases on some usability criteria (including false positives). Any serious integrator or adopter of these models would want to have that information available.
jdknezek 1 days ago [-]
> "confidence": 0
OP and the linked page talk about the confidence score and using it as an action threshold, so it looks like an appropriate total response to me.
evmaki 1 days ago [-]
Right, but that's not the same thing as reporting a benchmark across a test set. It doesn't help me determine how well the model does across a decently-large sample size of commands. It doesn't tell me with what reliability the confidence will be below a given threshold when it should be, above that threshold when it should be, etc.
HenryNdubuaku 1 days ago [-]
Thanks, noted!
planb 17 hours ago [-]
This is extremely impressive if it works. But on the other hand, if the number of cases where it works as expected is lower than what we could catch with a (old-Siri-style) heuristics based approach, and the rest fails in unpredictable ways, I'd prefer the dumb old "I did not understand that" response.
curious_cat_163 22 hours ago [-]
I think the test above is about tool calling... That's how I read it.
The issue here is known as "out of distribution detection" in the old-timey classification world.
I am not sure how a micro model will fundamentally solve it. Would love to understand what dannyw and team did there?
derangedHorse 22 hours ago [-]
How did you draw an association between dannyw and Cactus? There are no 'Danny's on the list of GH contributors nor is there one named in the paper. Just curious.
curious_cat_163 12 hours ago [-]
Ah, my bad. I may have misread it.
petu 1 days ago [-]
"confidence": 0, so I guess you could threshold it
justoneletter 19 hours ago [-]
Only if the confidences are calibrated, and they probably aren't. Any idea how the confidence is derived?
jszymborski 1 days ago [-]
no, this is the appropriate response to hearing the words "HN" :P
kristjan 15 hours ago [-]
It seems to love locking doors. I was hoping this would dim the lights.
> I'm hungover
{
"function_calls": [
{
"name": "lock_door",
"arguments": {
"door": "front door"
}
}
],
"reasoning": "User wants to lock the door. 'hungover' implies a security door. No specific door named, so use 'front door' as default.",
"confidence": 0
}
dvratil 11 hours ago [-]
Cannot unlock, though:
Prompt: "Unlock my doors"
{
"function_calls": [],
"reasoning": "No tool for unlocking or locking door exists. lock_door only performs lock/unlock actions, not unlock.",
"confidence": 0.9492
}
plingbang 1 days ago [-]
I've got an identical output with the prompt "do not lock the door".
hmokiguess 1 days ago [-]
yeah I got the same, almost like its biased heavily towards that as the 0 ranking -- my prompt was just the word 'potato'
Schiendelman 1 days ago [-]
Was that the first message you sent it?
HenryNdubuaku 1 days ago [-]
This is exactly why the confidence feature was introduced, the model knows when its wrong, we could hide that part and return a placeholder "sorry I only do function calls", would that be better or you prefer to see everything?
nc55g3g 22 hours ago [-]
[dead]
wky 8 hours ago [-]
Specifying units seems to be unreliable; I tried adding a description to the set_thermostat temperature:
Trying "in degrees Fahrenheit" for the tool description had similarly counterintuitive confidences.
Edit: to be clear, the counterintuitive behavior is that the confidence ended up higher for the wrong units.
ComputerGuru 9 hours ago [-]
Can you share more about the architectural/design tradeoffs you considered or decided upon? Particularly for me, why is a model that is intended mainly to just make tool calls and marshal the results back focusing on speed? Speed as an inherent result of small size, I get, but speed as a design focus confuses me because it’s simply not going to be dealing with large outputs as a rule, wouldn’t it be better to trade some of that raw speed for better smarts?
For example, I mocked a dumbed down version of what would be a reasonable intermediate tool call prompt:
> It's currently 58 degrees. User asks for house to be 8.5 degrees warmer. What temperature to set thermostat to?
The reply?
Reasoning: “User asks for temperature to set thermostat to 8.5 -> set_thermostat with temperature=8.5.”
Sounds like something Siri would do!
israrkhan 3 hours ago [-]
I tried using it on my x86_64 desktop. Could not use it.
Then I tried using it on a raspberry pi 4. Did not work.
This is specifically for 64 bit arm. I see no reason why you could not enable desktops as well? Just a matter of compilation
7 hours ago [-]
arthuqa 1 days ago [-]
That's really cool - I was already thinking of compressing `functiongemma-270m-it` down to 1-2 bits so it would work flawlessly in the browser.
Your `Fine-tuning` feature is even much more convenient.
HenryNdubuaku 1 days ago [-]
Thanks, give it a splin!
bottlero_cket 6 hours ago [-]
When I asked it to call my cable company I got this response. I am wondering how this response comes out… why does it say lights twice and not mention door? LLMs are a mystery to me but I suppose this reasoning is at the heart of the model and it’s only 40% confident in it
{
"function_calls": [],
"reasoning": "No tool for making phone calls or initiating a call. The available tools are limited to smart home lights, thermostat, and lights.",
"confidence": 0.4096
}
profsummergig 1 days ago [-]
Could someone please share how such open source micro-LLMs might have been created?
Do the creators take something like DeepSeek, and then delete most of the neurons to whittle down the size?
anigbrowl 22 hours ago [-]
There's a Manning book on creating your own LLM from scratch which answers your question exactly. There's another book from the same publisher specifically about small language models for specialty purposes.
HenryNdubuaku 1 days ago [-]
Technically, you could do that, but we trained this one from the ground up!
profsummergig 24 hours ago [-]
That sounds like an enormously expensive exercise.
salamo 23 hours ago [-]
As someone who's done something similar (https://blog.lukesalamone.com/posts/creating-tiny-semantic-s...) the expensive part wasn't the training itself but the data curation and evaluation post-training. For this, getting a reasonable distribution of tool calls when the tool call can be anything isn't easy.
Once you have that, the model is small enough batch sizes are probably enormous and training can probably be done on a consumer-grade GPU in a week or less. Or even faster on a bigger GPU.
ronsor 23 hours ago [-]
At <50M parameters, training costs are completely trivial. You'll spend a lot more on your rent this month.
HenryNdubuaku 13 hours ago [-]
Haha, my rent is cheap lol
kadoban 22 hours ago [-]
Training scales pretty badly, so smaller models like this are really not that bad in terms of cost.
ComputerGuru 9 hours ago [-]
At this size, it certainly doesn’t have to be.
genxy 22 hours ago [-]
You can train a model of this size on your laptop in a day.
profsummergig 6 hours ago [-]
So is the challenge, now, getting the right sort of data?
hgoel 24 hours ago [-]
Another option for something this small and narrowly specialized could be to get traditional LLMs to synthesize the training data. Model collapse is probably less of an issue at this size relative to terabyte sized models.
HenryNdubuaku 14 hours ago [-]
Your thinking is correct haha
a022311 11 hours ago [-]
Looks incredibly cool, although it ran at ~4 tok/s on my browser (I assume because I've disabled V8). I believe your numbers though. I always wanted to try using something like this to create an AI assistant for my phone. If you try to create something without AI, you'll quickly run into the problem of using NLP to map input to the equivalent of AI tool calls. On the other hand, shelling out to a full-blown LLM for every input is slow, costly and if you want it local, greatly restricts its computing power and the number of devices it can run on. Needle and similar models solve this problem nicely: they're incredibly cheap and handle NLP (not in a predictable way of course, but with great success rates), so you can quickly run predefined actions or shell out to a proper LLM for more complicated tasks. You have the best of two worlds.
You may have inspired me to give it a try!
HenryNdubuaku 10 hours ago [-]
Please give it a try and let us know!
stavros 10 hours ago [-]
310 tok/sec on my browser, hm.
HenryNdubuaku 10 hours ago [-]
Now you can try running the python package directly on your Mac to see proper speedup. WebAssembly has some limitations, running on pure hardware, especially ARM-based to see thousands (which 99% of consumer devices are ARM).
I see the reasoning is always last, but I have seen models do better if you put the reasoning as the first json fileld
havercosine 15 hours ago [-]
Congratulations. 28MB is impressive, I've not played around with actual queries/outputs.
I'm wondering what is the overall thesis/plan here and where exactly the innovation lies? Would love if you can throw light on below,
- If I understand, this is complete stack of a custom architecture (attention only transformers), custom quantisation format and a runtime engine all packaged together?
- How do you differentiate / compete against LiteRT (former TensorFlowLite) and Lite RT LM? Google is heavily investing in this ecosystem because Android is where they have distribution moat. Wouldn't it be easier for me as a developer to build on top of LiteRT since it is relatively open ecosystem and I can pack large number of open models from HF directly?
- What exact challenges you saw with TFLite, TVM etc that prompted this effort ?
- What will be the pricing model like?
HenryNdubuaku 14 hours ago [-]
So TFLite is an inference engine while Needle is a model, which you can port on TFLite btw.
redrix 1 days ago [-]
This is cool!
While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).
Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.
dalemhurley 1 days ago [-]
agreed, this is where we have the biggest opportunity for innovation.
HenryNdubuaku 1 days ago [-]
thanks!
hdhcbdb 3 hours ago [-]
I don't quite get what this can be used for.
It cannot be used for direct customer interaction, it's simply too limited.
Nobody wants to say "lock the front door", they would just do it
And even quite direct commands like "tighten security" are not understood by the model
cyprianaa263 9 hours ago [-]
A small on-device model with escalation is compelling. How do you decide when confidence is low enough to hand off to a larger model?
wky 9 hours ago [-]
Considering it as a classification problem, you could use a representative set of example queries, feed them to this model, then ask a "smart" language model to assess each query + result for whether the result is actually correct. Then you have a dataset you can compute FPR and FNR for any given threshold, and score based on the context, ex. turning the lights on or off is a lot less important if it is right compared to whether your door is locked. You could even pick it based on the tool call itself: Low threshold for locking the door, high threshold for unlocking the door.
peaknk 9 hours ago [-]
This is great, I have tried a couple different ways to hook up my Home assistant with Openclaw, Maybe my claw is bloated but a custom agent implementation which just has access to HA tool calling performs way better, although it costs money(Claude API). Would love to give this model a try and see how it performs.
kooi 19 hours ago [-]
Its pretty significant you've got this working locally in wasm. Very cool.
Re: robotics: I'm unsure how this could be helpful.
I changed it to "call path planner to navigate: a_star(x0, xf, obs)"
Another fail.
My intuition tells me micro llms will/are important for robotics. I just can't grok it. Can someone without control theory experience give me a good example?
Probably at the planning level of the navigation stack. That's where I see reasoning being helpful. Lower than that...idk
Give me an example of a robotics prompt that seems useful and I'll give you an example why we don't need LLMs to be a tracking controller, etc.
HenryNdubuaku 14 hours ago [-]
[dead]
semuggaemmanuel 6 hours ago [-]
This is so cool but for me safety remains my priority first in place. So, if it violates my registered rules, and does what it wasn't supposed too, then what next.
Today AI agents are becoming difficult to safety most especially in critical sectors like banking, a mere mistake may lead to loss of cash or life if it's a hospital
My question is how safe is it for critical sectors
hathym 1 days ago [-]
I tested with
import needle
@needle.tool
def add(a: int, b: int):
"Add two numbers."
return a + b
agent = needle.Needle(tools=[add])
print(agent.run("calculate 1 + 1?")["reasoning"])
python main.py
No calculator or math tool available.
conclusion: completly useless
HenryNdubuaku 1 days ago [-]
Try the following tool description: "Calculate the sum of two numbers. Use for any arithmetic or math question." instead of "Add two numbers." Let me know how it goes, thanks!
HenryNdubuaku 1 days ago [-]
It does better with clearer tool description, but we are taking note of these complaints for future improvements.
hathym 1 days ago [-]
works better that way, thanks :)
hathym 1 days ago [-]
but still struggle when changing the quesion:
import needle
@needle.tool
def add(a: int, b: int):
"Calculate the sum of two numbers. Use for any arithmetic or math question."
return a + b
agent = needle.Needle(tools=[add])
print(agent.run("what is 5 + 7?")["reasoning"])
>> No calculator or math tool available. Cannot compute numbers.
HenryNdubuaku 1 days ago [-]
Ah, another failure point on our end! So a simple "5 + 7" and "add 5 and 7" works. But to handle ambiguity, the python package ships pipelines to synthesize augmentations and fine-tune on your samples for robustness. Just run "needle playground" and use the UI. Apologies.
HenryNdubuaku 1 days ago [-]
Thanks, we shall improve this for the next release.
tolugenius 1 days ago [-]
This is really cool, I'm curious how much knowledge can their be in smaller models? It seems the current trade off is you need sizeably larger models for more performance but I'm curious if in your work how far this is true, as edge ai is really what needs to get better before physical ai can take off (my two cents).
msdz 1 days ago [-]
I imagine at such a low parameter count, there would be little to no world knowledge whatsoever, and the entire focus is on getting the structure of tool calling etc. right…?
But yeah, in terms of “physical” AI, robotics definitely comes to mind for me as well, where tool calls/structured “device” use in a “realtime”/edge application are highly beneficial (if you wanted to go with LLMs), but beefy hardware can’t be easily used.
rshemet 21 hours ago [-]
Roman from Cactus here -
yes you're right, there's only so much a 14MB model can do.
Needle excels at in-conext inference, with tightly defined environments. In our experience:
accurate descriptions + narrow tool scope = success
hgoel 1 days ago [-]
Makes me think of the demo from some time ago where someone got a ~29M parameter model running on an esp32. I wonder what kind of throughput this could get if a handful of esp32s were strung together...
Edit: I have a pile of d1 minis, but not much time.
HenryNdubuaku 1 days ago [-]
That demo was Needle 1 indeed and we are creating the guide for ESP32 now as we speak.
forsalebypwner 1 days ago [-]
They mention that this specific model is able to run on an ESP32-S3, or an ESP32-P4 which has 32MB of PSRAM. I'm trying to figure out how to do this now.
Robin_Message 15 hours ago [-]
> Turn the lights down low in the bedroom
Sets lights to 30% but also off
> Turn the lights low in the bedroom
Sets lights to on
This is a cool idea but I think humans assume more than 14MB of intelligence. This is like the unhelpful guard in the swamp castle of Monty Python's Holy Grail
HenryNdubuaku 14 hours ago [-]
Haha, its not a generalist model, cheap devices cannot run LLM in production, so we brig on-device intelligence to them in the size they can. Needle is niche, not a Gemma or GPT replacement.
Robin_Message 14 hours ago [-]
Right, but I don't think consumers would consider this "intelligence". It's too dumb.
pylotlight 21 hours ago [-]
What about use case for replacing regex?
I.e "random formatted title.extension" - extract the title or some tag or something for more dynamic string manipulation for pulling structured data out of strings efficiently and more simply than regex provides?
mmastrac 24 hours ago [-]
Congrats on this release. The WASM implementation is really cool. This is a surprisingly good fit for a lot of cases, and I totally want to try turning this into a helper assistant for an application.
Please, though, take a pass at humanizing the text on the page. It's Clauded up all over and makes it hard to read.
Very interesting, seems confidence is 0 when tool calls are right?
pylotlight 21 hours ago [-]
You may want to reword that.. what do you think 0 confidence means... ?
r0ze-at-hn 16 hours ago [-]
Curious, why did you go down to 2bit rather than 4 bits? 4bit with folding the layers should arrive at the same size, but with better quality?
HenryNdubuaku 14 hours ago [-]
A lot of Needle 1 users deployed on microcontrollers, we couldn't just throw them under the bus only t chase benchmarks, so its a price we are willing to pay :(
skavi 1 days ago [-]
I wonder if there's any way to get this to plan out a dag of tool calls? i.e. use the results from earlier calls as the parameters to later ones? I tried introducing a stack based system, but gave up pretty quickly.
HenryNdubuaku 1 days ago [-]
Yes, though for better results in production, after creating your tool json, use the provided data synthesis and fine-tuning pipeline. It tunes on on your mac.
dofm 1 days ago [-]
Naïve and clumsy question: how would you pair this with speech-text-speech stuff, wake words etc.? Are there good examples of this for a Pi 5?
The demo is super — I'm just having trouble seeing the whole picture for e.g. a screenless device.
That will get you a lot further than what you're asking, but if you dig a bit through Home Assistant features, resources, etc., you may find the current "best" answers to your questions.
If you want a quick answer: Whisper is a good open-source speech-to-text model which comes in a variety of sizes (https://huggingface.co/openai/whisper-tiny). You can definitely get something like this running on a Pi 5. There are plenty of other STT models out there, some of which are built specifically for this context (again, see the Home Assistant stuff), but Whisper comes up a lot as a good default choice.
So with something like Whisper, you could just have a simple script which is constantly listening to a rolling window of audio and transcribing it. When the transcription includes a key phrase, you can pass the rest of the transcription to Needle2 (or anything else for that matter). From there, you take the results and execute the necessary tool calls.
There's a bit more to all of this to make it work smoothly, but fundamentally this is all there is to it. All this would work very fast on a Pi 5 (although I wouldn't expect the results to be particularly good without some serious hand-crafted logic, fine-tuning, etc.). If you want to mess around this stuff, handing all of this to Claude, Codex, etc., can get you something spun up and functional very quickly.
silentbob7 18 hours ago [-]
The wyoming protocoll seems to be the path for home assistant audio, so you need STT (wyoming-faster-whisper), TTS (wyoming-piper for wide language support) API endpoints and some Ollama or OpenAI API endpoint available for your home assistant server.
dofm 16 hours ago [-]
That is interesting, thanks. And I see separable wake-word detection there. I'd always assumed that was likely to be closer to the metal (more like recognise a sound than turn sound to word).
dofm 1 days ago [-]
This is a very responsive answer, thank you so much. (I'd assumed maybe Whisper but the wake word "loop" detail there is illuminating.)
HenryNdubuaku 1 days ago [-]
Users often stack a transcription model on top to get the voice prompt, then decode to actions. Think of Alexa and Siri.
dofm 1 days ago [-]
Thank you.
shubhamsinghani 13 hours ago [-]
[flagged]
minimaltom 1 days ago [-]
Was really cool to see yous use Engrams to cut down compute!
Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup?
Also, why mHC over attention residuals?
HenryNdubuaku 1 days ago [-]
Yes, we ablated Engrams rigorously and found that it returned world knowledge like FFN without without compute expenditure.
minimaltom 1 days ago [-]
What about mHC? I'm surprised it helped with such a small compute budget.
HenryNdubuaku 1 days ago [-]
[flagged]
sroussey 1 days ago [-]
Looking forward to npm version of needle-rs supporting v2. I added needle support for tool use in my side project.
HenryNdubuaku 24 hours ago [-]
thanks, give the playground a go and let us know how to improve!
sroussey 23 hours ago [-]
I tried tweeking for structured extraction, but got issues with token budget. What is the context size?
raylad 13 hours ago [-]
It seems to fail. I sent the prompt:
“ 5° warmer”
And it said:
“ setting the temperature to 5°F”
rshemet 4 hours ago [-]
Did you give it a tool to increase temperature, or only one that sets temperature to an absolute value?
Either way, setting temperature to 5° is obviously wrong - even if it knew the current temperature - but models of this size can't reason about relative values very well.
Give it a tool to change temperature by a given amount, and see what happens!
brainless 10 hours ago [-]
This is awesome. I will take some time to dig in. When I am not working for client(s), I focus entirely on tiny LLMs - I have specific approach to prompting, avoid multi-turn chat and build harness to fit the selected LLM as closely as possible.
I’d be interested in attempting to run this in a network- enabled 32MiB RAM microVM.
messh 3 hours ago [-]
For the microcm, take a look at https://shellbox.dev, se3ms like a good fit. Real linux, and very affordable
HenryNdubuaku 1 days ago [-]
Thanks, how can we help?
yorwba 1 days ago [-]
"make it as dark as possible"
{
"function_calls": [
{
"name": "set_thermostat",
"arguments": {
"temperature": 72,
"mode": "cool",
"room": "living room"
}
}
],
"reasoning": "'as dark as possible' -> set_thermostat to warm; 'dark' implies higher temperature; 'cool' mode for darkness.",
"confidence": 0
}
... maybe this counts as dark humor at least.
Since it seems limited to matching a few templates and otherwise falling flat on its face, I wonder how 14MB of regexes would fare in its stead. Normally you wouldn't want to parse arbitrary natural language input with regex because of how tedious and brittle it would be, but for the tedium we have LLMs and this alternative isn't exactly robust either.
silentbob7 17 hours ago [-]
I also wonder how small a LLM trained on catching only subject (e.g. living room) and action (light on) from text input could be compared to needle - the json wrapping could be done afterwards using templates.
HenryNdubuaku 14 hours ago [-]
The problem is the target device, an LLM can't run on an average TV well.
HenryNdubuaku 14 hours ago [-]
SO when confused, it gives you a low confidence score as a signal, in which case its recommended to finetune using th python package, train on your mac/PC
snyp 20 hours ago [-]
This is so cool! Congrats to the team!
HenryNdubuaku 14 hours ago [-]
Thanks!
anr0 20 hours ago [-]
these micro LLMs could be a game changer for hearing aids
This is very interesting! I'm going to spend some time with this. This is really the only class of LLM I'm interested in at all. I sincerely hope on-device takes over and everyone looses their asses on these data centers.
HenryNdubuaku 14 hours ago [-]
Thanks! Let us know how it goes :)
varispeed 1 days ago [-]
What is the difference between this and random sentence generator?
HenryNdubuaku 1 days ago [-]
Random sentence is not a function call.
actionfromafar 1 days ago [-]
Ask it to lock a door for instance. It seems to convert simple instructions to reasonable tool calls. Check its confidence score.
yieldcrv 1 days ago [-]
what does the first L mean in LLM?
janalsncm 23 hours ago [-]
Fwiw people have told me that GPT2 doesn’t qualify as an LLM at 550MB despite being one of the first LLMs.
So the practical answer to your question is: not much.
kube-system 9 hours ago [-]
I don't think some people are aware that "large" has always referred to the training inputs not resulting the size of the model.
LLM is not "we made a language model and it is big" -- it's "we trained this model on a lot of language"
rshemet 21 hours ago [-]
it stands for Lets-not-be-sarcastic :)
peter_d_sherman 18 hours ago [-]
Utterly Fascinating!
For the longest time, I conceptualized LLM's as Text Input -> Text Output transformers, then later as Text Input -> Video Output transformers. Later still I conceptualized them (if they were general purpose) as Any Format Input -> Any Format Output transformers...
The idea of a smaller parameter model runable on smaller/slower/less complex hardware (computers with no GPU, slower CPU's, less memory, aka "Edge Devices") trained for Text Input -> JSON Output (used for tool calls, etc.) I could honestly not conceptualize before seeing the demo on the web page...
But now that I've seen it and conceptualized it -- I'd have to say: "Yes, there's definitely a huge niche, a huge market for this, directly between the non-LLM driven tools and software and SaaS's of yesteryear, and the latest, cutting edge Frontier AI models of today!"
So, I like Needle a lot!
I like Needle a lot, and I love the idea of any tiny resource-thrifty LLM that can run on older hardware, that outputs only JSON!
I can see a huge market for it!
HenryNdubuaku 14 hours ago [-]
You explained it better, we've done a bad job at communicating its nice :(
jreynar 11 hours ago [-]
[flagged]
SipitenoMK 6 hours ago [-]
[flagged]
rdos 11 hours ago [-]
[flagged]
liesliy 17 hours ago [-]
[flagged]
TokenLat 17 hours ago [-]
[flagged]
quantumeon 13 hours ago [-]
[flagged]
1 days ago [-]
hn7jmxa7oc 9 hours ago [-]
[dead]
sroussey 23 hours ago [-]
[dead]
rust-lang 11 hours ago [-]
[flagged]
jtgeibel 10 hours ago [-]
Please stop your obvious trolling. This contributes negative value to both the HN and Rust communities.
grenli 1 days ago [-]
The learned confidence gate is the crucial piece for a 14MB action model. On ambiguous requests such as the HN example, what calibration target decides between abstaining locally and escalating to the cloud?
HenryNdubuaku 1 days ago [-]
around +60% confidence threshold is cool from experiments, the problem is that you gotta test on your own workload, no existing benchmark could honestly paint the full picture, so we exposed the confidence threshold for everyone.
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).
For one thing, on beefy-enough recent CPUs, you could keep the weights hot in the L2 cache of a single CPU core. (Which is clearly not the use-case, but might be interesting to those looking for extreme TPS numbers. Or perhaps for efficient training!)
I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.
I do wonder if more smartness could be had by using sparser experts.... And possibly even having some kind of expert switching penalty to try to reduce the amount of data read from read only flash memory by encouraging subsequent tokens to use already loaded experts.
Query: "Make the living room dark" Agent: "User wants lights on in living room. 'dark' implies dim. Room 'living room', action 'on'." (And on every test I did, it just completely ignored the "brightness" parameter)
It also appears to have no concept of what a door or light actually is, whenever the query diverges from "Lock door X" or "Turn on light X", it tries to shoehorn whatever additional context is given into the device name:
Query: "Lock out the vacuum salesman at the front door" Agent tries to lock "front door vacuum salesman"
"The way you talk really makes me appreciate silence" is classified as "positive" with 82% confidence.
> Make it a little warmer in here.
The reply:
> "name": "set_thermostat", > "arguments": { > "temperature": 65, > "mode": "cool", > ... > "reasoning": "'warmer' implies need for cooling; set_thermostat with temperature 65 (typical warmth) and mode 'cool'.",
Maybe I'm doing it wrong?
So that could be a master home automation node, but why not also a single purpose device? I can think of more bad examples than I can good ones, but maybe I am doing some soldering and I need my soldering iron turned up a bit; my hands are full, so doing that by voice would be useful enough. Something I would never link up to a big AI model or home automation network, but could be useful to control by voice.
If it's something that can be burnt directly into a chip and shipped with the products for cheap, maybe that's a more pragmatic way to get AI into small devices (see taalas for a much bigger model doing that, althoug not yet cheap).
"Make it warmer"
tool that cools room called
Query: HN
Result:
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 }
I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.
Nonetheless, this is very cool work! If I can offer a small suggestion to the team at Cactus, it would be to evaluate your releases on some usability criteria (including false positives). Any serious integrator or adopter of these models would want to have that information available.
OP and the linked page talk about the confidence score and using it as an action threshold, so it looks like an appropriate total response to me.
I am not sure how a micro model will fundamentally solve it. Would love to understand what dannyw and team did there?
> I'm hungover
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. 'hungover' implies a security door. No specific door named, so use 'front door' as default.", "confidence": 0 }
Prompt: "Unlock my doors"
{ "function_calls": [], "reasoning": "No tool for unlocking or locking door exists. lock_door only performs lock/unlock actions, not unlock.", "confidence": 0.9492 }
Edit: to be clear, the counterintuitive behavior is that the confidence ended up higher for the wrong units.
For example, I mocked a dumbed down version of what would be a reasonable intermediate tool call prompt:
> It's currently 58 degrees. User asks for house to be 8.5 degrees warmer. What temperature to set thermostat to?
The reply?
Reasoning: “User asks for temperature to set thermostat to 8.5 -> set_thermostat with temperature=8.5.”
Sounds like something Siri would do!
Then I tried using it on a raspberry pi 4. Did not work.
This is specifically for 64 bit arm. I see no reason why you could not enable desktops as well? Just a matter of compilation
Do the creators take something like DeepSeek, and then delete most of the neurons to whittle down the size?
Once you have that, the model is small enough batch sizes are probably enormous and training can probably be done on a consumer-grade GPU in a week or less. Or even faster on a bigger GPU.
You may have inspired me to give it a try!
I'm wondering what is the overall thesis/plan here and where exactly the innovation lies? Would love if you can throw light on below,
While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).
Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.
It cannot be used for direct customer interaction, it's simply too limited.
Nobody wants to say "lock the front door", they would just do it
And even quite direct commands like "tighten security" are not understood by the model
Re: robotics: I'm unsure how this could be helpful.
It fails a pretty simple navigation prompt.
X0: (0.0, 0.0). Object bounding box: [1.0, 1.0, 2.0, 2.0]. navigate to (3.0,3.0)
I changed it to "call path planner to navigate: a_star(x0, xf, obs)"
Another fail.
My intuition tells me micro llms will/are important for robotics. I just can't grok it. Can someone without control theory experience give me a good example?
Probably at the planning level of the navigation stack. That's where I see reasoning being helpful. Lower than that...idk
Give me an example of a robotics prompt that seems useful and I'll give you an example why we don't need LLMs to be a tracking controller, etc.
Today AI agents are becoming difficult to safety most especially in critical sectors like banking, a mere mistake may lead to loss of cash or life if it's a hospital
My question is how safe is it for critical sectors
conclusion: completly useless
But yeah, in terms of “physical” AI, robotics definitely comes to mind for me as well, where tool calls/structured “device” use in a “realtime”/edge application are highly beneficial (if you wanted to go with LLMs), but beefy hardware can’t be easily used.
yes you're right, there's only so much a 14MB model can do.
Needle excels at in-conext inference, with tightly defined environments. In our experience:
accurate descriptions + narrow tool scope = success
Edit: I have a pile of d1 minis, but not much time.
Sets lights to 30% but also off
> Turn the lights low in the bedroom
Sets lights to on
This is a cool idea but I think humans assume more than 14MB of intelligence. This is like the unhelpful guard in the swamp castle of Monty Python's Holy Grail
Please, though, take a pass at humanizing the text on the page. It's Clauded up all over and makes it hard to read.
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "tv" } } ], "confidence": 0.0158 }
Very interesting, seems confidence is 0 when tool calls are right?
The demo is super — I'm just having trouble seeing the whole picture for e.g. a screenless device.
ETA: pun not intended
That will get you a lot further than what you're asking, but if you dig a bit through Home Assistant features, resources, etc., you may find the current "best" answers to your questions.
If you want a quick answer: Whisper is a good open-source speech-to-text model which comes in a variety of sizes (https://huggingface.co/openai/whisper-tiny). You can definitely get something like this running on a Pi 5. There are plenty of other STT models out there, some of which are built specifically for this context (again, see the Home Assistant stuff), but Whisper comes up a lot as a good default choice.
So with something like Whisper, you could just have a simple script which is constantly listening to a rolling window of audio and transcribing it. When the transcription includes a key phrase, you can pass the rest of the transcription to Needle2 (or anything else for that matter). From there, you take the results and execute the necessary tool calls.
There's a bit more to all of this to make it work smoothly, but fundamentally this is all there is to it. All this would work very fast on a Pi 5 (although I wouldn't expect the results to be particularly good without some serious hand-crafted logic, fine-tuning, etc.). If you want to mess around this stuff, handing all of this to Claude, Codex, etc., can get you something spun up and functional very quickly.
Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup?
Also, why mHC over attention residuals?
“ 5° warmer”
And it said:
“ setting the temperature to 5°F”
Either way, setting temperature to 5° is obviously wrong - even if it knew the current temperature - but models of this size can't reason about relative values very well.
Give it a tool to change temperature by a given amount, and see what happens!
My experiments are in https://github.com/brainless/
I will be happy to share what I learn.
In the meantime, if you have enough RAM for the current model (≈28MB), our repo will get you up & running:
https://github.com/cactus-compute/needle
Since it seems limited to matching a few templates and otherwise falling flat on its face, I wonder how 14MB of regexes would fare in its stead. Normally you wouldn't want to parse arbitrary natural language input with regex because of how tedious and brittle it would be, but for the tedium we have LLMs and this alternative isn't exactly robust either.
so many interesting lowfi hardware use cases
but if you're just looking for somewhere to try the model, use our in-browser playground! - https://cactuscompute.com/needle
could you say more? What kind of home assistant / what stack
https://www.home-assistant.io/voice_control/best_practices/
https://developers.home-assistant.io/docs/core/llm/
https://developers.home-assistant.io/docs/core/entity/conver...
Something like the Ollama integration:
https://www.home-assistant.io/integrations/ollama/
So the practical answer to your question is: not much.
LLM is not "we made a language model and it is big" -- it's "we trained this model on a lot of language"
For the longest time, I conceptualized LLM's as Text Input -> Text Output transformers, then later as Text Input -> Video Output transformers. Later still I conceptualized them (if they were general purpose) as Any Format Input -> Any Format Output transformers...
The idea of a smaller parameter model runable on smaller/slower/less complex hardware (computers with no GPU, slower CPU's, less memory, aka "Edge Devices") trained for Text Input -> JSON Output (used for tool calls, etc.) I could honestly not conceptualize before seeing the demo on the web page...
But now that I've seen it and conceptualized it -- I'd have to say: "Yes, there's definitely a huge niche, a huge market for this, directly between the non-LLM driven tools and software and SaaS's of yesteryear, and the latest, cutting edge Frontier AI models of today!"
So, I like Needle a lot!
I like Needle a lot, and I love the idea of any tiny resource-thrifty LLM that can run on older hardware, that outputs only JSON!
I can see a huge market for it!