My understanding of Neural Processing Unit is primarily design for speed up processes with matrix multiplications, the main usage currently is with tensorflow, blur in video feed and some finishing word sentences.
This is only brief, I’m sure there is more to it but I only have experience from ANE on Apple devices which did this exactly, and access was not quite straight only via coreml framework.
But since couple years things a pushing to bigger unified memories, faster “AI”( don’t like this buzzing word but it’s used for referencing previously mentioned multiplication ), why we don’t use this processing unit in game?
As far as I remember all was done from 90’s on CPU and Ram, then graphic cards came around and increased possibility of 3d, then Physx, etc.
Now on average Apple Silicone you see a quite few features which suppose to speed up video processing, etc.
Why don’t we see usage of NPU in game ?
If we do what is example ?
There are too few in the used systems. Only available in some new PCs.
What is possible with them, that’s not possible with a good GPU?
My conspiracy theory with these chips is that they are for the companies to run small local models on your system to surveillance the user. Like Recall (Copilot) or the new Gemini/Gemma in Chrome that Google is installing on your system. They all can analyze screenshots to summarize them and send the result (as text) back to MS or Google. Or to local monitor your chats and internet activities. They will offer some features of that to the user, like detecting people in your photos etc.
There are close to none local LLM solutions that can use the NPU, all using the GPU.
Maybe some developers will find other ways to use these chips, but it will take some time to see such support in Godot or other engines or apps.
But it’s hard to tell as the development is so fast in that field. Maybe it’s different with the latest Mac and PC CPUs with integrated GPU&NPU (APUs).
The underlying engine called llama.cpp is entirely capable of doing this, and almost all modern wrappers around it make use of it extensively, such as KoboldCPP, Ollama and LM Studio.
I mean this is a bit secretive on Apple side, as part of framework is open source, then you have strict guidelines, and rest is reverse engineering on good people side, and then in next sdk you see deprecated and replaced with something new.
But back to point of GPU and ANE( NPU on Apple ), we see this on mobile device architecture which ( Apple Silicone is), and there is lower power consumption for this multiplication so maybe it could take part of load into ANE instead of iGPU to unload those GPU cores, still wonder the unified memory how could this be used in those hardware encoder/decoder which be super cool to see used in cutscenes ( DavinciResolve actually utilising them during editing and export )
As I understand it the main part is used on the GPU, and you could use the NPU for some parts of the LLM, but the NPU is for small models. You could run a small neural network or small LLM over it for some tasks.
You could use the NPU for text to speech, or NPC control per NN, or object detection in images etc.
Default in llama.cpp the NPU is not used. You could use them with close to none benefit (more efficient). It’s slower than a GPU (on larger models). But the line between NPU and GPU gets more and more blurry. And Mac with unified memory is something else. Can’t speak for Mac hardware.
), why we don’t use this processing unit in game?