How does the Apple Neural Engine actually work?

The Apple Neural Engine (ANE) is a dedicated chip inside Apple's A-series and M-series processors designed to run machine learning tasks extremely fast while using very little power. It is separate from the CPU and GPU.

Machine learning tasks involve one type of calculation repeatedly: multiplying large matrices of numbers together. A general CPU does this slowly because it handles many task types. The ANE does only this but does it at trillions of operations per second (TOPS).

Technically, the ANE is a matrix multiplication engine. It contains multiply-accumulate units simple circuits that multiply two numbers and add the result to a running total. That is the core operation of any neural network.

When an app uses Core ML (Apple's machine learning framework), the software decides whether to run on the CPU, GPU, or ANE based on speed and efficiency. The ANE handles inference (using a trained model to make a prediction). Model training still happens on servers.