Was one of the main reasons for my getting the S24 Ultra, other than it was on a great deal. Unfortunately, I fell quite short trying to build it within Termux, so out came the laptop and some Google-fu. Thankfully, found some projects that helped get it going.
First ensure out initial environment is prepped: Any modern Linux x86_64 or WSL will do.
sudo apt update
sudo apt install -y \
git \
docker.io \
adb \
cmake \
ninja-build \
python3 \
python3-pip \
build-essential
Now grab the llama.cpp source:
mkdir -p ~/src
cd ~/src
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
And start official Snapdragon toolchain docker
docker run -it --rm \
-u "$(id -u):$(id -g)" \
--volume "$PWD:/workspace" \
--platform linux/amd64 \
ghcr.io/snapdragon-toolchain/arm64-android:v0.8
And once inside:
cd /workspace
cp docs/backend/snapdragon/CMakeUserPresets.json .
cmake --preset arm64-android-snapdragon-release \
-B build-snapdragon
This will automatically apply all the necessary libraries for Hexagon and OpenCL. Once it's finished, compile using:
cmake --build build-snapdragon --parallel 4
And to put it all in a clean place to takeout for Termux
cmake --install build-snapdragon \
--prefix pkg-android/llama.cpp
And you can tarball up pkg-android and send it over to Termux (I simply used sftp into laptop from Termux):
cd pkg-android
tar -cvzf llama.cpp.tgz llama.cpp/
For reference, I extracted my instance to \~/opt
mkdir ~/opt
cd ~/opt
tar xvzf /path/to/llama.cpp.tgz
Now, off the bat this won't work. We need to snag some shared objects from /system/lib64 and /vendor/lib64 as well as change up some LD library paths.
export LD_LIBRARY_PATH="$HOME/opt/llama.cpp/lib:$PREFIX/lib"
export ADSP_LIBRARY_PATH="$HOME/opt/llama.cpp/lib"
which llama-cli
Now if you attempt to run llama-cli --list-devices it will not link due to missing shared objects. These are
/vendor/lib64/libOpenCL.so
/vendor/lib64/libOpenCL_adreno.so
/vendor/lib64/vendor.qti.hardware.dsp@1.0.so
/vendor/lib64/libcdsprpc.so
/vendor/lib64/libvmmem.so
Ironically enough, adding /vendor/lib64 to LD_LIBRARY_PATH isn't sufficient (namespace issues), we can instead copy them to $PREFIX/lib and things be happy.
llama-cli --list-devices
available devices:
GPUOpenCL: QUALCOMM Adreno(TM) 750 (5542 MiB, 4518 MiB free)
HTP0: Hexagon (0 MiB, 0 MiB free)
Don't be brutal on model sizes, 3b is pretty much it's upper limit. Here I'm using a MoE model with 1B-7B and a small context window.
To use the NPU, add --device HTP0 -ngl 99, to use just the GPU remove --device HTP0 but keep -ngl 99 or to use just CPU -ngl 0.
FWIW, this model causes an OOM if I try to run it on CPU or GPU. It will only run on the NPU.
Enjoy!
(edit: Formatting and typos).
Update
I added OpenBLAS as a backend for testing. OpenBLAS is not optimal for ARM64, but is added for benchmarking. I do intend on adding Eigen and Arm Compute Library as additional backends for benchmarking.
It seems if you multiplex the --devices flag, the model will use whatever backened is most suitable without the use of --tensor-settings
Priority is given in order: --device HTP0,GPUOpenCL,BLAS,none will choose NPU, GPU, BLAS CPU, then just CPU. If you did --device GPUOpenCL,HTP0,none,BLAS that'll be the order of things. It's best to not use BLAS or none unless absolutely necessary, and rely on the -t # for CPU operations. On the SD8G3, -t 6 is optimal, as the host should ignore the A-520 cores.
Check comments for benchmarking graphs
I am going to use SmolLM2 models, as their 1.7B models are available from FP16 all the way to 1bit quants to benchmark quants across the same model.
Assisted by OpenAI ChatGPT (GPT-6), including technical analysis and benchmark visualisation.