KERNELS_DIR_PATH in Dockerfile should be /root/.local/share/jupyter/kernels, not /home/$NB_USER/.local/share/jupyter/kernels, as NB_USER doesn’t exist (commented out).
argv in kernel.jsonshould be ~/.local/share/jupyter/kernels/PySpark/pyspark-kernel.sh, not ~/.localshare/jupyter/kernels/PySpark/pyspark-kernel.sh.
/usr/bin/python3 in pyspark-kernel.sh should be /usr/local/bin/python3.
nigel.stanger
added a commit that referenced this issue
on 19 Sep 2021
Honestly, it’s a wonder this even works at all!
KERNELS_DIR_PATHinDockerfileshould be/root/.local/share/jupyter/kernels, not/home/$NB_USER/.local/share/jupyter/kernels, asNB_USERdoesn’t exist (commented out).argvinkernel.jsonshould be~/.local/share/jupyter/kernels/PySpark/pyspark-kernel.sh, not~/.localshare/jupyter/kernels/PySpark/pyspark-kernel.sh./usr/bin/python3inpyspark-kernel.shshould be/usr/local/bin/python3.