ADAM optimizer, perplexity, and a bunch of CUDA kernels
Opcodes Novigrad opcodes are currently these: pub enum OpCode { GemmNTN, GemmNNN, GemmTNN, GemmTNT, ReduceSum, Add, ScalarAdd, ScalarMul, Clip, Normalize, Mul, Div, Sqrt, Softmax, Sub, Reshape(Vec<usize>), Sigmoid, CrossEntropyLoss, ResidualSumOfSquares, Dropout(f32), Concat, Unconcat, } see https://github.com/sebhtml/novigrad/blob/b691fefdc88c6d7a2ba1634bad6568b9fe44eaa9/src/neural_machine/opcode.rs CUDA I read Optimizing Parallel Reduction in CUDA by Mark Harris of NVIDIA. I am not a CUDA ninja. But I wrote a bunch of kernels. The reduction stuff is nice. See cross_entropy_loss_kernel.cu : https://github.com/sebhtml/novigrad/blob/7e7738af5ba3d8e0c51d7e1a5dbae66e5bc9f397/src/devices/cuda/kernels/cross_entropy_loss_kernel.cu All of them: https://github.com/sebhtml/novigrad/tree/b691fefdc88c6d7a2ba1634bad6568b9fe44eaa9/src/devices/cuda/kernels Adam I implemented the ADAM optimizer. S