Posts

Important bug found: Bad context size for the ARC prize

-
Image
 For the ARC price https://arcprize.org/, today I have: - I fixed a bug that caused my neural network to not see all the input text of a sample because the context was 140 characters but the samples were 211 characters - I increased the number of layers from 4 to 8. To do this on the NVIDIA A40 GPU on runpod.io/, I changed my batch_size from 1024 to 512 and changed num_layers from 4 to 8.  
Post a Comment
Read more

ADAM optimizer, perplexity, and a bunch of CUDA kernels

-
Opcodes Novigrad opcodes are currently these: pub enum OpCode { GemmNTN, GemmNNN, GemmTNN, GemmTNT, ReduceSum, Add, ScalarAdd, ScalarMul, Clip, Normalize, Mul, Div, Sqrt, Softmax, Sub, Reshape(Vec<usize>), Sigmoid, CrossEntropyLoss, ResidualSumOfSquares, Dropout(f32), Concat, Unconcat, }  see https://github.com/sebhtml/novigrad/blob/b691fefdc88c6d7a2ba1634bad6568b9fe44eaa9/src/neural_machine/opcode.rs   CUDA I read Optimizing Parallel Reduction in CUDA by Mark Harris of NVIDIA.  I am not a CUDA ninja. But I wrote a bunch of kernels. The reduction stuff is nice. See cross_entropy_loss_kernel.cu : https://github.com/sebhtml/novigrad/blob/7e7738af5ba3d8e0c51d7e1a5dbae66e5bc9f397/src/devices/cuda/kernels/cross_entropy_loss_kernel.cu All of them: https://github.com/sebhtml/novigrad/tree/b691fefdc88c6d7a2ba1634bad6568b9fe44eaa9/src/devices/cuda/kernels Adam I implemented the ADAM optimizer. S
Post a Comment
Read more