DSKernel: AI and Strength Training

Hi dear reader, Puzzle rules The example 1 of the puzzle 3aa6fb7a of the ARC prize looks like this: A playout starts with an blank board of width 7 and of height 7. There are therefore 49 cells. Each cell can be blank or can contain one of the 10 possible colors. The colors are identified from 0 to 9.  I added a rule. This rule is that each cell has to be changed exactly once.  The game ends when all cells have been changed. The player wins if all the cell colors match the cell colors in the expected output. Some numbers: Each cell has 11 possible values: 10 colors + blank state There are 49! ( 6.0828186e+62) distinct possible orders of cell changes in a playout.  There are 49!*10 ( 6.0828186e+63) 49 * 10 = 490 possible actions for the first action.  There are 11^49 ( 1.0671896e+51) possible board states The number of possible pairs (s, a) is even larger and upper-bounded by 11^49 * 49 * 10 = 5.2292289e+53 . Given a board state s and a proposed action a, an actio...

 For the ARC price https://arcprize.org/, today I have: - I fixed a bug that caused my neural network to not see all the input text of a sample because the context was 140 characters but the samples were 211 characters - I increased the number of layers from 4 to 8. To do this on the NVIDIA A40 GPU on runpod.io/, I changed my batch_size from 1024 to 512 and changed num_layers from 4 to 8.  

Opcodes Novigrad opcodes are currently these: pub enum OpCode { GemmNTN, GemmNNN, GemmTNN, GemmTNT, ReduceSum, Add, ScalarAdd, ScalarMul, Clip, Normalize, Mul, Div, Sqrt, Softmax, Sub, Reshape(Vec<usize>), Sigmoid, CrossEntropyLoss, ResidualSumOfSquares, Dropout(f32), Concat, Unconcat, }  see https://github.com/sebhtml/novigrad/blob/b691fefdc88c6d7a2ba1634bad6568b9fe44eaa9/src/neural_machine/opcode.rs   CUDA I read Optimizing Parallel Reduction in CUDA by Mark Harris of NVIDIA.  I am not a CUDA ninja. But I wrote a bunch of kernels. The reduction stuff is nice. See cross_entropy_loss_kernel.cu : https://github.com/sebhtml/novigrad/blob/7e7738af5ba3d8e0c51d7e1a5dbae66e5bc9f397/src/devices/cuda/kernels/cross_entropy_loss_kernel.cu All of them: https://github.com/sebhtml/novigrad/tree/b691fefdc88c6d7a2ba1634bad6568b9fe44eaa9/src/devices/cuda/kernels Adam I implemented the ADAM...