AI: bump vocab_size from 256 to 34816 using byte pair encoding

-

BPE

My byte-pair encoding (BPE) implementation works.

 

 

PR: https://github.com/sebhtml/novigrad/pull/17

I increased vocab_size from 256 to 34816 with BPE.

 

GPU

Thanks to NVIDIA, I can do this on my laptop that has a NVIDIA RTX 4060.



 

Tokens

The text I am using has 78970 characters.

With my BPE implementation, this text is encoded into 44513 tokens.



Loss

The loss goes from 108.2 to 0.

Epoch 0 Total_error 108.20975, change: NaN
Epoch 100 Total_error 0, change: -1
Epoch 300 Total_error 0, change: NaN
 

 

Example 0

input_text: {{short description|Video game franchise}}

{{About|the vi

expected_output_text: de


input_tokens: [22796, 30981, 11071, 20834, 19622, 31405, 30073, 11, 23050, 12550, 13, 32014, 19623, 16841, 6070, 17, 125, 28674, 1460, 15833, 20052, 8, 22923, 20, 22797, 21, 24292, 31531, 13, 5289, 126, 24990]


expected_output_token: [19624]

 

Example 0 before training:

Loss 11.230275
actual_output_token: [13142]
actual_output_text: me

 

Example 0 after training:

Loss -0
actual_output_token: [19624]
actual_output_text: de

 

Weirdness

The weird thing is that the neural network can generate a token that is not known to the BPE decoder. In this case, I chose to simply emit '?'.
 

Comments

Post a Comment

Popular posts from this blog

The Thorium actor engine is operational now, we can start to work on actor applications for metagenomics

-
I have been very busy during the last months. In particular, I completed my doctorate on April 10th, 2014 and we moved from Canada to the United States on April 15th, 2014. I started a new occupation on April 21st, 2014 at Argonne National Laboratory (a U.S. Department of Energy laboratory). But the biggest change, perhaps, was not one listed in the enumeration above. The biggest change was to stop working on Ray. Ray is built on top of RayPlatform, which in turn uses MPI for the parallelism and distribution. But this approach is not an easy way of devising applications because message passing alone is a very leaky, not self-contained, abstraction. Ray usually works fine, but it has some bugs . The problem with leaky abstractions is that they lack simplicity and are way too complex to scale out. For example, it is hard to add new code to an existing code base without breaking anything. This is the case because MPI only offers a fixed number of ranks. Sure, the MPI standard has s...
Read more

Learning to solve the example 1 of puzzle 3aa6fb7a in the ARC prize

-
Image
Hi dear reader, Puzzle rules The example 1 of the puzzle 3aa6fb7a of the ARC prize looks like this: A playout starts with an blank board of width 7 and of height 7. There are therefore 49 cells. Each cell can be blank or can contain one of the 10 possible colors. The colors are identified from 0 to 9.  I added a rule. This rule is that each cell has to be changed exactly once.  The game ends when all cells have been changed. The player wins if all the cell colors match the cell colors in the expected output. Some numbers: Each cell has 11 possible values: 10 colors + blank state There are 49! ( 6.0828186e+62) distinct possible orders of cell changes in a playout.  There are 49!*10 ( 6.0828186e+63) 49 * 10 = 490 possible actions for the first action.  There are 11^49 ( 1.0671896e+51) possible board states The number of possible pairs (s, a) is even larger and upper-bounded by 11^49 * 49 * 10 = 5.2292289e+53 . Given a board state s and a proposed action a, an actio...
Read more

Adding ZVOL VIRTIO disks to a guest running on a host with the FreeBSD BHYVE hypervisor

-
On atlantis.domain (my FreeBSD BHYVE hypervisor host), I use these commands: Install FreeBSD on a 10G disk. vm create -s 10G guestname   vm install FreeBSD-12.1-RELEASE-amd64-memstick.img vm console guestname Add 8 30GB zvol-based disks vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname You will need to do vm poweroff guestname vm start guestname because the guest does not seem to see the 8 30GB without power cycling. (rebooting won't work) *** If you want the boot disk (the one create by "vm create") to be a zvol, you must edit the template: root@atlantis:~ # zfs snapshot tank/vm-bhyve@2020-06-11-2145 root@atlantis:~ # mount -t zfs tank/vm-bhyve@2020-06-11-2145 /tank/vm-bhyve@2020-06-11-2145 r...
Read more