Step 2 of 8

Preprocessing

Normalize the corpus, add sentence boundaries, and shrink the vocabulary by replacing rare words with <UNK>.

No corpus loaded yet. Pick one first.

Tweak and re-run.

LowercaseCollapse extra spacesKeep punctuation as tokensAdd <s> and </s>

Vocabulary cap1000

StartEndUnknown

Click Apply preprocessingto see the tokenized output.