Daniel J. Dorado
Computational Linguist
Natural Language Processing
» project
Computational Linguist
» project
Congratulations on making it to the last post in making a tokenizer. Let’s jump right in and bring in our...
Grammar So as mentioned in the first post, the grammar is where rules are specified. These grammatical rules will check...
Exceptions So let’s get started making the exceptions.py file. As you see this, is pretty basic. It’s just a Python...
Introduction So in this post, we’ll start making a simple English tokenizer in pure Python. Simple in the sense that...