Daniel J. Dorado
Computational Linguist
Natural Language Processing
» nlp
Computational Linguist
» nlp
Congratulations on making it to the last post in making a tokenizer. Let’s jump right in and bring in our...
Grammar So as mentioned in the first post, the grammar is where rules are specified. These grammatical rules will check...
Exceptions So let’s get started making the exceptions.py file. As you see this, is pretty basic. It’s just a Python...
Introduction So in this post, we’ll start making a simple English tokenizer in pure Python. Simple in the sense that...
Introduction Tokenization is an import step in the NLP pipeline. It is often part of the text normalization process. Many...
Normalizing Text with Regex Groups in Python In this post we’re going to look at how regex groups can help...