Back
Sanskar
Keen Learner and Exp... • 18h
Day 2 of learning AI/ML as a beginner. Topic: text preprocessing (tokenization) in NLP. I have moved further and decided to learn about Natural Language Process(NLP) which is used especially for translations, chatbots, and help them to generate human like responses (in human readable language). I have also created a roadmap of learning NLP which I will be following to learn it in a more structured manner. I have already started with text preprocessing theory more specifically of tokenization. Tokenization is the process of breaking down text into smaller units called tokens. These tokens can be sentences or even words depending upon the level of tokenization applied. Tokenization have four main technical jargons namely: 1. Corpus - this refers to paragraphs. 2. Documents - this refers to sentences. 3. Vocabulary - these are the unique words used in a sentence or paragraph. 4. Words - these are the normal words we use. Tokenization typically depends upon the use of punctuation in order to create tokens. I have scratched the surface of NLP and will most probably apply this practically in my python code. I will warmly welcome all the questions, suggestions, recommendations and "constructive" criticism (the one which contains the problem and its likely solution, I will research the rest). And also here are my notes which I made while learning this.
More like this
Recommendations from Medial
Binit shrestha
Believe in yourself • 1y
Kirill Bensonoff, co-founder of New Silver, highlights how blockchain technology is revolutionizing real estate through smart contracts and tokenization. Smart contracts automate and speed up transactions like property transfers by executing predefin
See More
Account Deleted
Hey I am on Medial • 9m
🤖 Named Entity Recognition (NER) plays a crucial role in Natural Language Processing by helping machines understand and categorize key information from text. Discover its significance and applications in our latest article! Read more: http://news.e
See More
Swastik Biswas
CTO @OctranTechnolog... • 5m
OpenAI has recently released their latest reasoning model, "o1-pro", in their developer APIs. It is estimated to be the most expensive AI model yet. OpenAI is charging $150 per million tokens (~750,000 words) fed into the model and $600 per million
See More
Dr Bappa Dittya Saha
We're gonna extinct ... • 1y
Studying content 101 Soon videos will be replaced by Text on ur screens- I mean stories! The Drama, The Music, The lights! The Darks! All in words of same sheds! And the target audience will be far better when it raises equations about their sociod
See More
Download the medial app to read full posts, comements and news.