Welcome to tidyX’s documentation!

tidyX is a Python package designed for cleaning and preprocessing text for machine learning applications, especially for text written in Spanish and originating from social networks. This library provides a complete pipeline to remove unwanted characters, normalize text, group similar terms, etc. to facilitate NLP applications.

Installation

Install the package using pip:

pip install tidyX

Make sure you have the necessary dependencies installed. If you plan on lemmatizing, you’ll need spaCy along with the appropriate language models. For Spanish lemmatization, we recommend downloading the es_core_web_sm model:

python -m spacy download es_core_news_sm

For English lemmatization, we suggest the en_core_web_sm model:

python -m spacy download en_core_web_sm

To see a full list of available models for different languages, visit Spacy’s documentation.

Usage

• Standardize Text Pipeline
• Stemming and Lemmatizing
• Group similar terms
• Remove Unwanted Elements
  • Special characters
  • Emojis and/or accents
  • URLs
  • Extra spaces
  • Mentions, RT prefix or hashtags
  • Stopwords or any other concrete word
  • Repetition of characters
• Dependency Parsing Visualization

Tutorials

• Stemming and Lemmatizing Texts Efficiently
• Word Cloud
• Topic Modelling
  • Introduction
  • Context
  • Preprocessing Tweets
  • Tokenize and lemmatize tweets in the dataset
  • Seemingly used words and social media bad writting addressing
  • Conclusion and Analysis of toy-Topic Modelling Results

User Documentation

• TextPreprocessor
  • TextPreprocessor
    • TextPreprocessor.create_bol()
    • TextPreprocessor.get_most_common_strings()
    • TextPreprocessor.load_data()
    • TextPreprocessor.load_stopwords()
    • TextPreprocessor.preprocess()
    • TextPreprocessor.remove_RT()
    • TextPreprocessor.remove_accents()
    • TextPreprocessor.remove_extra_spaces()
    • TextPreprocessor.remove_hashtags()
    • TextPreprocessor.remove_last_repetition()
    • TextPreprocessor.remove_mentions()
    • TextPreprocessor.remove_repetitions()
    • TextPreprocessor.remove_special_characters()
    • TextPreprocessor.remove_urls()
    • TextPreprocessor.remove_words()
    • TextPreprocessor.space_between_emojis()
    • TextPreprocessor.unnest_tokens()
• TextNormalization
  • TextNormalization
    • TextNormalization.is_emoji()
    • TextNormalization.lemmatizer()
    • TextNormalization.stemmer()
• TextVisualizer
  • TextVisualizer
    • TextVisualizer.dependency_parse_visualizer_text()

Contributing

Contributions to enhance tidyX are welcome! Feel free to open issues for bug reports, feature requests, or submit pull requests in our github repo. If this package has been helpful, please give us a star :D