# Project: Analyze Text Data with Yellowbrick

Tasks such as assessing document similarity, topic modelling and other text mining endeavors are predicated on the notion of “closeness” or “similarity” between documents. In this project, we define various distance metrics (e.g. Euclidean, Hamming, Cosine, Manhattan, etc) and understand their merits and shortcomings as they relate to document similarity. We will apply these metrics on documents within a specific corpus and visualize our results.

Duration (mins)

Learners

#### NA / 5

Rating

We will cover the following tasks in 59 minutes:

We will understand the Rhyme interface and our learning environment. You will be provided with a cloud desktop with Jupyter Notebooks and all the software you will need to complete the project. Jupyter Notebooks are very popular with Data Science and Machine Learning Engineers as one can write code in cells and use other cells for documentation.

Next, we will load the `hobbies` corpus from a local directory and explore its attributes.

### Vectorizing the Documents

Text data requires special preparation before you can start using it for predictive modeling. The text must be parsed to remove words, called tokenization. Then the words need to be encoded as integers or floating point values for use as input to a machine learning algorithm, called feature extraction (or vectorization).

We will use scikit-learn’s `TfidfVectorizer` to vectorize our corpus and then extract the documents and labels.

### Clustering Similar Documents with Squared Euclidean Distance And Euclidean Distance

To effectively cluster similar documents, we will use t-distributed Stochastic Neighbor Embeddings. It decomposes high-dimensional document vectors into 2 dimensions using probability distributions from both the original dimensionality and the decomposed dimensionality, By decomposing to 2 or 3 dimensions, the documents can be visualized with a scatter plot.

We will use Yellowbrick’s `TSNEVisualizer`, with two metrics: squared Euclidean distance and the Euclidean distance.We will find that the Euclidean distance is not an ideal metric to be using. That’s because when we vectorize a corpus, we end up with huge, sparse vectors.

### Manhattan (aka “Taxicab” or “City Block”) Distance

In general, choosing which distance metric to use is an important, but often ignored modelling problem. A lot of practitioners end up using the `L2` distance just because it’s easy to remember. However, in many situations, the L1 or Manhattan distance makes more sense to use and is more robust.

In this task, we define what constitutes a good distance metric and apply the Manhattan distance to our text data.

### Bray Curtis Dissimilarity and Canberra Distance

Here we take a look at two other non-Euclidean metrics and see if they are applicable to document similarity. Both metrics have a different set of assumptions and have been used historically for ecological purposes.

### Cosine Distance

We can also measure vector similarity with cosine distance, using the cosine of the angle between the two vectors to assess the degree to which they share the same orientation.

Cosine distance is often an excellent option for text data because it corrects for any variations in the length of the documents (since we’re measuring the angle between vectors rather than their magnitudes). Moreover, it can be a very efficient way to compute distance with sparse vectors because it considers only the non-zero elements.

### What Metrics Not to Use

There are also a bucket full of distance metrics that make no sense at all for sparse, non-numeric data. Let’s take a look!

### Omitting Class Labels - Using KMeans Clustering

In this task, we experiment with omitting the class labels entirely during visualization to see if any meaningful patterns are observed.

We can then use cluster membership from K-Means to label each document. This will allow us to look for clusters of related text by their contents.

## Watch Preview

Preview the instructions that you will follow along in a hands-on session in your browser.

## About the Host (Snehan Kekre)

Snehan Kekre is a Machine Learning and Data Science Instructor at Coursera. He studied Computer Science and Artificial Intelligence at Minerva Schools at KGI, based in San Francisco. His interests include AI safety, EdTech, and instructional design. He recognizes that building a deep, technical understanding of machine learning and AI among students and engineers is necessary in order to grow the AI safety community. This passion drives him to design hands-on, project-based machine learning courses on Rhyme.

## Frequently Asked Questions

##### How is this different from YouTube, PluralSight, Udemy, etc.?
In Rhyme, all projects are completely hands-on. You don't just passively watch someone else. You use the software directly while following the host's (Snehan Kekre) instructions. Using the software is the only way to achieve mastery. With the "Live Guide" option, you can ask for help and get immediate response.
##### What do I need to install / configure to join this session?
Nothing! Just join through your web browser. Your host (Snehan Kekre) has already installed all required software and configured all data.
##### Is this session really free?
Absolutely! Your host (Snehan Kekre) has provided this session completely free of cost!
##### How do I create my own projects like this?
You can go to https://rhyme.com, sign up for free, and follow this visual guide How to use Rhyme to create your own projects. If you have custom needs or company-specific environment, please email us at help@rhyme.com
##### Can I buy Rhyme sessions for my company or learning institution?
Absolutely. We offer Rhyme for workgroups as well larger departments and companies. Universities, academies, and bootcamps can also buy Rhyme for their settings. You can select projects and trainings that are mission critical for you and, as well, author your own that reflect your own needs and tech environments. Please email us at help@rhyme.com
##### What kind of accessibility options does Rhyme provide?
Rhyme strives to ensure that visual instructions are helpful for reading impairments. The Rhyme interface has features like resolution and zoom that will be helpful for visual impairments. And, we are currently developing a close-caption functionality to help with hearing impairments. Most of the accessibility options of the cloud desktop's operating system or the specific application can also be used in Rhyme. If you have questions related to accessibility, please email us at accessibility@rhyme.com
##### Why don't you just use containers or virtual browsers?
We started with windows and linux cloud desktops because they have the most flexibility in teaching any software (desktop or web). However, web applications like Salesforce can run directly through a virtual browser. And, others like Jupyter and RStudio can run on containers and be accessed by virtual browsers. We are currently working on such features where such web applications won't need to run through cloud desktops. But, the rest of the Rhyme learning, authoring, and monitoring interfaces will remain the same.
##### I have a different question
Please email us at help@rhyme.com and we'll respond to you within one business day.

## More Projects by Snehan Kekre

43 minutes
###### Classify Toxic Comments using Convolutions with Keras
1 hour and 11 minutes
42 minutes
57 minutes
41 minutes
52 minutes
###### Project: Facial Expression Recognition in Keras
1 hour and 24 minutes
###### Project: Real-Time Object Detection with YOLOv3
1 hour and 5 minutes
###### Project: Image Super Resolution using Autoencoders in Keras
1 hour and 4 minutes
53 minutes
52 minutes
###### scikit-learn: Logistic Regression for Sentiment Analysis
1 hour and 26 minutes
45 minutes
58 minutes
57 minutes
###### Predictive Analytics for Business with H2O and R
1 hour and 6 minutes
47 minutes
43 minutes
###### Create Interactive Dashboards with Streamlit and Python
1 hour and 33 minutes
50 minutes
53 minutes
###### Project: Regression Analysis with Yellowbrick
1 hour and 10 minutes
55 minutes
###### Project: Named Entity Recognition using LSTMs with Keras
1 hour and 10 minutes
47 minutes
###### Build a Machine Learning Web App with Streamlit and Python
1 hour and 21 minutes
###### Logistic Regression with NumPy and Python
1 hour and 3 minutes
47 minutes
58 minutes
###### Explainable Machine Learning with LIME and H2O in R
1 hour and 24 minutes
###### Automatic Machine Learning with H2O AutoML
1 hour and 1 minute
58 minutes
###### Build a Data Science Web App with Streamlit and Python
1 hour and 21 minutes
###### Project: Support Vector Machines with scikit-learn
1 hour and 40 minutes
48 minutes
52 minutes
52 minutes
###### Project: Predict Employee Turnover with scikit-learn
1 hour and 4 minutes
###### Project: Predictive Modelling with Azure Machine Learning Studio
1 hour and 6 minutes
###### Project: Anomaly Detection in Time Series Data with Keras
1 hour and 3 minutes
###### Project: Generate Synthetic Images with DCGANs in Keras
1 hour and 7 minutes
###### Project: Perform Feature Analysis with Yellowbrick
1 hour and 15 minutes
52 minutes