Practical-5

 Aim: Data Pre-processing and text analytics using Orange.

Theory:

Text Analytics

Text analytics is the automated process of translating large volumes of unstructured text into quantitative data to uncover insights, trends, and patterns.

Sentiment Analysis

Sentiment Analysis is the process of determining whether a piece of writing is positive, negative or neutral. A sentiment analysis system for text analysis combines natural language processing (NLP) and machine learning techniques to assign weighted sentiment scores to the entities, topics, themes and categories within a sentence or phrase.

WHY?

Sentiment analysis is extremely useful in social media monitoring as it allows us to gain an overview of the wider public opinion behind certain topics. Sentiment analysis is useful for quickly gaining insights using large volumes of text.

Preprocessing is a key component in Data Science. Orange tool has various ways to achieve the activities.

Discretization: It is the process of transferring continuous functions, models, variables, and equations into discrete counterparts. This process is usually carried out as a first step toward making them suitable for numerical evaluation and implementation data by the models.

Discretization replaces continuous features with the corresponding categorical features:

Sample Code:

import Orange

store = Orange.data.Table("superstore.tab")
disc = Orange.preprocess.Discretize()
disc.method = Orange.preprocess.discretize.EqualFreq(n=3)
d_store = disc(store)

print("Original dataset:")
for e in store[:3]:
    print(e)

print("Discretized dataset:")
for e in d_store[:3]:
    print(e)

Continuization: 

Given a data table, return a new table in which the discretize attributes are replaced with continuous or removed.

• binary variables are transformed into 0.0/1.0 or -1.0/1.0 indicator variables, depending upon the argument zero_based.
• multinomial variables are treated according to the argument multinomial_treatment.
• discrete attribute with only one possible value are removed

Sample Code:

import Orange
products = Orange.data.Table("Products")
continuizer = Orange.preprocess.Continuize()
products1 = continuizer(products)

Normalization: It is a systematic approach of decomposing tables to eliminate data redundancy(repetition) and undesirable characteristics like Insertion, Update and Deletion Anomalies.

Sample Code:

>>> from Orange.data import Table
>>> from Orange.preprocess import Normalize
>>> data = Table("Customers")
>>> normalizer = Normalize(norm_type=Normalize.NormalizeBySpan)
>>> normalized_data = normalizer(data)

Randomization: A method based on chance alone by which study participants are assigned to a treatment group. Randomization minimizes the differences among groups by equally distributing people with particular characteristics among all the trial arms.

Sample Code:

>>> from Orange.data import Table
>>> from Orange.preprocess import Randomize
>>> data = Table("Returns")
>>> randomizer = Randomize(Randomize.RandomizeClasses)
>>> randomized_data = randomizer(data)