Data Science

  What is Text Analysis?  Text analysis is the process of automatically classifying and extracting meaningful information from unstructured text. It involves detecting and interpreting trends and patterns to obtain relevant insights from data in just seconds. Text Analysis - Transition from Structured Data to Unstructured Data Text analysis is usually performed on the text that is present inside the document in a  Unstructured Format,  unlike supervised or unsupervised learning as discussed earlier. Thus the first task of Text Analysis comes out to be conversion of this  Unstructured Data  to  Structured Data  i.e to represent the data in the tabular form and this process is known as the  Text Vectorisation  (Discussed Later in the Post). Understanding the Document As text analysis involves extraction of information from the document it becomes important to understand the basic  characteristics  of the document that are to be f...

  Hierarchical Clustering Create a nested series of partitions represented in the form of a dendogram  Shows how objects are grouped together step by step  Typical stopping criteria  Number of clusters  Minimum distance between clusters being greater than a user defined threshold Types of Hierarchical Clustering Agglomerative  Starts by assuming that each object is a separate cluster  Successively merges clusters until some stopping criterion is met. For example, until the number of clusters is 1    Divisive Starts by assuming the existence of only one cluster  Successively splits a cluster into two or more sub-clusters Single-Link Distance between two clusters is defined as the minimum distance between pairs of objects, one from each cluster  Can result in a chaining effect  C2 is closer to C1  Builds minimum spanning tree  Edge only exist between “closest” nodes  Complete-Link  Distance between to clu...

  BIRCH :  BIRCH (balanced iterative reducing and clustering using hierarchies) is an unsupervised data mining algorithm used to perform hierarchical clustering over particularly large data-sets. An advantage of BIRCH is its ability to incrementally and dynamically cluster incoming, multi-dimensional metric data points in an attempt to produce the best quality clustering for a given set of resources (memory and time constraints). In most cases, BIRCH only requires a single scan of the database.  Algorithm:   algorithm takes as input a set of  N  data points, represented as real-valued vectors, and a desired number of clusters  K . It operates in four phases, the second of which is optional. The first phase builds a clustering feature ( {\displaystyle CF} ) tree out of the data points, a height-balanced tree data structure, defined as follows: Given a set of N d-dimensional data points, the...

  K-means : In Continuation to previous post, the complexity of K means is O(tkn) where t is  the no. of the iterations, k is no. of clusters and n is no. of objects. When k value is large it may become inefficient but is efficient than many other algorithms. It can lead to local optimum solution for various random k values. By using deterministic annealing and genetic algorithm can lead to good local optimum. Weakness of K means :  Need to specify k, number of clusters in advance. Applicable only when mean is defined. Not suitable to discover clusters with non-convex shapes. Choosing Initial Centroid : Choose one centroid. Pick point furthest from existing centroids as centroid. In this, we don't know when to stop. Hierarchical Clustering: Here, we will have information loss due to its height or height increases loss of information increases. Here, choose a sample of data, perform hierarchical clustering, extract k - cluster, use centroid as initial centroids for k-means...