Data Scrapping

 Monday 31 August was a guest lecture at Sabudh Foundation where we were introduced to the concept of Data Scraping.

What is Data Scraping?

Data scraping, also known as web scraping, is the process of importing information from a website into a csv or any file on local. It’s one of the most efficient ways to get data from the web, and in some cases to channel that data to another website. 


Popular uses of data scraping include:

  • Research for web content/business intelligence
  • Pricing for travel booking sites or price comparison sites
  • Finding sales leads or conducting market research by crawling public data sources (e.g. Yell and Twitter)
  • Sending product data from an e-commerce site to another online vendor (e.g. Google Shopping)

Scraping using Python


There are numerous number of packages available for web scraping in python but we only need a handful of them in order to scrape almost any site. Some of these libraries are are named below:

  • Requests
  • BeautifulSoup
  • lxml
  • Selenium
  • Scrapy
It is to be not every package mentioned above is required for Scraping the web, Request is the one which is always required while scraping whereas the others depend on the use case of the project.

Note: In this lecture we were provided with the idea of Beautiful Soup

The Request Module

The Requests Module is a simple yet powerful HTTP module, which means that one can use it to access web pages. It the library that tells how one can communicate with websites.

Its simplicity is definitely its greatest strength and one can easily use that without reading documentation. 

For example, if we want to pull down the contents of a page, it can be done as:

import requests
page = requests.get('http://examplesite.com')
contents = page.content

The variable content will hold the raw html of the website given to the get method of request library.

Other than this the request module can access API’s, post to forms, and can do various other stuff.

Beautiful Soup

Beautiful Soup (BS4) is a parsing library that can use different parsers. A parser is simply a program that can extract data from HTML and XML documents.

One advantage of BS4 is its ability to automatically detect encodings, this allows it to gracefully handle HTML documents with special characters. In addition to it BS4 can help us to  navigate a parsed document and find us what we need. This makes it quick and painless to build common applications.

For example, if we wanted to find all the links in the web page we fetched earlier (using requests),  this can be done as

from bs4 import BeautifulSoup
soup = BeautifulSoup(contents, 'html.parser')
soup.find_all('a')


soup is the object of BeautifulSoup Class which takes in the html of the website and parser to go through the html.

Once the object is created we use find_all() method to get all the anchor tags ('a') which returns the list of all the

links present in the web page.

Methods From Beautiful Soup

text

The Beautiful Soup object has a text attribute that returns the plain text of a HTML. Given our simple soup of <p> Hello world </p> the text attribute returns Hello World.

find()

find() method is used get the first instance of any tag of html like anchor tag, paragraph tag etc., after getting the first instance of the particular tag one can get the text using text attribute of Beautiful Soup.

attrs

attrs is like the text in Beautiful Soup which helps us to get the values present in the attributes of various tags. For Example we can get the src attribute of the image tag,  href value of the anchor tag.


Comments

Post a Comment

Popular posts from this blog

Supervised Learning(Part-5)

Image
  Decision Trees (Continued) Attribute Selection Measures If the dataset consists of N attributes then deciding which attribute to place at the root or at different levels of the tree as internal nodes is a complicated step. By just randomly selecting any node to be the root can’t solve the issue. If we follow a random approach, it may give us bad results with low accuracy. For solving this attribute selection problem, researchers worked and devised some solutions. They suggested using some criteria like : Entropy, Information gain, Gini index, Gain Ratio, Reduction in Variance Chi-Square These criteria will calculate values for every attribute. The values are sorted, and attributes are placed in the tree by following the order i.e, the attribute with a high value(in case of information gain) is placed at the root. While using Information Gain as a criterion, we assume attributes to be categorical, and for the Gini index, attributes are assumed to be continuous. Entropy Entropy is a me
Read more

Supervised Learning(Part-2)

Image
  Support Vector Machine A Support Vector Machine (SVM) is a discriminative classifier formally defined by a separating hyperplane. In other words, given labeled training data, the algorithm outputs an optimal hyperplane which categorizes new examples.  In two dimensional space this hyperplane is a line dividing a plane in two parts where in each class lay in either side. Suppose given plot of two label classes on graph as shown in image. You might have come up with something similar to following image (image B). It fairly separates the two classes. Any point that is left of line falls into black circle class and on right falls into blue square   In real world application, finding perfect class for millions of training data set takes lot of time. This is called regularization parameter.  Two terms regularization parameter and gamma are tuning parameters in SVM classifier. Varying those we can achieve considerable non linear classification line with more accuracy in reasonable amount of
Read more

Text Analysis (Part - 4)

Image
  The article covers a very important concepts of Text Analysis from Bayesian's i.e the  Latent Dirichlet Allocation  which is the part of topic modelling . What is Topic Modelling? Topic modelling is a type of statistical modelling for discovering the abstract “topics” that occur in a collection of documents. This can be useful for search engines, customer service automation, and any other instance where knowing the topics of documents is important . Thus, Topic Models , in a nutshell, are a type of statistical language models used for uncovering hidden structure in a collection of texts. There are multiple methods of for topic modelling and one of them includes Latent Dirichlet Allocation (LDA) . Latent Dirichlet Allocation Latent Dirichlet Allocation (LDA) is an example of topic model and is used to classify text in a document to a particular topic. It builds a topic per document model and words per topic model, modelled as Dirichlet distributions. Basic Idea LDA is a  genera
Read more