Data enrichment is the process of combining first-party data from internal sources with disparate data from other internal systems or third-party data from external sources.
\nUsually, the data available from clients or stakeholders are not enough to solve the given problem statement, like if a client comes with a problem statement to build a recommendation engine for his mutual fund industry, the usual data they have is old purchase data but that's not enough as client behaviour changes with time and is impacted by the present market condition, oil prices, etc. which needs to be incorporated in the model to make it efficient.
\nCodes for this tutorial is at Link
\nThe whole process id divided into four steps:
\nI have implemented a full pipeline of data science from scrapping data from web to implementing ml and NLP classification.
\nHere I have scraped data from IMDB website (imdb.py)
\nI have tried to implement simple ML regression on the data (ml_imdb.py)
\nI have prepared the data for NLP classification (multilabel_prep.py)
\nI have implemented multilabel NLP classifier using various techniques like chain classifier etc. (multilabelnlpclassifier.ipynb)
\nWeb scraping is the process of extracting and parsing raw data from the web. Web scraping is a technique which helps data scientist to make their data-rich and is an efficient technique of data collection.
\nThis world is full of data, but unfortunately, most of them are not in the form to be used. Data is like crude oil, or we say it is in unstructured form. For a data scientist or engineer, our first challenge is to make the data model consumption ready, which takes the majority of the time, and this whole process is collectively known as data preprocessing.
\nHTML is a form of primary markup language and the base framework of mostly all websites. For performing web scraping its necessary to know it
\nHere we will start with requesting the web page using python package requests.
\n from requests import get\n url = 'http://www.imdb.com/search/title?release_date=2017&sort=num_votes,desc&page=1'\n response = get(url)\n print(len(response.text))The whole web page is now stored in the variable object response.\nThen we parse the web page using beautifulsoup package.
\n from bs4 import BeautifulSoup\n html_soup = BeautifulSoup(response.text, 'html.parser')\n type(html_soup)Then I will store all the div with the class named lister-item mode-advanced in variable movie_containers.
\nmovie_containers = html_soup.find_all('div', class_ = 'lister-item mode-advanced')Then I iterate through this object and store the information in lists to make my final DataFrame, using simple for loops.
\n# Lists to store the scraped data in\nnames = []\nyears = []\nimdb_ratings = []\nmetascores = []\nvotes = []\n#gross=[] #many movies have no record\nmovie_description=[]\nmovie_duration=[]\nmovie_genre=[]\n# Extract data from individual movie container\nfor container in movie_containers:\n# If the movie has Metascore, then extract:\n if container.find('div', class_ = 'ratings-metascore') is not None:\n# The name\n name = container.h3.a.text\n names.append(name)\n# The year\n year = container.h3.find('span', class_ = 'lister-item-year').text\n years.append(year)\n# The IMDB rating\n imdb = float(container.strong.text)\n imdb_ratings.append(imdb)\n# The Metascore\n m_score = container.find('span', class_ = 'metascore').text\n metascores.append(int(m_score))\n# The number of votes\n vote = container.find('span', attrs = {'name':'nv'})['data-value']\n votes.append(int(vote))\n# Gross income of movie\n #gross_inc =container.find_all('span', attrs = {'name':'nv'})[1]['data-value']\n #gross.append(gross_inc)\n\n# movie description\n movie_desc=container.find_all('p', class_ = 'text-muted')[1].text\n movie_description.append(movie_desc)\n movie_det=container.find_all('p', class_ = 'text-muted')[0]\n\n\n# Movie duration\n movie_dur=movie_det.find('span',class_='runtime').text\n movie_duration.append(movie_dur)\n\n# Movie genre\n movie_gen=movie_det.find('span',class_='genre').text\n movie_genre.append(movie_gen)\n\nimport pandas as pd\none_df = pd.DataFrame({'movie': names,\n'year': years,\n'imdb': imdb_ratings,\n'metascore': metascores,\n'votes': votes,\n#'gross':gross,\n'movie decription':movie_description,\n'movie duration':movie_duration,\n'movie genre':movie_genre\n})\nprint(one_df.info())\none_df.to_csv('50_movie_details.csv')But this was only for one page which has data for 50 movies only which is not enough to build a model.
\nPlease refer my code to understand how I use simple for loops to iterate through all the movies and downloading data for 20 years(approx).
\nWhats is linear regression??
\nIt is one of the most popular and used statistical techniques\n• Used to understand the relationship between variables
\n• Can also be used to predict a value of interest for new observations
\n• The aim is to predict the value of a continuous numeric variable of interest (known as the response or dependent or target variable)
\n• The values of one or more predictor (or independent) variables are used to make the prediction
\n• One predictor = simple regression
\n• More predictors = multiple regression
\nHere I just tried to use metascore of movies firstly to predict IMDB ratings and secondly I wanted to enhance it by using metascore and votes to predict IMDB rating.
\n## ML model\n\nX = data.loc[:, 'metascore'].values\ny = data.loc[:, 'imdb'].values\n\n# Splitting the dataset into the Training set and Test set\nfrom sklearn.cross_validation import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state = 0)\n\nfrom sklearn.linear_model import LinearRegression\n\nregressor = LinearRegression()#making object for reg package\nregressor.fit(X_train.reshape(-1,1), y_train.reshape(-1,1))#to fit the regressor to our training data\n\n#predict the test results\ny_pred =regressor.predict(X_test.reshape(-1,1))\n\nfrom sklearn.metrics import mean_squared_error\nmean_squared_error(y_test, y_pred)\n\n# 0.18041462828221905## Let try with imdb and votes\nX1 = data.loc[:, ['metascore','votes']].values\ny1 = data.loc[:, 'imdb'].values\n\n# Splitting the dataset into the Training set and Test set\nfrom sklearn.cross_validation import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X1, y1, test_size = 0.33, random_state = 0)\n\nfrom sklearn.linear_model import LinearRegression\n\nregressor = LinearRegression()#making object for reg package\nregressor.fit(X_train, y_train)#to fit the regressor to our training data\n\n#predict the test results\ny_pred =regressor.predict(X_test)\n\n\nfrom sklearn.metrics import mean_squared_error\nmean_squared_error(y_test, y_pred)\n# 0.15729132122310804 good scoreI tried to scrape data from the IMDB site and then applied ML regression techniques on it. Later I found that the movies listed are multi-class like Logan belongs to Action, Drama, Sci-Fi, which led me to think about how to implement the classifier model in the multilabel data. Usually, the data we get in real-world is mostly multi labelled like chatbot data; the intent is many and like these movies which are multi-class.
\nHere we will first see how we prep our data for multilabel classification.
\nHere we have all tags in one single column which is not usable while we do classification, so we have to make separate columns for all labels, and if the row doesn't belong to that category, it will be filled by 0 else 1.
\nimport os\nos.chdir('Desktop/web_scraping/imdb scrapper_ml/')\n\nimport pandas as pd\n\ndata=pd.read_csv('multilabel_nlp_classification.csv')\n\nmovie_list=[x for x in data['movie genre']]\nmovie_list1=''\nfor x in data['movie genre']:\n movie_list1+=','+x\n\nli_m=movie_list1.split(',')\nli=[x.strip() for x in li_m]\nlist_s=list(set(li))\n\nfor x in list_s:\n data[x]=0\n\ndata['movie_genre']=[x.strip().split(',') for x in data['movie genre']]\n\nde=data.copy()\n#data.loc[0,'Action']=1\nde['id']=range(0,6116)\n#print(de.loc[de['id']==0,'Action'])\nfor i in range(0,6116):\n for x in de.loc[de['id']==i,'movie_genre']:\n for y in x:\n y=y.strip()\n de.loc[de['id']==i,y]=1\n\nde.to_csv('multilabel_nlp_classification.csv')Now, as our data is ready, we can start with NLP implementation.
\nFor multilabel classification, I used techniques like classifier chain, label powerset, etc.
\nHere the problem statement is that using the movie description our model has to guess which genre the movie belongs to. It is a popular use case. Take an example of ecommerce product description data; now instead of manually assigning the labels to it, we can use a model which will find relevant labels or genre for it and make the content relevant to the type it belongs.
\nI start with Exploratory data analysis and then data cleaning, which is the most crucial step as if all the description has some very 30-50 common words it will simply make the data-heavy and model slow and inefficient.
\nThen we go on to make the data model ready as ML models don't understand text data we have to feed numbers in it. For that purpose, we use TfidfVectorizer.
\nTfidfVectorizer - Transforms text to feature vectors that can be used as input to the estimator.
\nThen simply diving the data in train and test split.
\nx_train = vectorizer.transform(train_text)\ny_train = train.drop(labels = ['id','movie decription'], axis=1)\n\nx_test = vectorizer.transform(test_text)\ny_test = test.drop(labels = ['id','movie decription'], axis=1)I tried first with applying logistic regression and one vs rest classifier.
\nOneVsRestClassifier strategy splits a multi-class classification into one binary classification problem per class.\nOneVsRestClassifier is when we want to do multi-class or multilabel classification, and its strategy consists of fitting one classifier per class. For each classifier, the class is fitted against all the other classes.
\n# Using pipeline for applying logistic regression and one vs rest classifier\nLogReg_pipeline = Pipeline([\n ('clf', OneVsRestClassifier(LogisticRegression(solver='sag'), n_jobs=-1)),\n ])\n\nfor category in categories:\n printmd('**Processing {} comments...**'.format(category))\n \n # Training logistic regression model on train data\n LogReg_pipeline.fit(x_train, train[category])\n \n # calculating test accuracy\n prediction = LogReg_pipeline.predict(x_test)\n print('Test accuracy is {}'.format(accuracy_score(test[category], prediction)))\n print("\\n")Next, I tried with BinaryRelevance
\nIt is a simple technique which treats each label as a separate single class classification problem.
\n# using binary relevance\nfrom skmultilearn.problem_transform import BinaryRelevance\nfrom sklearn.naive_bayes import GaussianNB\n\n# initialize binary relevance multi-label classifier\n# with a gaussian naive bayes base classifier\nclassifier = BinaryRelevance(GaussianNB())\n\n# train\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)Next, I tried using ClassifierChain.
\nIt is almost similar to BinaryRelevance, here the first classifier is trained just on the input data, and then each next classifier is trained on the input space and all the previous classifiers in the chain.
\nfrom skmultilearn.problem_transform import ClassifierChain\nfrom sklearn.linear_model import LogisticRegression\n\n\n# initialize classifier chains multi-label classifier\nclassifier = ClassifierChain(LogisticRegression())\n\n# Training logistic regression model on train data\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)Next, I tried using Label Powerset.
\nHere we transform the problem into a multi-class problem with one multi-class classifier is trained on all unique label combinations found in the training data.
\nfrom skmultilearn.problem_transform import LabelPowerset\n# initialize label powerset multi-label classifier\nclassifier = LabelPowerset(LogisticRegression())\n\n# train\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)Please refer my notebook multilabelnlpclassifier.ipynb from my repo for more details.
\n