![\"Streamlit](\"/9202979fe96eb015f5209a8591dbc320/streamlit-1.webp\" "\\"Streamlit")
In this tutorial, you'll learn how to easily convert an OpenCV project into a web app that you can showcase. You'll use a library called Streamlit, which helps you easily build a web user interface in Python. Yes, you heard it right — no HTML, CSS, or JavaScript required. Just Python!
\nLet's get started.
\nInstall OpenCV and Streamlit using pip. You would also need Pillow — another image library.
\npip install opencv-python streamlit Pillow\n\nIf you use Python 2.7, note that OpenCV doesn't directly support Python 2.7 in their latest versions. Accordingly, you have to specify an older version like this
\npip install opencv-python==4.2.0.32
import cv2\n\ndef brighten_image(image, amount):\n img_bright = cv2.convertScaleAbs(image, beta=amount)\n return img_bright\n\ndef blur_image(image, amount):\n img = cv2.cvtColor(image, 1)\n blur_img = cv2.GaussianBlur(img, (11, 11), amount)\n return blur_img\n\ndef enhance_details(img):\n hdr = cv2.detailEnhance(img, sigma_s=12, sigma_r=0.15)\n return hdr\n\nimg = cv2.imread(filename='tony_stark.webp')\n\n# do some cool image processing stuff\nimg = enhance_details(img)\nimg = brighten_image(img, amount=25)\nimg = blur_image(img, amount=0.2)\n\ncv2.imshow('Tony Stark', img)\ncv2.waitKey(0)\ncv2.destroyAllWindows()Here are three image processing functions that accept an image, do some processing, and return the processed image.
\nbrighten_image — increases the brightness of the image.blur_image — applies a blur effect on the image.enhance_details — applies an effect to enhance the details of the image.These functions make use of the OpenCV functions to do the actual processing — for e.g.: cv2.GaussianBlur, etc. I'm not explaining in-depth about them and the various parameters they accept since this tutorial is more focused on integrating OpenCV with Streamlit. However, feel free to jump to the OpenCV documentation to know more details about them.
\nThis program reads the image from the filepath using cv2.imread(); then, it passes the image to these functions that do the processing. Finally, the image is displayed using cv2.imshow(). cv2.waitKey(0) is to wait till the user presses any key, after which the program is exited.
Streamlit offers some common UI components out of the box that you can place on your webpage. This makes it super easy to code up something real quick. The way Streamlit works is that it reruns the Python script every time a user interacts with the components. They have some caching and optimizations, but this simple design makes it easy to build interactive webpages using Streamlit.
\n\n\nSomeone said, “Talk is cheap. Show me the code.” So, let's see some code.
\n
Open an editor and copy-paste this to demo-app.py
import streamlit as st\n\nst.title("OpenCV Demo App")\nst.subheader("This app allows you to play with Image filters!")\nst.text("We use OpenCV and Streamlit for this demo")\nif st.checkbox("Main Checkbox"):\n st.text("Check Box Active")\n\nslider_value = st.slider("Slider", min_value=0.5, max_value=3.5)\nst.text(f"Slider value is {slider_value}")\n\nst.sidebar.text("text on side panel")\nst.sidebar.checkbox("Side Panel Checkbox")To start a streamlit app, simply run the command streamlit run with the filename — for example:
streamlit run demo-app.pyYou should get a similar output as follows.
\n You can now view your Streamlit app in your browser.\n\n Local URL: http://localhost:8501\n Network URL: http://192.168.1.8:8501Click on the links in the output to open the Streamlit app in your browser. You'll see something as follows.
\n
If you see the code, it's very straightforward. It imports streamlit as st. The default is a simple linear layout where you can place components on the webpage in a sequential manner.
\nFor example, st.title() , st.checkbox(), st.slider() places the components on the main page in the order they're called..
Streamlit also offers a side panel. In order to place components in the sidepanel, you can do as follows:
\nst.sidebar.title() , st.sidebar.checkbox(), st.sidebar.slider()
There are other components also apart from these. You can explore more in the docs.
\nLet's integrate your OpenCV program into Streamlit. Here is the complete code:
\nimport cv2\nimport streamlit as st\nimport numpy as np\nfrom PIL import Image\n\n\ndef brighten_image(image, amount):\n img_bright = cv2.convertScaleAbs(image, beta=amount)\n return img_bright\n\n\ndef blur_image(image, amount):\n blur_img = cv2.GaussianBlur(image, (11, 11), amount)\n return blur_img\n\n\ndef enhance_details(img):\n hdr = cv2.detailEnhance(img, sigma_s=12, sigma_r=0.15)\n return hdr\n\n\ndef main_loop():\n st.title("OpenCV Demo App")\n st.subheader("This app allows you to play with Image filters!")\n st.text("We use OpenCV and Streamlit for this demo")\n\n blur_rate = st.sidebar.slider("Blurring", min_value=0.5, max_value=3.5)\n brightness_amount = st.sidebar.slider("Brightness", min_value=-50, max_value=50, value=0)\n apply_enhancement_filter = st.sidebar.checkbox('Enhance Details')\n\n image_file = st.file_uploader("Upload Your Image", type=['jpg', 'png', 'jpeg'])\n if not image_file:\n return None\n\n original_image = Image.open(image_file)\n original_image = np.array(original_image)\n\n processed_image = blur_image(original_image, blur_rate)\n processed_image = brighten_image(processed_image, brightness_amount)\n\n if apply_enhancement_filter:\n processed_image = enhance_details(processed_image)\n\n st.text("Original Image vs Processed Image")\n st.image([original_image, processed_image])\n\n\nif __name__ == '__main__':\n main_loop()Let's understand what's happening here.
\nApart from the image processing functions, you have a 'main_loop' function to add the logic for the webpage.
\nNothing fancy about st.title(), st.subheader(), st.text() — they just print some text in different sizes.
Next, there are two sliders to get the amount by which you need to apply the blur and brightness filters.
\nNote that st.sidebar places these components in the sidebar.
slider() takes in some arguments: name of the slider, min value, max value, and the default value of the slider. This function returns the current value of the slider.
Next, there's a checkbox component.
\ncheckbox() returns True if the checkbox is checked; else, it would return False.
Next, there's a file_uploader component, through which users can upload files of different types specified by the type parameter. Here, it's restricted to the image file types needed for the demo usecase.
When the program starts, there are no files selected by the user. At this time, this component returns None. When a file is uploaded from the UI, this component returns the path of the file.
\nThis is why there's an if-check on the return value of this component. If no files are selected, you can skip the rest of the program by returning from the main_loop() function.
Remember that the entire program is rerun whenever there's user interaction on any components of the page. When a user uploads a file, the whole program is executed again — and the if-check fails so that the program executes the image processing logic.
\nPillow.Image() is used to open this file, then it's converted to a numpy array using np.array() so that OpenCV can process it.
Now, it's passed to the different processing functions along with the amount parameter.
Finally, the program displays the original and processed images using the st.image() component.
The web app is ready!
\nTo start the app, simply run:
\nstreamlit run demo-app.pyNow you can play with the filters. Of course, these are some basic filters, but you can extend it to more interesting filters — like cartoonify filters, etc. — using the rich features of OpenCV.
\n![\"Streamlit](\"/8a3adfc96648a8ef90ba512e93ba1648/streamlit-final.webp\" "\\"Streamlit")
To publish your app, you need to host this program somewhere. There are different hosting providers for complex projects. But for this hobby project, is there a simple and quick solution?
\nYes, there's one!
\nYou can host the Streamlit application in Streamlit Cloud for free. You can host up to 3 apps in an account for free with up to 1GB of memory.
\nCheck out this tutorial from the community to host your app in Streamlit Cloud.
\nYou can find the source code for this tutorial here. The web app discussed above is hosted here if you want to try it out.
\nWant to add user authentication and registration in your app quickly? Check out the LoginRadius Python integration.
\n","frontmatter":{"date":"November 10, 2021","updated_date":null,"title":"How to Build an OpenCV Web App with Streamlit","tags":["OpenCV","Streamlit","Python"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/818e31111fb212b94c72ff9edb09bd3f/58556/coverimage.webp","srcSet":"/static/818e31111fb212b94c72ff9edb09bd3f/61e93/coverimage.webp 200w,\n/static/818e31111fb212b94c72ff9edb09bd3f/1f5c5/coverimage.webp 400w,\n/static/818e31111fb212b94c72ff9edb09bd3f/58556/coverimage.webp 800w,\n/static/818e31111fb212b94c72ff9edb09bd3f/99238/coverimage.webp 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Dingu Sagar","github":"dingusagar","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/guest-post/user-authentication-in-python/"},"html":"Authentication and user identity management are challenging tasks you are bound to run into when building applications. For example, you will need to create profiles for users, validate provided passwords, implement a password reset functionalities, manage user sessions (sometimes on multiple devices), manage social media authentication, and many others.
\nYou still have to work on other parts of your application, and you might not have a lot of time. A lot of developers might hack their way through authentication, but that could lead to improper implementations. It is not advisable to do this as you can create doorways for cyber-related attacks in your application.
\nIn this tutorial, you will learn how to properly implement user authentication and identity management in a Flask application.
\n\n\nHere for the code alone? Head over to the implementation section of this article or visit this GitHub gist to browse demo code.
\n
User authentication is the process of validating a person’s identity to ascertain that they are who they claim to be. Authentication is achievable using passwords, one-time pins (OTP), biometrics, authentication apps, access tokens, certificates, and many more.
\nUser identity is an entity used to identify a user of an application uniquely. Forms of user identifiers include full names, email addresses, system-generated values, and UUIDs.
\nAn identity provider is a system that helps create, maintain, and manage user identity information. It also provides authentication services to external applications to ease their authentication flow and make it seamless.
\nWhen referring to authentication in Python, we talk about user authentication concerning web applications built with it. Python is actively used in making web applications with many supporting frameworks, including but not limited to Flask, Django, FastAPI, Bottle, and Hug.
\nEvery web application built with Python at one point or another would need to implement user authentication features. This article will cover implementing authentication and proper handling of user identity information using LoginRadius and Flask.
\nLoginRadius is a cloud-based consumer identity and access management (CIAM) platform that allows seamless user authentication and SSO integration into your application. LoginRadius is simple to use, completely secure, and highly customizable.
\nTo proceed with this tutorial, you will need an account with LoginRadius. If you have not created one before now, create one on the LoginRadius website.
\nLogin to your LoginRadius dashboard, then navigate to the app you want to integrate with Python (LoginRadius will set up a free app for you when you create an account).
\n![\"LoginRadius](\"/9f9fcb0430b89fa5c0ab46252144e277/pw6s1mqnn-yrtard7nbx.webp\")
Next, head over to the Configuration tab on the LoginRadius sidebar (left side of the screen).
![\"LoginRadius](\"/ea9ed1468469fcdba3d8c0fdebf2c088/i_alrgdnugpmtschkuuj.webp\")
Your API credentials are located under the API Key And Secret section. Once you have retrieved this, copy the APP Name, API Key, and API Secret and store them somewhere secure and easily retrievable.
![\"LoginRadius](\"/1d658bc9362be0fd7ed1a643af35012e/7lev6yc_ebtxcg62wrbe.webp\")
LoginRadius requires you to whitelist domains you will be integrating with your app. To whitelist, a domain, scroll down to the Whitelist Your Domain section in the Configuration tab of your app dashboard and add it.
![\"Domain](\"/689af03e0b523f8f87a86a4f38ffe91a/5ng50vbosmuhdhfuz-gi.webp\")
\n\nBy default, LoginRadius whitelists your local computer (localhost).
\n
We need to install the LoginRadius Python SDK. It provides functionalities that allow Python programs to communicate with LoginRadius APIs.
\nIn the terminal, type:
\npip install LoginRadius-v2 requests cryptography pbkdf2First, we need to install the Flask framework from PyPI. In the terminal, type:
\npip install flaskAfter that, create a file named server.py and save the following code in it:
from flask import *\n\napp = Flask(__name__)\napp.config["SECRET_KEY"] = "SECRET_KEY"\n\n\n@app.route("/")\ndef index():\n return "Hello World!"\n\n\nif __name__ == "__main__":\n app.run(debug=True)When you run the server.py script and open your browser, you will get a response similar to the image below:
![\"Hello](\"/b5ad3d364b16b60f9e8630be44e3cf84/vordrrnvz-vekwuickak.webp\")
Update the server.py file with the code below:
from LoginRadius import LoginRadius as LR\n\nLR.API_KEY = "API Key"\nLR.API_SECRET = "API Secret"\nloginradius = LR()Replace the values of the API_KEY and API_SECRET variables with your LoginRadius application keys we saved earlier.
To register users, you have to redirect them from your application to your LoginRadius Auth Page (IDX). Each LoginRadius app has a custom IDX. You can access it with the following URL pattern.
\nhttps://{APP_NAME}.hub.loginradius.com/auth.aspx?action={AUTH_ACTION}&return_url={RETURN_URL}APP_NAME parameter refers to your LoginRadius app name, which you can retrieve from the API Key And Secret section in the Configuration tab of your dashboard.AUTH_ACTION parameter refers to the authentication action you’re attempting to perform. It is either register or login.RETURN_URL parameter refers to the URL LoginRadius should redirect your users to after successful authentication. It is usually a route on your application server.Update the server.py file with the code below:
LR_AUTH_PAGE = "https://<APP_NAME>.hub.loginradius.com/auth.aspx?action={}&return_url={}"\n\n@app.route("/register/")\ndef register():\n # redirect the user to our LoginRadius register URL\n return redirect(LR_AUTH_PAGE.format("register", request.host_url))In the code above, we created a register route that redirects users to our LoginRadius registration IDX. We also set our AUTH_ACTION to “register” and our RETURN_URL to our application home page.
![\"LoginRadius](\"/79192f0d84a9ebca455ede5522497ab9/smn-8jr5ahgmhtynezje.webp\")
\n\nNOTE: Don’t forget to replace the <APP_NAME> placeholder with your LoginRadius app name we saved earlier.
\n
To authenticate registered users, you have to redirect them to your IDX page, passing “login” as the AUTH_ACTION.
Update the server.py file with the code below:
@app.route("/login/")\ndef login():\n access_token = request.args.get("token")\n if access_token is None:\n # redirect the user to our LoginRadius login URL if no access token is provided\n return redirect(LR_AUTH_PAGE.format("login", request.base_url))\n\n return "You have successfully logged in!"\n\nWhen LoginRadius successfully authenticates a user, it attaches a
\ntokenparameter to theREDIRECT_URLbefore redirecting your user there. This parameter contains the access token of the user that we authenticated.
In the code above, we redirect users to our LoginRadius login IDX if the token parameter is absent (this means LoginRadius did not redirect the user here). We also set our AUTH_ACTION to “login” and our RETURN_URL to our login page.
![\"LoginRadius](\"/8624b9e29ef548d70f975ecf65a31f92/7gs3xz6qhxhas7qgtjma.webp\")
![\"LoggedIn\"](\"/30d20a60f432e9415ebaf7645af42f80/xwzztcogdtmnyfq4j5nc.webp\")
We also want to fetch user profiles from the access token given by LoginRadius. It comes in handy when we want to verify if a given access token is valid (or has expired) or just fetch information about the current user.
Update the login route with the code below. We also added a dashboard route where we will redirect users after successful authentication.
@app.route("/login/")\ndef login():\n access_token = request.args.get("token")\n if access_token is None:\n # redirect the user to our LoginRadius login URL if no access token is provided\n return redirect(LR_AUTH_PAGE.format("login", request.base_url))\n\n # fetch the user profile details with their access tokens\n result = loginradius.authentication.get_profile_by_access_token(\n access_token)\n\n if result.get("ErrorCode") is not None:\n # redirect the user to our login URL if there was an error\n return redirect(url_for("login"))\n\n session["user_acccess_token"] = access_token\n\n return redirect(url_for("dashboard"))\n\n\n@app.route("/dashboard/")\ndef dashboard():\n return "You have successfully logged in!"In the code above, we used the authentication.get_profile_by_access_token method from the LoginRadius SDK to fetch our user’s details. If the request was successful and the result does not contain an ErrorCode parameter, we save the access token in the user’s session and redirect them to the dashboard route. But if an error occurs somewhere, e.g., the access token is invalid/expired, we redirect the user back to the login route.
![\"LoggedIn\"](\"/d15eed00435b16b1d05cbb201630c333/f2p7ddnwin3yihucx2em.webp\")
Next, we want to add more functionality to the dashboard route. Instead of just displaying a dummy text, let it show the user information we fetched earlier. Update the dashboard route with the code below:
@app.route("/dashboard/")\ndef dashboard():\n access_token = session.get("user_acccess_token")\n if access_token is None:\n return redirect(url_for("login"))\n\n # fetch the user profile details with their access tokens\n result = loginradius.authentication.get_profile_by_access_token(\n access_token)\n\n if result.get("ErrorCode") is not None:\n # redirect the user to our login URL if there was an error\n return redirect(url_for("login"))\n\n return jsonify(result)Here, we fetched the access token stored in the user’s session earlier, used it to get their details, and rendered the result.
\n![\"LoggedIn\"](\"/ed807cbc18a2c163fbbce57eb7789e5c/1zsvbg3rk013zlbpxx2u.webp\")
Invalidating access tokens means rendering particular access tokens useless and unusable. It comes in handy when we log out users. The LoginRadius SDK provides an auth_in_validate_access_token method that takes in an access token to be invalidated.
To add this to our server, create a logout route with the code below:
@app.route("/logout/")\ndef logout():\n access_token = session.get("user_acccess_token")\n if access_token is None:\n return redirect(url_for("login"))\n\n # invalidate the access token with LoginRadius API\n loginradius.authentication.auth_in_validate_access_token(access_token)\n session.clear()\n\n return "You have successfully logged out!"![\"Log](\"/bc9fb664300826a7397cfb8826284223/xyvodwpjtxjrjgxoffx5.webp\")
This article taught us about user authentication, user identity management, and implementing it correctly. In addition, we saw how easy it is to integrate LoginRadius services into a Python application to ease the implementation of authentication and user identity management.
\nThe source code of the demo application is available as a GitHub gist. You can learn more about the LoginRadius Python SDK features from the official documentation.
\n","frontmatter":{"date":"July 07, 2021","updated_date":null,"title":"Implementing User Authentication in a Python Application","tags":["Python","Authentication","Flask"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/2005b7a832c4ee0e2301194e9f8eb64b/58556/coverImage.webp","srcSet":"/static/2005b7a832c4ee0e2301194e9f8eb64b/61e93/coverImage.webp 200w,\n/static/2005b7a832c4ee0e2301194e9f8eb64b/1f5c5/coverImage.webp 400w,\n/static/2005b7a832c4ee0e2301194e9f8eb64b/58556/coverImage.webp 800w,\n/static/2005b7a832c4ee0e2301194e9f8eb64b/99238/coverImage.webp 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Solomon Esenyi","github":"LordGhostX","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/python-basics-in-minutes/"},"html":"Python is a general-purpose programming language and has overtaken Java in popularity according to a recent Stackoverflow survey.
\nPeople who have never programmed before are tempted to try due to the simplicity to learn and use it. Well, let's get down to business.
\nBefore we get started, there are a few things you should know about python:
\nWe are assuming that you have already installed python. If you do not, please click here.
\nTo write your first Python code, open your text editor and type:
\nprint("Hello world, this is my first python code")Save the file as helloworld.py and put it into the python interpreter to be executed.\nYou also can run your code on the command line:
C:\\Users\\Your Name>python helloworld.pyIf everything went well, you should see something like this:
\n> Hello world, this is my first python codeYou can also use the print function to show integers, variables, lists, etc. Try it!
\n#show the sum of two integers\nnumber1 = 5\nnumber2 = 7\nprint(number1+number2)#dividing two decimals\nnumber1 = float(12.1)\nnumber2 = float(4)#concatenating strings\nname = "Python"\nphrase = str("I hate ")\nprint("I love ", name)\nprint(phrase+name)Python has the following data types built-in by default, in these categories:
\nstr (string)int (integer), float (decimal), complexlist, tuple, rangedictset, frozensetboolbytes, bytearray, memoryview\n\nNote: Variables in Python are interpreted as integers by default. Is a good practice to declare them as another type explicitly (if they aren't integers). You can see the kind of a variable using the function
\ntype().
Operators are used to performing operations on variables and values.\nThe main groups are:
\nArithmetic operators are used with numeric values to perform common mathematical operations.
\nx + yx - yx * yx / yx % y\tx ** y\tx // yComparison operators are used to compare two values.
\nx == yx != yx > yx < y\tx >= yx <= yLogical operators are used to combine conditional statements.
\nx < 2 and x < 4x < 10 or x < 9not(x < 2 and x < 4)Look to the code below:
\nage = 18\nif (age<21):\n print("The student is underage!")\nelif (age==21):\n print("The student is 21!")\nelse:\n print("The student isn't underage!")In programming, we often have to choose what to do depending on the situation. It is essential to know how to use conditional arguments like ifand else.\nThe code above print a different message according to a condition.
Try to write a code that asks for two test scores. If the average is less than 7, the user should see \"I'm sorry, you didn't do well on the tests\". If the average is exactly 7, the user should see \"You did it!\". And if it is greater than 7, the user should see \"Congratulations!! You're a great student.\".
\n\n\nNote: to request a response from the user, you need to use the
\ninput()function. For example:
name = str(input("What's your name?"))\nage = int(input("How old are you? "))\nprint("I'm %s and I'm %d"% (name, age))Python has two primitive loop commands: while and for.
With the while loop, we can execute a set of statements as long as a condition is true.
\ncount = 0\nwhile count < 10:\n print(count)\n count += 1 #this line is the same as count = count+1The code above will print count as long as count is less than 10.
\nA for loop is used for iterating over a sequence. Using it, we can execute a set of statements, once for each item in a list, tuple, set, string, etc. For example:
\n#loop through a string\nfor x in "banana":\n print(x)#loop through a list of fruits\nfruits = ["apple", "banana", "melon"]\nfor x in fruits:\n print("I like", x)Did you understand why python became so popular? In a few minutes, you were able to learn the main concepts of this fantastic language.
\nPlease comment and share this article!
\n","frontmatter":{"date":"November 06, 2020","updated_date":null,"title":"Python basics in minutes","tags":["Python"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/e65362be2b3196aeea5b79ea97dab3a6/58556/python.webp","srcSet":"/static/e65362be2b3196aeea5b79ea97dab3a6/61e93/python.webp 200w,\n/static/e65362be2b3196aeea5b79ea97dab3a6/1f5c5/python.webp 400w,\n/static/e65362be2b3196aeea5b79ea97dab3a6/58556/python.webp 800w,\n/static/e65362be2b3196aeea5b79ea97dab3a6/99238/python.webp 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Sara Lins","github":"saranicoly","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/beginners-guide-to-tweepy/"},"html":"As we all know, Twitter is one of the most preferred microblogging websites when it comes to putting your thoughts on the internet. Big organizations use this platform for advertising their product; government institutions even use it to provide prompt customer resolution; various groups use Twitter to run social awareness campaigns and media campaigns. Twitter has close to 330 million monthly active users worldwide, out of which 17 million hits are generated from India itself along with more than 1 billion downloads on Play Store. With all this in mind, Twitter allows access to Twitter API to developers to create some cool applications like bots, automation tools, etc. This Twitter API gives developers access to almost all of Twitter's functionalities like likes, retweets, tweets, etc. Tweepy, a python package, helps us in achieving all this.
\nTweepy is a python package that smoothly and transparently accesses Twitter's endpoints made available for the developers. Without Tweepy, the user would have to take care of various low-level details about HTTP requests, rate limiting, authentication, serialization, etc. Tweepy handles all this mess on behalf of the user making the application prone to errors.\nIn simple words, Tweepy is an open-source python package that provides a way for developers to communicate with the Twitter API. But keep in mind that Twitter levies a rate limit on the number of requests made to the Twitter API. To be precise, 900 requests/15 minutes are allowed; Twitter feeds anything above that an error.
\nTweepy can be installed by using Python package manager pip. A simple demonstration can be seen below:\n![\"installation\"](\"/8400326bc3a2fdc039d749397d750ee8/installation.webp\" "\\"Installation\\"")
Installation on Linux and macOS should follow similar steps as well.
\nTwitter API uses OAuth for authentication, so initially, you need to apply for authentication credentials from Twitter. These authentication credentials basically consists of 4 components namely : consumerkey, consumersecretkey, accesstoken, accesstokensecret . These credentials from Twitter are used to instantiate the API. Each account gets a unique key, so don't reuse someone else's keys.\nFor getting those credentials from Twitter, apply for a developer account on the Twitter Developers page.\n![\"twitterdev\"](\"/a2766dccfa074e3f507c089a3fdd370d/twitterdev.webp\" "\\"Twitter")
\nThis is what the account looks like. Here you will get detailed information about the total no. of requests made, your API credentials, and much more information. After creating an account, you need to create an app wherein you will be asked to name your app and a short description. You must be wondering what an app is?\nThe app is like a gateway that contains a set of permissions and keys used to access the Twitter API. An app is needed for accessing the Twitter API as a part of Twitter's OAuth authentication. After creating an app, generate new authentication tokens for authorization purposes.
![\"keys\"](\"/fde857dd8c739c14bfa27571d14bc364/keys.webp\")
There are four common basic steps in any Tweepy application.
\nimport tweepy\n# authenticating twitter api credentials\nconsumer_key = '2OsNoPKOYCpxxxxxxxxxxxxxxxxxxxxxx'\nconsumer_secret = 'Xw07uU51xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\naccess_token = '24621057xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\naccess_token_secret = 'pXt5xxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\n# instantiating the api\nauth = tweepy.OAuthHandler(consumer_key, consumer_secret)\nauth.set_access_token(access_token, access_token_secret)\n\n# creating API object\n\napi = tweepy.API(auth,wait_on_rate_limit=True,wait_on_rate_limit_notify=True)\n\nimport tabulate\nuser = api.get_user("Cristiano")\nprint("User Details:")\nprint(tabulate([["Name","Description","Location"],[user.name,user.description,user.location]],headers="firstrow"))\n\nNote: Use your credentials in the hidden keys above._
\n
Objects created from the Tweepy. API class helps us access most of Twitter's available functionality like tweets, retweets, likes, etc. In the code snippet, we used the api.get_user method for getting information about a certain user on Twitter. Likewise, there can be several use cases of different methods(discussed below) made available by Twitter to developers. You can find the link for this code here.
\nNow we will see different methods provided by Twitter. The API methods have been divided into groups based on their functionality. The detailed guide for the API methods can be found at the official API Reference documentation.
\nThese methods handles the tweets, retweets, statuses on your/someone else's timeline as long as the account is public.
\ntweets = api.home_timeline()\nfor tweet in tweets:\n print(f"{tweet.user.name} said {tweet.text}")api.home_timeline() is an API method that returns the 20 most recent tweets on the user's timeline.
\nThese methods deal with creating, fetching tweets, retweeting tweets.
\napi.update_status("Hello World!")api.update_status() is an API method used to create a tweet on the user's timeline. For each update request, it will check the user's recent tweets. If any duplication is found, the request will be blocked by Twitter as a user cannot post the same tweet more than once.
\nThese methods help to find the user details using various paramaeter like name, location, description,friends, followers,etc. as long as the account is public.
\nuser = api.get_user("ISRO")\nprint(user.name)\nprint(user.decscription)\nprint(user.followers)\nprint(user.location)We have used get_user() previously to fetch user details of specific accounts.
\nThese methods help the user to follow, unfollow certain accounts, list the accounts user follows, etc.
\napi.create_friendship("elonmusk")create_friendship() will add @elonmusk to the list of accounts you follow.
\nLikes or unlikes(if already liked) the status specified in the ID parameter as the authenticating user.
\napi.create_favorite(tweet.id)create_favorite() will like a tweet based on the tweet id provided.
\nUsed to block, unblock, list blocked accounts of the user.
\napi.create_block(id/screen_name)createblock() will block the specific user using the id/screenname provided.
\nThese methods help the user to search specific tweet based upon the search query and parameters provided. But not all tweets will be indexed or made available through the search methods.
\nfor tweets in api.search(q="iphone", lang="en"):\n print(tweet.text)search() will look out for all the tweets available for the query keyword 'q' provided.
\nIt returns the trends going on at a specific geographical location.
\napi.trends_place(1)trends_place() will show trends in the specific area. Here one stands for worldwide.
\nIt returns the geographical location like latitude, the longitude of the place id provided.
\napi.geo_id(id)geo_id() returns more geographical information of the concerned place id.
\nTweepy allows the user to concentrate on the application's logic by hiding many low-level details, thus making the application bug-free. You can use tweepy to do some cool projects like bots, automation, machine learning applications, etc.
\nMore information about tweepy can be found at docs. Make sure to look at the official documentation as it will provide you with a greater picture of the package. So what are you waiting for? Go ahead, use your imagination, and get started with the side-project you've always been thinking of.
\n","frontmatter":{"date":"November 02, 2020","updated_date":null,"title":"Beginners Guide to Tweepy ","tags":["Twitter","Tweepy","Python"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/f706bedc2cff52e9c0f73c8e799ee39c/58556/coverimage.webp","srcSet":"/static/f706bedc2cff52e9c0f73c8e799ee39c/61e93/coverimage.webp 200w,\n/static/f706bedc2cff52e9c0f73c8e799ee39c/1f5c5/coverimage.webp 400w,\n/static/f706bedc2cff52e9c0f73c8e799ee39c/58556/coverimage.webp 800w,\n/static/f706bedc2cff52e9c0f73c8e799ee39c/99238/coverimage.webp 1200w,\n/static/f706bedc2cff52e9c0f73c8e799ee39c/7c22d/coverimage.webp 1600w,\n/static/f706bedc2cff52e9c0f73c8e799ee39c/25f09/coverimage.webp 1920w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Sameer Mahajan","github":"sameermahajan101","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/how-to-make-telegram-bot/"},"html":"According to Wikipedia, It is a software application that runs automated tasks (scripts) over the Internet. In other terms, It is a program written to mimic human behaviour in order to perform some tasks. Bots take input and do simple and repetitive tasks, which is much faster than humans.
\nThere are two types of bots:
\nIn this, We will make Telegram Chatbot, which would send a copy of the input that the user has sent.
\nWe will be using Python language to make the bot and will be using Telegram package for our bot.\nYou can know more about the telegram package from here
\nFirstly, we will generate our telegram token. Below are the steps to follow to generate your own token.
\nAfter generating our token, we will make a python program to create a Telegram bot that will send the Text, Emojis, and Stickers similar to the user's input.
\nimport logging\nfrom telegram.ext import Updater, Filters, CommandHandler, MessageHandlerlogging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', #take time,level,name\n level=logging.INFO)\nlogger = logging.getLogger(__name__)This script does a basic configuration for the logging system. It takes time, level, and name.
\nTOKEN = "ENTER YOUR TOKEN" def start(bot,update):\n name = update.message.from_user.first_name #first name of the user messaging\n reply = "Hi!! {}".format(name)\n bot.send_message(chat_id = update.message.chat_id, text = reply) #sending message\n\ndef help(bot,update):\n reply = "How can I help You"\n bot.send_message(chat_id = update.message.chat_id, text = reply) #sending message\n\ndef echo_text(bot,update):\n reply = update.message.text\n bot.send_message(chat_id = update.message.chat_id, text = reply)\n\ndef sticker(bot,update):\n reply = update.message.sticker.file_id\n bot.send_sticker(chat_id = update.message.chat_id, sticker = reply)\n\ndef error(bot,update):\n logger.error("Shit!! Update {} caused error {}".format(update,update.error))start function would be taking the user's input and would be sending, Hi, with the user's name.
\nhelp function will prompt the message of how can I help you.
\necho_text and sticker function will send the same message or the stickers which the user has sent.
\nerror function would be printing the error message on the command prompt.
\ndef main():\n updater = Updater(TOKEN) #take the updates\n dp = updater.dispatcher #handle the updates\n\n dp.add_handler(CommandHandler("start", start))\n dp.add_handler(CommandHandler("help", help))\n dp.add_handler(MessageHandler(Filters.text, echo_text)) #if the user sends text\n dp.add_handler(MessageHandler(Filters.sticker, sticker)) #if the user sends sticker\n dp.add_error_handler(error)\n updater.start_polling()\n logger.info("Started...")\n updater.idle()\n\nif __name__=="__main__":\n main()\n\nSave the program and execute it to check the working of the telegram bot on telegram.
\n
You can get the source code to make the Telegram bot from this Github Repository Bot :robot:. You can also make some changes to the code to make the bot do cool stuff such as Getting News, Articles, or Movies recommendations.
\n","frontmatter":{"date":"October 20, 2020","updated_date":null,"title":"How to make a telegram bot","tags":["Telegram Bot","Python"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.4814814814814814,"src":"/static/aceaa6a49195f6d0417cb80f93f56be4/58556/bot.webp","srcSet":"/static/aceaa6a49195f6d0417cb80f93f56be4/61e93/bot.webp 200w,\n/static/aceaa6a49195f6d0417cb80f93f56be4/1f5c5/bot.webp 400w,\n/static/aceaa6a49195f6d0417cb80f93f56be4/58556/bot.webp 800w,\n/static/aceaa6a49195f6d0417cb80f93f56be4/99238/bot.webp 1200w,\n/static/aceaa6a49195f6d0417cb80f93f56be4/135cd/bot.webp 1280w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Rajan Puri","github":"rjn01","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/python-virtual-environments/"},"html":"In this article, we'll talk about the basic concept of virtual environment in python - what it is and how to use it and also how to build and manage separate environments for your Python projects using virtual environments.
\nA virtual environment is simply a tool that separates the dependencies of different projects by creating a separate isolated environment for each project.
\nThese are simply the directories so that unlimited virtual environments can be created. This is one of the popular tools used by most of the Python developers.
\nPython has various modules and packages for different applications. During our project, it may require a third-party library, which we install. Another project also uses the same directory for retrieval and storage but doesn't require any other third-party packages.
\nSo, the virtual environment can come into play and make a separate isolated environment for both projects, and each project can store and retrieve packages from their specific environment.
\nAlso, let us consider another case where we are creating a web application using Django. Suppose you are working on two projects project1 and project2.
\nIf project1 uses Django-2.2 and project2 uses Django-3.2, they would be stored in the same directory with the same name, and the error may occur. Then, in such cases, virtual environments can be really helpful for you to maintain the dependencies of both the projects.
\nTo create a virtual environment, we need a module named virtualenv. It creates a folder with all the necessary executables to run Python projects. Make sure pip is installed on your computer. If not, then use the following command:
\n![\"Pip\"](\"/9fa10fcc5576d9428768731619cba792/pip.webp\")
Open the terminal and paste the following command to install a virtualenv:
\n![\"Install\"](\"/f9658abd5f1e49b3e951afcd884adb6a/install.webp\")
If you want to confirm whether it is installed or not, paste the command below in your terminal:
\n![\"Check\"](\"/5221153f420814832190b6a8eb7c5ec4/check.webp\")
After successful installation of virtualenv, now you can create a virtual environment with your desired name using following command:
\n![\"Check\"](\"/76cce15130697d584fb75e02df60601b/name.webp\")
Here, I created venv as the name of my virtual environment. You can name it whatever you desire. The folder with the name venv is created that contains all the necessary executables to run the Python project. This is the folder where all your python packages will run.
To specify the Python interpreter of your choice, you can do it easily by specifying Python{version-name}.\nEg: To create python2.7 virtual environment, we use following command:
\n![\"Version\"](\"/a5b7c69d3dff106d6fe355ae725541dd/version.webp\")
After successfully creating the virtual environment, you need to activate it to enter into that particular isolated environment. Always remember to activate the required virtual environment before working on the project. To activate it, we move to the location where the name of your desired virtual environment is located and follow the command below:
\n![\"Activate\"](\"/9d3d2c7f55a4fd52999a514f73ac46ef/activate.webp\")
Note: Instead of venv, you write the name of your virtual environment you created.
![\"Windows\"](\"/9fb235d96fd1d14f07bfa3fffa942419/windows.webp\")
After activating the virtual environment, you can see your name on the virtual environment on the terminal's left side. It confirms the activation of your virtual environment and is currently active.
\n![\"Activated\"](\"/affe1c8aa1265fe5783364e3f41de2fc/activated.webp\")
Finally, you successfully created and activated your virtual environment. Now, you can install the dependencies and requirements your project asks for without interfering with other projects.
\nOnce you completed your work, then you can return to the system default environment by just using the deactivate command as shown below:
\n![\"Deactivate\"](\"/2d752ab9b41bfc22473be6e2614d02c2/deactivate.webp\")
\n","frontmatter":{"date":"October 16, 2020","updated_date":null,"title":"Python Virtual Environment: What is it and how it works?","tags":["Python"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":2.127659574468085,"src":"/static/674ce17a9dc4e13f0616a1fd935bc596/58556/python.webp","srcSet":"/static/674ce17a9dc4e13f0616a1fd935bc596/61e93/python.webp 200w,\n/static/674ce17a9dc4e13f0616a1fd935bc596/1f5c5/python.webp 400w,\n/static/674ce17a9dc4e13f0616a1fd935bc596/58556/python.webp 800w,\n/static/674ce17a9dc4e13f0616a1fd935bc596/7bd08/python.webp 1171w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Ashish Maharjan","github":"AshishMhrzn10","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/speed-up-python-code/"},"html":"Python is one of the most popular languages all over the world. Nowadays it is being used in competitive programming also because of its simple syntax and rich libraries. Most of us probably started coding with python. At first, everything goes simple and easy. But while solving a hard algorithmic problem, most of us suffer from Time Limit Exceeded. However, it is not a problem of python; it is the programmer's problem. I am not saying that language is not slow, but if a programmer writes an efficient programme, it will get Accepted for sure. Here are some tips to speed up your python programme.
Use of proper data structure has a significant effect on runtime. Python has list, tuple, set and dictionary as the built-in data structures. However, most of the people use the list in all cases. But it is not a right choice. Use proper data structures depending on your task. Especially use a tuple instead of a list. Because iterating over tuple is easier than iterating over a list.
\nAs for loop is dynamic in python, it takes more time than while loop. So, use while loop instead of for loop.
\nDo not use any other technique if you can use list comprehension. For example, here is a code to list all the numbers between 1 and 1000 that is the multiplier of 3:
\nL = []\nfor i in range (1, 1000):\n if i%3 == 0:\n L.append (i)Using list comprehension, it would be:
\nL = [i for i in range (1, 1000) if i%3 == 0]List comprehension works faster than using the append method.
\nDo not assaign variables like this:
\na = 2\nb = 3\nc = 5\nd = 7Instead, assign variables like this:
\na, b, c, d = 2, 3, 5, 7Python has global keyword to declare global variables. But global variables take higher time during operation than a local variable. So, do not use global variables if it is not necessary.
Do not write your function (manually) if it is already in the library. Library functions are highly efficient, and you will probably won't be able to code with that efficiency.
\nIn python, you can concatenate strings with + operation.
concatenatedString = "Programming " + "is " + "fun."It can also be done with join() method.
concatenatedString = " ".join (["Programming", "is", "fun."])join() concatenates strings faster than + operation because + operators create a new string and then copies the old content at each step. But join() doesn't work that way.
If you have a large amount of data in your list and you need to use one data at a time and for once then use generators. It will save you time.
See the below code:
\nL = []\nfor element in set(L):\n ...The above code may seem efficient because it used set to delete duplicate data. But the reality is that the code is not efficient. Do not forget that converting a list into set takes time. So this code will work better than the previous:
\nfor element in L:\n ...Try to avoid dot operation. See the below programme.
\nimport math\nval = math.sqrt(60)Instead of the above style write code like this:
\nfrom math import sqrt\nval = sqrt(60)Because when you call a function using . (dot) it first calls __getattribute()__ or __getattr()__ which then use dictionary operation which costs time. So, try using from module import function.
Use while 1 instead of while True. It will reduce some runtime.
Try new ways to write your code efficiently. See the below code.
\nif a_condition:\n if another_condition:\n do_something\nelse:\n raise exceptionInstead of the above code you can write:
\nif (not a_condition) or (not another_condition):\n raise exception\ndo_somethingFor python's slow speed, some projects have been taken to decrease runtime. Pypy and Numba two of them. In most of the programming contests, you will see pypy if it allows python. These applications will reduce the runtime of your programme.
\nC/C++ is faster than python. So, many packages and modules have been written in C/C++ that you can use in your python programme. Numpy, Scipy and Pandas are three of them and are popular for processing large datasets.
Python is updated and upgraded regularly, and every release is faster and more optimized. So always use the latest version of python.
\nThese were some of the tips to decrease the runtime of python code. There are a few more techniques that you can use. Use a search engine to find those and write efficient code!
\n","frontmatter":{"date":"October 15, 2020","updated_date":null,"title":"Speed Up Python Code","tags":["Python","Performance"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/65558b9da57861b919205b297c81828b/58556/speed-up-python-code-1.webp","srcSet":"/static/65558b9da57861b919205b297c81828b/61e93/speed-up-python-code-1.webp 200w,\n/static/65558b9da57861b919205b297c81828b/1f5c5/speed-up-python-code-1.webp 400w,\n/static/65558b9da57861b919205b297c81828b/58556/speed-up-python-code-1.webp 800w,\n/static/65558b9da57861b919205b297c81828b/99238/speed-up-python-code-1.webp 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Md. Tahmid Hossain","github":"tahmid02016","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/full-data-science-pipeline-implementation/"},"html":"Data enrichment is the process of combining first-party data from internal sources with disparate data from other internal systems or third-party data from external sources.
\nUsually, the data available from clients or stakeholders are not enough to solve the given problem statement, like if a client comes with a problem statement to build a recommendation engine for his mutual fund industry, the usual data they have is old purchase data but that's not enough as client behaviour changes with time and is impacted by the present market condition, oil prices, etc. which needs to be incorporated in the model to make it efficient.
\nCodes for this tutorial is at Link
\nThe whole process id divided into four steps:
\nI have implemented a full pipeline of data science from scrapping data from web to implementing ml and NLP classification.
\nHere I have scraped data from IMDB website (imdb.py)
\nI have tried to implement simple ML regression on the data (ml_imdb.py)
\nI have prepared the data for NLP classification (multilabel_prep.py)
\nI have implemented multilabel NLP classifier using various techniques like chain classifier etc. (multilabelnlpclassifier.ipynb)
\nWeb scraping is the process of extracting and parsing raw data from the web. Web scraping is a technique which helps data scientist to make their data-rich and is an efficient technique of data collection.
\nThis world is full of data, but unfortunately, most of them are not in the form to be used. Data is like crude oil, or we say it is in unstructured form. For a data scientist or engineer, our first challenge is to make the data model consumption ready, which takes the majority of the time, and this whole process is collectively known as data preprocessing.
\nHTML is a form of primary markup language and the base framework of mostly all websites. For performing web scraping its necessary to know it
\nHere we will start with requesting the web page using python package requests.
\n from requests import get\n url = 'http://www.imdb.com/search/title?release_date=2017&sort=num_votes,desc&page=1'\n response = get(url)\n print(len(response.text))The whole web page is now stored in the variable object response.\nThen we parse the web page using beautifulsoup package.
\n from bs4 import BeautifulSoup\n html_soup = BeautifulSoup(response.text, 'html.parser')\n type(html_soup)Then I will store all the div with the class named lister-item mode-advanced in variable movie_containers.
\nmovie_containers = html_soup.find_all('div', class_ = 'lister-item mode-advanced')Then I iterate through this object and store the information in lists to make my final DataFrame, using simple for loops.
\n# Lists to store the scraped data in\nnames = []\nyears = []\nimdb_ratings = []\nmetascores = []\nvotes = []\n#gross=[] #many movies have no record\nmovie_description=[]\nmovie_duration=[]\nmovie_genre=[]\n# Extract data from individual movie container\nfor container in movie_containers:\n# If the movie has Metascore, then extract:\n if container.find('div', class_ = 'ratings-metascore') is not None:\n# The name\n name = container.h3.a.text\n names.append(name)\n# The year\n year = container.h3.find('span', class_ = 'lister-item-year').text\n years.append(year)\n# The IMDB rating\n imdb = float(container.strong.text)\n imdb_ratings.append(imdb)\n# The Metascore\n m_score = container.find('span', class_ = 'metascore').text\n metascores.append(int(m_score))\n# The number of votes\n vote = container.find('span', attrs = {'name':'nv'})['data-value']\n votes.append(int(vote))\n# Gross income of movie\n #gross_inc =container.find_all('span', attrs = {'name':'nv'})[1]['data-value']\n #gross.append(gross_inc)\n\n# movie description\n movie_desc=container.find_all('p', class_ = 'text-muted')[1].text\n movie_description.append(movie_desc)\n movie_det=container.find_all('p', class_ = 'text-muted')[0]\n\n\n# Movie duration\n movie_dur=movie_det.find('span',class_='runtime').text\n movie_duration.append(movie_dur)\n\n# Movie genre\n movie_gen=movie_det.find('span',class_='genre').text\n movie_genre.append(movie_gen)\n\nimport pandas as pd\none_df = pd.DataFrame({'movie': names,\n'year': years,\n'imdb': imdb_ratings,\n'metascore': metascores,\n'votes': votes,\n#'gross':gross,\n'movie decription':movie_description,\n'movie duration':movie_duration,\n'movie genre':movie_genre\n})\nprint(one_df.info())\none_df.to_csv('50_movie_details.csv')But this was only for one page which has data for 50 movies only which is not enough to build a model.
\nPlease refer my code to understand how I use simple for loops to iterate through all the movies and downloading data for 20 years(approx).
\nWhats is linear regression??
\nIt is one of the most popular and used statistical techniques\n• Used to understand the relationship between variables
\n• Can also be used to predict a value of interest for new observations
\n• The aim is to predict the value of a continuous numeric variable of interest (known as the response or dependent or target variable)
\n• The values of one or more predictor (or independent) variables are used to make the prediction
\n• One predictor = simple regression
\n• More predictors = multiple regression
\nHere I just tried to use metascore of movies firstly to predict IMDB ratings and secondly I wanted to enhance it by using metascore and votes to predict IMDB rating.
\n## ML model\n\nX = data.loc[:, 'metascore'].values\ny = data.loc[:, 'imdb'].values\n\n# Splitting the dataset into the Training set and Test set\nfrom sklearn.cross_validation import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state = 0)\n\nfrom sklearn.linear_model import LinearRegression\n\nregressor = LinearRegression()#making object for reg package\nregressor.fit(X_train.reshape(-1,1), y_train.reshape(-1,1))#to fit the regressor to our training data\n\n#predict the test results\ny_pred =regressor.predict(X_test.reshape(-1,1))\n\nfrom sklearn.metrics import mean_squared_error\nmean_squared_error(y_test, y_pred)\n\n# 0.18041462828221905## Let try with imdb and votes\nX1 = data.loc[:, ['metascore','votes']].values\ny1 = data.loc[:, 'imdb'].values\n\n# Splitting the dataset into the Training set and Test set\nfrom sklearn.cross_validation import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X1, y1, test_size = 0.33, random_state = 0)\n\nfrom sklearn.linear_model import LinearRegression\n\nregressor = LinearRegression()#making object for reg package\nregressor.fit(X_train, y_train)#to fit the regressor to our training data\n\n#predict the test results\ny_pred =regressor.predict(X_test)\n\n\nfrom sklearn.metrics import mean_squared_error\nmean_squared_error(y_test, y_pred)\n# 0.15729132122310804 good scoreI tried to scrape data from the IMDB site and then applied ML regression techniques on it. Later I found that the movies listed are multi-class like Logan belongs to Action, Drama, Sci-Fi, which led me to think about how to implement the classifier model in the multilabel data. Usually, the data we get in real-world is mostly multi labelled like chatbot data; the intent is many and like these movies which are multi-class.
\nHere we will first see how we prep our data for multilabel classification.
\nHere we have all tags in one single column which is not usable while we do classification, so we have to make separate columns for all labels, and if the row doesn't belong to that category, it will be filled by 0 else 1.
\nimport os\nos.chdir('Desktop/web_scraping/imdb scrapper_ml/')\n\nimport pandas as pd\n\ndata=pd.read_csv('multilabel_nlp_classification.csv')\n\nmovie_list=[x for x in data['movie genre']]\nmovie_list1=''\nfor x in data['movie genre']:\n movie_list1+=','+x\n\nli_m=movie_list1.split(',')\nli=[x.strip() for x in li_m]\nlist_s=list(set(li))\n\nfor x in list_s:\n data[x]=0\n\ndata['movie_genre']=[x.strip().split(',') for x in data['movie genre']]\n\nde=data.copy()\n#data.loc[0,'Action']=1\nde['id']=range(0,6116)\n#print(de.loc[de['id']==0,'Action'])\nfor i in range(0,6116):\n for x in de.loc[de['id']==i,'movie_genre']:\n for y in x:\n y=y.strip()\n de.loc[de['id']==i,y]=1\n\nde.to_csv('multilabel_nlp_classification.csv')Now, as our data is ready, we can start with NLP implementation.
\nFor multilabel classification, I used techniques like classifier chain, label powerset, etc.
\nHere the problem statement is that using the movie description our model has to guess which genre the movie belongs to. It is a popular use case. Take an example of ecommerce product description data; now instead of manually assigning the labels to it, we can use a model which will find relevant labels or genre for it and make the content relevant to the type it belongs.
\nI start with Exploratory data analysis and then data cleaning, which is the most crucial step as if all the description has some very 30-50 common words it will simply make the data-heavy and model slow and inefficient.
\nThen we go on to make the data model ready as ML models don't understand text data we have to feed numbers in it. For that purpose, we use TfidfVectorizer.
\nTfidfVectorizer - Transforms text to feature vectors that can be used as input to the estimator.
\nThen simply diving the data in train and test split.
\nx_train = vectorizer.transform(train_text)\ny_train = train.drop(labels = ['id','movie decription'], axis=1)\n\nx_test = vectorizer.transform(test_text)\ny_test = test.drop(labels = ['id','movie decription'], axis=1)I tried first with applying logistic regression and one vs rest classifier.
\nOneVsRestClassifier strategy splits a multi-class classification into one binary classification problem per class.\nOneVsRestClassifier is when we want to do multi-class or multilabel classification, and its strategy consists of fitting one classifier per class. For each classifier, the class is fitted against all the other classes.
\n# Using pipeline for applying logistic regression and one vs rest classifier\nLogReg_pipeline = Pipeline([\n ('clf', OneVsRestClassifier(LogisticRegression(solver='sag'), n_jobs=-1)),\n ])\n\nfor category in categories:\n printmd('**Processing {} comments...**'.format(category))\n \n # Training logistic regression model on train data\n LogReg_pipeline.fit(x_train, train[category])\n \n # calculating test accuracy\n prediction = LogReg_pipeline.predict(x_test)\n print('Test accuracy is {}'.format(accuracy_score(test[category], prediction)))\n print("\\n")Next, I tried with BinaryRelevance
\nIt is a simple technique which treats each label as a separate single class classification problem.
\n# using binary relevance\nfrom skmultilearn.problem_transform import BinaryRelevance\nfrom sklearn.naive_bayes import GaussianNB\n\n# initialize binary relevance multi-label classifier\n# with a gaussian naive bayes base classifier\nclassifier = BinaryRelevance(GaussianNB())\n\n# train\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)Next, I tried using ClassifierChain.
\nIt is almost similar to BinaryRelevance, here the first classifier is trained just on the input data, and then each next classifier is trained on the input space and all the previous classifiers in the chain.
\nfrom skmultilearn.problem_transform import ClassifierChain\nfrom sklearn.linear_model import LogisticRegression\n\n\n# initialize classifier chains multi-label classifier\nclassifier = ClassifierChain(LogisticRegression())\n\n# Training logistic regression model on train data\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)Next, I tried using Label Powerset.
\nHere we transform the problem into a multi-class problem with one multi-class classifier is trained on all unique label combinations found in the training data.
\nfrom skmultilearn.problem_transform import LabelPowerset\n# initialize label powerset multi-label classifier\nclassifier = LabelPowerset(LogisticRegression())\n\n# train\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)Please refer my notebook multilabelnlpclassifier.ipynb from my repo for more details.
\nAs my previous blog, I use the python web Crawler library to help crawl the static website. For the Scrapy, there can be customize download middle ware, which can deal with static content in the website like JavaScript.
\nHowever, the Scrapy already helps us with much of the underlying implementation, for example, it uses it own dispatcher and it has pipeline for dealing the parsing word after download. One drawback for using such library is hard to deal with some strange bugs occurring because they run the paralleled jobs.
\nFor this tutorial, I want to show the structure of a simple and efficient web crawler.
\nFirst of all, we need a scheduler, who can paralleled the job. Because the most of the time is on the requesting. I use the gevent to schedule the jobs. Gevent uses the libevent as its underlying library, which combines the multithreading and event-based techniques to parallel the job.
\nThere is the sample code:
\nimport gevent\nfrom gevent import Greenlet\nfrom gevent import monkey\nfrom selenium import webdriver\nmonkey.patch_socket()\nclass WebCrawler:\n def __init__(self,urls=[],num_worker = 1):\n self.url_queue = Queue()\n self.num_worker = num_worker\n def worker(self,pid):\n driver = self.initializeAnImegaDisabledDriver() #initilize the webdirver\n#TODO catch the exception\n while not self.url_queue.empty():\n url = self.url_queue.get()\n self.driver.get(url)\n elem = self.driver.find_elements_by_xpath("//script | //iframe | //img") # get such element from webpage\n def run(self):\n jobs = [gevent.spawn(self.worker,i) for i in xrange(self.num_worker)]The next part is the headless browser part. I use the phantomjs with --webdriver=4444 --disk-cache=true --ignore-ssl-errors=true --load-images=false --max-disk-cache-size=100000. You can get the detailed option from their documents.
Phantomjs uses selenium webdriver as front-end to handle the request. However phantomjs is using the webkit and QT as its underlying browser and controller. It has memory leak bugs therefore the phantomjs will consume ton of memory and it only can use one core of your CPU but you can deploy many instances of the phantomjs on different ports. I wrote a daemon process to monitor the memory and its situation but later I realize I can use Perl script to get the status of process and when it exceeds the limits like 1G memory and send kill signal to the process.
\nTo speed up the crawler, I choose to use static browser to verify the website first because the website is bad written, there might be deadlock occurring so just skip them.
\n","frontmatter":{"date":"July 14, 2015","updated_date":null,"title":"Write a highly efficient python Web Crawler","tags":["Python","Coding"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1,"src":"/static/c068d4f41cc4f2a33e3bd2737ca105df/7fbdd/python-web-crawler.webp","srcSet":"/static/c068d4f41cc4f2a33e3bd2737ca105df/61e93/python-web-crawler.webp 200w,\n/static/c068d4f41cc4f2a33e3bd2737ca105df/1f5c5/python-web-crawler.webp 400w,\n/static/c068d4f41cc4f2a33e3bd2737ca105df/7fbdd/python-web-crawler.webp 610w","sizes":"(max-width: 610px) 100vw, 610px"}}},"author":{"id":"Mark Duan","github":null,"avatar":null}}}}]}},"pageContext":{"tag":"Python"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}