{"componentChunkName":"component---src-templates-blog-list-template-js","path":"/engineering/26","result":{"data":{"allMarkdownRemark":{"edges":[{"node":{"excerpt":"What is data enrichment? and its importance Data enrichment is the process of combining first-party data from internal sources with…","fields":{"slug":"/engineering/full-data-science-pipeline-implementation/"},"html":"

What is data enrichment? and its importance

\n

Data enrichment is the process of combining first-party data from internal sources with disparate data from other internal systems or third-party data from external sources.

\n

Usually, the data available from clients or stakeholders are not enough to solve the given problem statement, like if a client comes with a problem statement to build a recommendation engine for his mutual fund industry, the usual data they have is old purchase data but that's not enough as client behaviour changes with time and is impacted by the present market condition, oil prices, etc. which needs to be incorporated in the model to make it efficient.

\n

Codes for this tutorial is at Link

\n

The whole process id divided into four steps:

\n

I have implemented a full pipeline of data science from scrapping data from web to implementing ml and NLP classification.

\n\n

Here I have scraped data from IMDB website (imdb.py)

\n\n

I have tried to implement simple ML regression on the data (ml_imdb.py)

\n\n

I have prepared the data for NLP classification (multilabel_prep.py)

\n\n

I have implemented multilabel NLP classifier using various techniques like chain classifier etc. (multilabelnlpclassifier.ipynb)

\n

What is web scraping?

\n

Web scraping is the process of extracting and parsing raw data from the web. Web scraping is a technique which helps data scientist to make their data-rich and is an efficient technique of data collection.

\n

This world is full of data, but unfortunately, most of them are not in the form to be used. Data is like crude oil, or we say it is in unstructured form. For a data scientist or engineer, our first challenge is to make the data model consumption ready, which takes the majority of the time, and this whole process is collectively known as data preprocessing.

\n

HTML is a form of primary markup language and the base framework of mostly all websites. For performing web scraping its necessary to know it

\n

Here we will start with requesting the web page using python package requests.

\n
  from requests import get\n  url = 'http://www.imdb.com/search/title?release_date=2017&sort=num_votes,desc&page=1'\n  response = get(url)\n  print(len(response.text))
\n

The whole web page is now stored in the variable object response.\nThen we parse the web page using beautifulsoup package.

\n
  from bs4 import BeautifulSoup\n  html_soup = BeautifulSoup(response.text, 'html.parser')\n  type(html_soup)
\n

Then I will store all the div with the class named lister-item mode-advanced in variable movie_containers.

\n
movie_containers = html_soup.find_all('div', class_ = 'lister-item mode-advanced')
\n

Then I iterate through this object and store the information in lists to make my final DataFrame, using simple for loops.

\n
# Lists to store the scraped data in\nnames = []\nyears = []\nimdb_ratings = []\nmetascores = []\nvotes = []\n#gross=[] #many movies have no record\nmovie_description=[]\nmovie_duration=[]\nmovie_genre=[]\n# Extract data from individual movie container\nfor container in movie_containers:\n# If the movie has Metascore, then extract:\n    if container.find('div', class_ = 'ratings-metascore') is not None:\n# The name\n        name = container.h3.a.text\n        names.append(name)\n# The year\n        year = container.h3.find('span', class_ = 'lister-item-year').text\n        years.append(year)\n# The IMDB rating\n        imdb = float(container.strong.text)\n        imdb_ratings.append(imdb)\n# The Metascore\n        m_score = container.find('span', class_ = 'metascore').text\n        metascores.append(int(m_score))\n# The number of votes\n        vote = container.find('span', attrs = {'name':'nv'})['data-value']\n        votes.append(int(vote))\n# Gross income of movie\n        #gross_inc =container.find_all('span', attrs = {'name':'nv'})[1]['data-value']\n        #gross.append(gross_inc)\n\n# movie description\n        movie_desc=container.find_all('p', class_ = 'text-muted')[1].text\n        movie_description.append(movie_desc)\n        movie_det=container.find_all('p', class_ = 'text-muted')[0]\n\n\n# Movie duration\n        movie_dur=movie_det.find('span',class_='runtime').text\n        movie_duration.append(movie_dur)\n\n# Movie genre\n        movie_gen=movie_det.find('span',class_='genre').text\n        movie_genre.append(movie_gen)\n\nimport pandas as pd\none_df = pd.DataFrame({'movie': names,\n'year': years,\n'imdb': imdb_ratings,\n'metascore': metascores,\n'votes': votes,\n#'gross':gross,\n'movie decription':movie_description,\n'movie duration':movie_duration,\n'movie genre':movie_genre\n})\nprint(one_df.info())\none_df.to_csv('50_movie_details.csv')
\n

But this was only for one page which has data for 50 movies only which is not enough to build a model.

\n

Please refer my code to understand how I use simple for loops to iterate through all the movies and downloading data for 20 years(approx).

\n

Implementing simple linear algorithms in numerical data we just scrapped

\n

Whats is linear regression??

\n

It is one of the most popular and used statistical techniques\n• Used to understand the relationship between variables

\n

• Can also be used to predict a value of interest for new observations

\n

• The aim is to predict the value of a continuous numeric variable of interest (known as the response or dependent or target variable)

\n

• The values of one or more predictor (or independent) variables are used to make the prediction

\n

• One predictor = simple regression

\n

• More predictors = multiple regression

\n

Here I just tried to use metascore of movies firstly to predict IMDB ratings and secondly I wanted to enhance it by using metascore and votes to predict IMDB rating.

\n
## ML model\n\nX = data.loc[:, 'metascore'].values\ny = data.loc[:, 'imdb'].values\n\n# Splitting the dataset into the Training set and Test set\nfrom sklearn.cross_validation import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state = 0)\n\nfrom sklearn.linear_model import LinearRegression\n\nregressor = LinearRegression()#making object for reg package\nregressor.fit(X_train.reshape(-1,1), y_train.reshape(-1,1))#to fit the regressor to our training data\n\n#predict the test results\ny_pred =regressor.predict(X_test.reshape(-1,1))\n\nfrom sklearn.metrics import mean_squared_error\nmean_squared_error(y_test, y_pred)\n\n# 0.18041462828221905
\n
## Let try with imdb and votes\nX1 = data.loc[:, ['metascore','votes']].values\ny1 = data.loc[:, 'imdb'].values\n\n# Splitting the dataset into the Training set and Test set\nfrom sklearn.cross_validation import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X1, y1, test_size = 0.33, random_state = 0)\n\nfrom sklearn.linear_model import LinearRegression\n\nregressor = LinearRegression()#making object for reg package\nregressor.fit(X_train, y_train)#to fit the regressor to our training data\n\n#predict the test results\ny_pred =regressor.predict(X_test)\n\n\nfrom sklearn.metrics import mean_squared_error\nmean_squared_error(y_test, y_pred)\n# 0.15729132122310804 good score
\n

I tried to scrape data from the IMDB site and then applied ML regression techniques on it. Later I found that the movies listed are multi-class like Logan belongs to Action, Drama, Sci-Fi, which led me to think about how to implement the classifier model in the multilabel data. Usually, the data we get in real-world is mostly multi labelled like chatbot data; the intent is many and like these movies which are multi-class.

\n

Here we will first see how we prep our data for multilabel classification.

\n

Here we have all tags in one single column which is not usable while we do classification, so we have to make separate columns for all labels, and if the row doesn't belong to that category, it will be filled by 0 else 1.

\n
import os\nos.chdir('Desktop/web_scraping/imdb scrapper_ml/')\n\nimport pandas as pd\n\ndata=pd.read_csv('multilabel_nlp_classification.csv')\n\nmovie_list=[x for x in data['movie genre']]\nmovie_list1=''\nfor x in data['movie genre']:\n    movie_list1+=','+x\n\nli_m=movie_list1.split(',')\nli=[x.strip() for x in li_m]\nlist_s=list(set(li))\n\nfor x in list_s:\n    data[x]=0\n\ndata['movie_genre']=[x.strip().split(',') for x in data['movie genre']]\n\nde=data.copy()\n#data.loc[0,'Action']=1\nde['id']=range(0,6116)\n#print(de.loc[de['id']==0,'Action'])\nfor i in range(0,6116):\n    for x in de.loc[de['id']==i,'movie_genre']:\n        for y in x:\n            y=y.strip()\n            de.loc[de['id']==i,y]=1\n\nde.to_csv('multilabel_nlp_classification.csv')
\n

Now, as our data is ready, we can start with NLP implementation.

\n

For multilabel classification, I used techniques like classifier chain, label powerset, etc.

\n

Here the problem statement is that using the movie description our model has to guess which genre the movie belongs to. It is a popular use case. Take an example of ecommerce product description data; now instead of manually assigning the labels to it, we can use a model which will find relevant labels or genre for it and make the content relevant to the type it belongs.

\n

I start with Exploratory data analysis and then data cleaning, which is the most crucial step as if all the description has some very 30-50 common words it will simply make the data-heavy and model slow and inefficient.

\n

Then we go on to make the data model ready as ML models don't understand text data we have to feed numbers in it. For that purpose, we use TfidfVectorizer.

\n

What is TfidfVectorizer?

\n

TfidfVectorizer - Transforms text to feature vectors that can be used as input to the estimator.

\n

Then simply diving the data in train and test split.

\n
x_train = vectorizer.transform(train_text)\ny_train = train.drop(labels = ['id','movie decription'], axis=1)\n\nx_test = vectorizer.transform(test_text)\ny_test = test.drop(labels = ['id','movie decription'], axis=1)
\n

I tried first with applying logistic regression and one vs rest classifier.

\n

What is OneVsRestClassifier??

\n

OneVsRestClassifier strategy splits a multi-class classification into one binary classification problem per class.\nOneVsRestClassifier is when we want to do multi-class or multilabel classification, and its strategy consists of fitting one classifier per class. For each classifier, the class is fitted against all the other classes.

\n
# Using pipeline for applying logistic regression and one vs rest classifier\nLogReg_pipeline = Pipeline([\n                ('clf', OneVsRestClassifier(LogisticRegression(solver='sag'), n_jobs=-1)),\n            ])\n\nfor category in categories:\n    printmd('**Processing {} comments...**'.format(category))\n    \n    # Training logistic regression model on train data\n    LogReg_pipeline.fit(x_train, train[category])\n    \n    # calculating test accuracy\n    prediction = LogReg_pipeline.predict(x_test)\n    print('Test accuracy is {}'.format(accuracy_score(test[category], prediction)))\n    print("\\n")
\n

Next, I tried with BinaryRelevance

\n

What is BinaryRelevance?

\n

It is a simple technique which treats each label as a separate single class classification problem.

\n
# using binary relevance\nfrom skmultilearn.problem_transform import BinaryRelevance\nfrom sklearn.naive_bayes import GaussianNB\n\n# initialize binary relevance multi-label classifier\n# with a gaussian naive bayes base classifier\nclassifier = BinaryRelevance(GaussianNB())\n\n# train\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)
\n

Next, I tried using ClassifierChain.

\n

What is ClassifierChain?

\n

It is almost similar to BinaryRelevance, here the first classifier is trained just on the input data, and then each next classifier is trained on the input space and all the previous classifiers in the chain.

\n
from skmultilearn.problem_transform import ClassifierChain\nfrom sklearn.linear_model import LogisticRegression\n\n\n# initialize classifier chains multi-label classifier\nclassifier = ClassifierChain(LogisticRegression())\n\n# Training logistic regression model on train data\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)
\n

Next, I tried using Label Powerset.

\n

What is LabelPowerset?

\n

Here we transform the problem into a multi-class problem with one multi-class classifier is trained on all unique label combinations found in the training data.

\n
from skmultilearn.problem_transform import LabelPowerset\n# initialize label powerset multi-label classifier\nclassifier = LabelPowerset(LogisticRegression())\n\n# train\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)
\n

Please refer my notebook multilabelnlpclassifier.ipynb from my repo for more details.

\n

Improvement:

\n
    \n
  1. More feature engineering and data to avoid this overfitting and make more efficient pipeline
    2. \n
  2. If we collect more data, deep learning and state of the art algorithms like BERT can help us to leverage the efficiency of the model.
    3. \n
\n

Summary:

\n\n","frontmatter":{"date":"October 09, 2020","updated_date":null,"description":"Learn how to implement the full data science pipeline right from collecting the data to implementing ML algorithms.","title":" Full data science pipeline implementation","tags":["DataScience","Python","Web scraping","NLP","Machine learning"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/5a2492e69b5cbb1c2f5bb0a3d4ca9fc2/58556/ds.webp","srcSet":"/static/5a2492e69b5cbb1c2f5bb0a3d4ca9fc2/61e93/ds.webp 200w,\n/static/5a2492e69b5cbb1c2f5bb0a3d4ca9fc2/1f5c5/ds.webp 400w,\n/static/5a2492e69b5cbb1c2f5bb0a3d4ca9fc2/58556/ds.webp 800w,\n/static/5a2492e69b5cbb1c2f5bb0a3d4ca9fc2/df6a9/ds.webp 1125w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Rinki Nag","github":"eaglewarrior","avatar":null}}}},{"node":{"excerpt":"The two types of email you can send and receive are plain text emails (any email that contains just plain old text with no formatting) and…","fields":{"slug":"/engineering/html-email-concept/"},"html":"

The two types of email you can send and receive are plain text emails (any email that contains just plain old text with no formatting) and HTML emails, these are formatted and styled using HTML and inline CSS.\nHTML email is the use of HTML to provide formatting and semantic markup capabilities in an email that are not available in plain text.

\n

An HTML email is designed just like a website with the help of graphics, table columns, colors and links. A non-programmer can also create it since email marketing services provide flexible campaign builders. Email client vendors have not been as developing as web browser vendors in adopting new standards.

\n

Definition

\n

Emails which are formatted using Hypertext Markup Language(HTML), as opposed to plain text email.

\n

How to Create an HTML Email

\n

Many tools that create and send an email will offer pre-formatted, already built HTML templates that allow you to design emails without knowing or accessing any code of back-end.

\n

The best way to understand any process is to do it yourself, from level zero. We make any changes in the email editor; those changes will be automatically coded into the final result. This email building tool is the best option if you don't have an email designer, but you still want to send any professional marketing emails.

\n

If you want more control over the code of your emails and you are comfortable with HTML(that is just basic and easy), most email tools will allow you to import HTML files directly for using it as custom email templates. They have a wide variety of free HTML email templates available on the internet, and if you are familiar with HTML, it is a straightforward process to use that template in the email building tool of your own choice.

\n

To create an HTML email from scratch, you will need to have advanced knowledge of HTML. Because creating an HTML email from scratch can be quite tricky, we recommend you to work with a developer for this process, or you may go with a template for an easy process.

\n

If you choose to code your HTML email by hand, these are the necessary steps you need to use while creating HTML email:

\n
    \n
  1. The perfect email template size should have 600-700 max-width.
    2. \n
  2. If the design has animation, then use .gif animated file because interactive elements like Flash, JavaScript, or HTML forms won't work in most email inboxes.
    3. \n
  3. Try to use HTML tables (HTML tables present tabular data in a semantic and structurally appropriate manner) for your presentation.
    4. \n
  4. To improve the presentation of Web, use inline CSS within your HTML email.
    5. \n
  5. CSS style should be either in a separate CSS file or below the body tag and not under the head tag.
    6. \n
  6. To save yourself from trouble, avoid the use of CSS shorthand code.
    7. \n
  7. The most genuine way of coding background colors is to use six-digit hexadecimal code for color (like #000000, i.e. for black).
    8. \n
  8. Be sure always to use \"display: block;\" for your image tags (either inline or embedded CSS) because this takes the baseline out of the equation and keeps everything arranged neatly and in order.
    9. \n
  9. If you wish to have padding on columns, it might be more cross-browser, so you can always create spacer DIVs in between the columns (or between rows).
    10. \n
  10. You need to use absolute paths for your images.
    11. \n
  11. Try adding a line-height and font-size of 1 under \"\" (or the desired size).
    12. \n
  12. Inline styles to and tables are the right way to go for Html email.
    13. \n
  13. In an HTML table, you can set the cell padding and cell spacing to zero to eliminate the unwanted spacing in your layout.
    14. \n
\n

How to send HTML emails through Outlook?

\n
    \n
  1. Select more commands to customize your quick access toolbar(suggestion).
    2. \n
  2. Choose \"attach\" function and then \"add\" it to the toolbar.
    3. \n
  3. Open \"attach a file\" window from the quick access toolbar.
    4. \n
  4. Select HTML file you need to import BUT do not click to INSERT yet.
    5. \n
  5. Switch \"insert\" button with the \"insert as a text\" button and click.
    6. \n
  6. Now, you can send it to your audience.
    7. \n
\n

You can check HTML Email Template here:

\n

HTML Email Templates

\n","frontmatter":{"date":"October 09, 2020","updated_date":null,"description":"If you choose to code your HTML email by hand, there are many different things you need to use while creating HTML email.","title":"HTML Email Concept","tags":["Html","Email"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.7699115044247788,"src":"/static/67ac86643dbb14dedfd4fe52ff7e659c/58556/email_picture.webp","srcSet":"/static/67ac86643dbb14dedfd4fe52ff7e659c/61e93/email_picture.webp 200w,\n/static/67ac86643dbb14dedfd4fe52ff7e659c/1f5c5/email_picture.webp 400w,\n/static/67ac86643dbb14dedfd4fe52ff7e659c/58556/email_picture.webp 800w,\n/static/67ac86643dbb14dedfd4fe52ff7e659c/54d25/email_picture.webp 1080w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Nivedita Singh","github":"Nivedita967","avatar":null}}}},{"node":{"excerpt":"These days we have all come across one of the coolest buzzwords in the IT industry: \"The Blockchain\". It might seem to be a new magic word…","fields":{"slug":"/engineering/blockchain-the-new-technology-of-security-trust/"},"html":"

These days we have all come across one of the coolest buzzwords in the IT industry: \"The Blockchain\". It might seem to be a new magic word in the market that companies spell interest in their businesses. However, the complexity of it is incredibly far-reaching. Blockchain integrates the openness and flexibility of the internet with the security of cryptography to come out with a safer, faster way of verification of information and most importantly establishes trust in this open world.

\n

Blockchain was first developed by an anonymous programmer or group of programmers known by a name 'Santoshi Nakamoto'.It was an underlying technology for the Bitcoin, which is used for peer-to-peer transactions. Blockchain at its heart is a list of transactions like a distributed ledger open to all in the network. It stores the data in such a way that it seems virtually impossible to add, update or remove any information stored without the notice of other users in a peer-to-peer network.

\n

How does blockchain work?

\n

Blockchain mainly performs two tasks: collect and order data in blocks- similar to the traditional computer database and then chain them securely using cryptography.

\n

Let us take a closer look at each block in this enormous chain –

\n\n

\"Blockchain\"

\n

Establishing the trust

\n

The copy of the complete blockchain is with all the participants so that they can detect tampering if the hash matches across the chain then everyone knows that it is trustworthy.\nBlockchain is an emerging technology but has been evolving ever since its innovation. This technology has the unlimited potential to bring about an upheaval in the way everyone- organisations, governments, individuals work together. It promises a simple, secure, paperless path to establish trust for virtual transactions of money, products and other confidential information worldwide.

\n

Blockchain in action

\n

Blockchain is one of the technologies that has gained popularity from its very birth, and now it is being used in many fields.

\n

Financial market

\n

In financial markets, trade is very dynamic where there is the exchange of money, assets involving multiple banks; this may lead to unexpected errors. To reduce this bottleneck blockchain came up with the idea of smart contracts which is a small computer program that describes the transactions step by step combining multiple blockchains, multiple assets and executes the transactions securely.

\n

Digital ID

\n

Blockchain can keep track of many commercial transactions and efficiently hold sensitive information. A digital id via blockchain secures the data stored and can be used worldwide in your fingertips.

\n

Supply chain

\n

Blockchain can be very handy in monitoring the supply chain in food and manufacturing industries by removing paper-based trails and also removing intermediaries between producers to customers. Not just the above applications it is used in many more places and has changed, is going to change the world around us.

\n

Blockchain the next-gen technology

\n

Though it is a new technology, it has an enormous ability to transform everything existing now. As a coin has its two faces, blockchain technology also has some glitches as it can destroy the middlemen in many of the industries.\nBlockchain has already spread its root firmly in soils of the new world, and the swarm of transformation has already begun. It is the responsibility of all the young generation to make complete usage of this technology as it matures and make it to become a huge money plant.

\n","frontmatter":{"date":"October 08, 2020","updated_date":null,"description":"Learn about Blockchain technology and how it works.","title":"Blockchain: The new technology of trust","tags":["Blockchain","Cyber Security"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.6666666666666667,"src":"/static/01f18fbf0dec445fa25aa3b13483883e/58556/CoverPage.webp","srcSet":"/static/01f18fbf0dec445fa25aa3b13483883e/61e93/CoverPage.webp 200w,\n/static/01f18fbf0dec445fa25aa3b13483883e/1f5c5/CoverPage.webp 400w,\n/static/01f18fbf0dec445fa25aa3b13483883e/58556/CoverPage.webp 800w,\n/static/01f18fbf0dec445fa25aa3b13483883e/1fb14/CoverPage.webp 960w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Shraddha V Prasad","github":"shraddhavp","avatar":null}}}},{"node":{"excerpt":"The acronym for Vim is Vi IMproved. Written by Bram Moolenaar, it is a free and open-source text editor. It was first released for UNIX…","fields":{"slug":"/engineering/vim-getting-started/"},"html":"

The acronym for Vim is Vi IMproved. Written by Bram Moolenaar, it is a free and open-source text editor. It was first released for UNIX variants in 1991, and its primary purpose was to develop the Vi editor, which was released back in 1976.

\n

Vim is Vi's newest and most popular reincarnation that supports both text and graphical interfaces and is supported on any known platform.

\n

The issue with learning Vim is not that it's challenging to do. It's that you have to continue to do it. This guide aims to break the loop so that for the last time, you can learn it.

\n

The purpose of this tutorial is to take you through every step of progress from understanding the philosophy of Vim. In short, in a way that sticks with you for a lifetime, we're going to master Vim.

\n

Let's start off!

\n

What is Vim?

\n

Vim is a text editor for Unix that comes with Linux, BSD, and macOS. It is known to be fast and powerful, partly because it is a small program that can run in a terminal (although it has a graphical interface). It is mainly because it can be managed entirely without menus or a mouse with a keyboard.

\n

For the following five reasons, I feel people should use Vim:

\n
    \n
  1. It's omnipresent. You don't have to think about learning about several boxes with a new editor.
    2. \n
  2. It is highly scalable. You can use it only to edit configuration files or become your entire forum for writing.
    3. \n
  3. It has a shallow memory footprint.
    4. \n
  4. It's command-centered. With a few commands, you can perform complex text-related tasks.
    5. \n
  5. It is highly configurable and uses simple text files to store its settings.
    6. \n
\n

Why Use Vim?

\n

In all POSIX systems, Vim is the default fallback editor. Vim is sure to be open, whether you have just installed the operating system, or you have booted into a minimal system repair environment, or you are unable to access any other editor. While you can switch out other tiny editors on your systems, such as GNU Nano or Jove, it's Vim that's all but guaranteed to be on every other system in the world.

\n

In short, I think competence with Vim should be considered the way you view competence with your native language, or simple maths, etc. In technology, so much begins with understanding your editor.

\n

Also, It is extremely customizable and you can adapt it to your way of coding and your way of doing things.\nTo give you a taste of vim , below are a few commands frequently used to code at lightning speed:

\n\n

The list goes on and on...

\n

Basically we have standard keybindings like d for delete, c for change, f for find etc. and we can pair them with numbers to repeat that action n number of times. Moreover doubling a command like so yy makes the command operate on the entire line.

\n

So to sum up,

\n
\n

Vim is for programmers who want to raise their game. In the hands of an expert, It shreds text at the speed of thought.

\n
\n

How to use Vim

\n

Well, there is a learning curve, but it's worth it. Although this steep curve is exaggerated in the online world and the quitting vim joke is getting very old (It's:q btw).

\n

Install Vim sudo apt-get vim or pacman -S vim for Linux user and type vim <filename> in your terminal. Press i to go to insert mode. Type some gibberish. Press :wq to save and quit . :q! to quit without saving.

\n

First of all, don't get disheartened when you see VIM for the first time, it's known to make a horrible first impression. We'll learn to customize it later in the blog.

\n

\"BadVim\"

\n

To get started, get your hands dirty on vimtutor . It's a 20-30 minute vim tutorial which teaches you all the basics. Just type vimtutor in your terminal (in a machine where you have vim installed) to open it.

\n

Try Vim Adventures. An interactive game that teaches you vim skills in a step by step manner and lets you unlock new abilities after you've proved your worth with the previous ones.

\n

Just get started with it . Learn as you go . You can always refer to the Vim CheatSheet to get better. Practice makes a man perfect. Watch screencasts if that's helpful (was helpful to me).

\n

How do I configure my Vim ?

\n

The best way to configure your Vim is through .vimrc file, which is hidden inside your home directory. Open a terminal and type vim ~/.vimrc to open your vimrc file using vim. Cool huh? Using vim to configure vim ..no? Nevermind!\nHere you can define your keybindings, set themes, install plugins etc.

\n

Here is an example of a custom keybinding\nmap <C-n> :NERDTreeToggle<CR>\nYou can check out this article on keybindings and how to use them.

\n

The easiest way to download and manage plugins for vim is by using a plugin manager. There are quite a few vim-pathogen , Vundle.vim , Vim-plug etc.\nVim-plug being my favourite. Installations and usage for each plugin manager can be found in their respective repositories.

\n

I personally like to call all my plugins in a separate file and link that file to my vimrc. Paste this line in your vimrc at the top if you want to do the same source ~/.vim/plugin/plugins.vim\nIn your plugins.vim file you can install different plugins like this:-

\n
call plug#begin('~/.vim/plugged')\n\nPlug 'scrooloose/nerdtree'\nPlug 'moll/vim-node'\nPlug 'yegappan/mru'\n\ncall plug#end()
\n

The first and last lines are for vim-plug to know what plugins are to be installed. The lines beginning with Plug is to install that particular plugin.

\n

Now just reload .vimrc (or quit vim and re-enter) and :PlugInstall to install plugins.

\n

What are some good plugins for Vim?

\n

Glad you asked because Vim is no fun without plugins made by the awesome open source community.\nHere are some good plugins you should definitely install. Just search these on GitHub in case you wanna know more about them or their usage.

\n\n

Pheww!!..\nSo you see there's a plugin for anything and everything. These plugins are more than enough to make vim a full-blown IDE.

\n

Summing up!!

\n

VIM makes you write code at the speed of thoughts. Just get started, and the rest will follow. Steal your friend's vimrc or find one on the internet (there are tons of them) if you don't want to make one of your own.

\n

If you want to slowly blend in into the world of vim then use some vim plugin for your text editor. All popular text editors have a vim plugin ( Here's one for VsCode )\nHere you go, best of both worlds! When you're comfortable enough, go for that complete switch.

\n

\"vscodevim\"

\n

Here's how my vim setup looks.

\n

\"Vim\"

\n

Now go give it a try. I wish you luck!!

\n","frontmatter":{"date":"October 08, 2020","updated_date":null,"description":"Vim is Vi's newest and most common reincarnation that is supported on every known platform, check out the article to learn about what is ViM, and why to use it.","title":"Vim: What is it and Why to use it?","tags":["Vim","Text Editor"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/aa9e337b13aca7c3043c49bba3b61a23/58556/Vim.webp","srcSet":"/static/aa9e337b13aca7c3043c49bba3b61a23/61e93/Vim.webp 200w,\n/static/aa9e337b13aca7c3043c49bba3b61a23/1f5c5/Vim.webp 400w,\n/static/aa9e337b13aca7c3043c49bba3b61a23/58556/Vim.webp 800w,\n/static/aa9e337b13aca7c3043c49bba3b61a23/99238/Vim.webp 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Shubhankar Khare","github":"iamshubhankarkhare","avatar":null}}}},{"node":{"excerpt":"Generally, MIM attacks are breaking to believe the traditional encryption technologies and targeting the intermediate nodes between the…","fields":{"slug":"/engineering/learn-about-vdn-for-cybersecurity/"},"html":"

Generally, MIM attacks are breaking to believe the traditional encryption technologies and targeting the intermediate nodes between the sender and receiver.

\n

What is a MIM attack?

\n

MIM stands for man-in-the-middle. In the World of Cybersecurity, a man-in-the-middle attack (MIM) is an attack where the attacker breaks into the middle of the network pathway silently such that the sender and receiver are not able to intercept and they believe they are directly communicating with each other. One such example of a MIM attack is active eavesdropping. In this example, an attacker can be sitting with a piece of software somewhere in the network path and capturing all the relevant network traffic for later analysis. The attacker can intercept all relevant messages passing between the two victims and by smartly monitor and alter old ones or inject new ones. It can become complicated and arise problems for the organization.

\n

Does the question arise how about using SSL and Virtual Private Networks?

\n

SSL and Virtual Private Networks do not always protect messages as they travel across intermediary pathways. So, that where virtual dispersive networking comes into the picture.

\n

VDN follows the approaches or methods of traditional military radio spread spectrum security. Radios rotate through the frequencies randomly, and communications are divided or split into multiple pieces (or streams). So now, only one receiving radios are programmed to reassemble these pieces into their original form.\nVDN divides the original message into some multiple parts, and it will encrypt each component separately and routes them over many servers, computers, and even mobile phones. The data also move out dynamically to optimum paths — both randomizing the paths the messages take while simultaneously taking into the server congestion or other network issues.\nWhen it comes to the role of Hackers, they are left scrambling to find out data parts as they go through like data centers, Cloud, Internet, and so on.

\n

Why Virtual Dispersive Networking?

\n
    \n
  1. Unparalleled Security: Dispersing the data over multiple different paths, eliminates the Man-in-the-Middle threat. Hackers can only obtain a small chunk of the original data on any given pathway, rendering any data obtained meaningless and nearly impossible to decrypt.
    2. \n
  2. Network Resilience: Suppose a connection is lost on any of the several pathways due to network failure,\ndata packets are then rerouted to an already existing path, or an additional path is established which leads us resulting in negligible network downtime.
    3. \n
  3. Speed / Performance: Since the data-flow is from multiple independent paths using unique methods, it increases available bandwidth and optimizes data flow on individual pathways. Therefore, speed and performance are improved.
    4. \n
\n

Conclusion

\n

Today, Dispersive Technologies' business is largely government-centric, with initial forays into commercial industries with high-value targets like banks and utilities. There are many opportunities for Dispersive technology in the cloud computing world. The cloud can host the redirects at the center of their methodology without much of a stretch. However, cloud conditions can use Dispersive to set up secure communications between clouds or between on-premise information. These hybrid cloud situations regularly rely on VPNs, which will, in general, be flaky and moderate. Dispersive Technologies have become replacement of VPNs, thereby improving the security, performance, and manageability of hybrid clouds as well as virtual private clouds.

\n

As with any industry, change can be frightening (especially when sensitive data is part of the equation), but if the organization is aware of the new developments, they can begin implementing some of these security technologies into their existing IT infrastructure and enjoy some peace of mind without worrying about future threats.

\n","frontmatter":{"date":"October 07, 2020","updated_date":null,"description":"Learn About VDN for CyberSecurity","title":"Virtual Dispersive Networking","tags":["VDN","Cyber Security"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/d2e724f66a1588268845362a0ddec71b/58556/MIM.webp","srcSet":"/static/d2e724f66a1588268845362a0ddec71b/61e93/MIM.webp 200w,\n/static/d2e724f66a1588268845362a0ddec71b/1f5c5/MIM.webp 400w,\n/static/d2e724f66a1588268845362a0ddec71b/58556/MIM.webp 800w,\n/static/d2e724f66a1588268845362a0ddec71b/210c1/MIM.webp 900w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Binay Agarwal","github":"agarwalBinay18","avatar":null}}}},{"node":{"excerpt":"React context API, In this article you will explore what is Context API, what is react context API and how to use it in your React project…","fields":{"slug":"/engineering/react-context-api/"},"html":"

React context API, In this article you will explore what is Context API, what is react context API and how to use it in your React project. The Context API is a React structure that enables you to exchange unique details and assists in solving prop-drilling from all levels of your application.

\n

What is Context API?

\n

The React Context API is a way for a React app to effectively produce global variables that can be passed around. This is the alternative to \"prop drilling\" or moving props from grandparent to child to parent, and so on. Context is also touted as an easier, lighter approach to state management using Redux.

\n

Talking about the Context APIs, they’re a (kind of) new feature added in version 16.3 of React that allows one to share state across the entire app (or part of it) lightly and with ease.

\n

React context API: How it works?

\n

React.createContext() is all you need. It returns a consumer and a provider. Provider is a component that as it's names suggests provides the state to its children. It will hold the \"store\" and be the parent of all the components that might need that store. Apart from the react context Provider, Consumer as it so happens is a component that consumes and uses the state. More information can be found on React's documentation page

\n

Context API will replace redux?

\n

No. Well, not entirely.

\n

Redux is great and came perfectly to answer the need for state management. Actually, it answered this need so well that it came to be known that you can't be a \"true\" React developer if you don't know your way around Redux.\nHowever, Redux has its disadvantages, and that's why it's important to know what Context API gives us that Redux doesn't:

\n\n

Important Note

\n

Redux is just a concept. If you are comfortable with using reducers and actions and don't find hindering than you may also create reducers and actions that wrap Context API as the store as Redux's author Dan Abramov explained in his medium article about whether Redux is always required

\n

How to use Context API?

\n

You might think to yourself: \"Well, I'm convinced. How do I implement Context API in react for my app?\" First, make sure you need it. Sometimes people use shared state across nested components instead of just passing it as props. And if you do need it you should follow these very few steps:

\n
    \n
  1. Create a folder under your app root named contexts (not required. just a convention)
    2. \n
  2. Create a file named <your context name>Context.js, e.g. userContext.js
    3. \n
  3. import and create a context like so:
    4. \n
\n
import React, { createContext } from "react";\nconst UserContext = createContext();
\n
    \n
  1. Create a component that will wrap the provider named Provider e.g. UserProvider
    \nExample using React Hooks:
    2. \n
\n
const UserProvider = ({ children }) => {\n  const [name, setName] = useState("John Doe");\n  const [age, setAge] = useState(1);\n  const happyBirthday = () => setAge(age + 1);\n  return (\n    <UserContext.Provider value={{ name, age, happyBirthday }}>\n      {children}\n    </UserContext.Provider>\n  );\n};
\n
    \n
  1. Create a higher order component to consume the context named: with e.g. withUser
    \nExample using React Hooks:
    2. \n
\n
const withUser = (Child) => (props) => (\n  <UserContext.Consumer>\n    {(context) => <Child {...props} {...context} />}\n    {/* Another option is:  {context => <Child {...props} context={context}/>}*/}\n  </UserContext.Consumer>\n);
\n

The difference between the two options above is if you want the context to be a single nested property by this name, to explode it to its properties (which in my opinion is more convenient).

\n
    \n
  1. Finally export them
    2. \n
\n
export { UserProvider, withUser };
\n
    \n
  1. And use them however you like
    \nFor example:
    2. \n
\n
ReactDOM.render(\n  <UserProvider>\n    <App />\n  </UserProvider>,\n  document.getElementById("root")\n);
\n
export default withUser(LoginForm);
\n

You'll also be able to notice I used the new \"Hooks\" feature that is shipped with React since version 16.8 to make it even neater and easier to create contexts.

\n

Final piece of advice

\n

I mentioned how people overuse Redux, and it also applies to Context API. Use it only when you need to share state between lot's of components with a lot of nesting.\nMost of the time, the data you have in one component will only be relevant to its children, so passing it in props is more indicative and nicer.

\n","frontmatter":{"date":"October 07, 2020","updated_date":null,"description":"Context API is a (kind of) new feature added in version 16.3 of React that allows one to share state across the entire app (or part of it) lightly and with ease. Let's see how to use it.","title":"React Context API: What is it and How it works?","tags":["React","Redux","Hooks"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5151515151515151,"src":"/static/6fc42d03576ecaa3bcc58404bdc7afc5/58556/react.webp","srcSet":"/static/6fc42d03576ecaa3bcc58404bdc7afc5/61e93/react.webp 200w,\n/static/6fc42d03576ecaa3bcc58404bdc7afc5/1f5c5/react.webp 400w,\n/static/6fc42d03576ecaa3bcc58404bdc7afc5/58556/react.webp 800w,\n/static/6fc42d03576ecaa3bcc58404bdc7afc5/99238/react.webp 1200w,\n/static/6fc42d03576ecaa3bcc58404bdc7afc5/ce765/react.webp 1499w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Eylon Ronen","github":"eylonronen","avatar":null}}}}]},"markdownRemark":{"excerpt":"Introduction Ever wondered how apps like Spotify, Netflix, or Slack manage seamless login experiences across devices? Many of them use JWT…","fields":{"slug":"/engineering/how-to-integrate-jwt/"},"html":"

Introduction

\n

Ever wondered how apps like Spotify, Netflix, or Slack manage seamless login experiences across devices? Many of them use JWT, or JSON Web Tokens, a compact, stateless method for securely transmitting user identity and session data across services.

\n

With JWT token authentication, identity information is embedded in a signed token, allowing you to maintain user sessions without server-side storage. This approach is highly scalable and ideal for modern architectures like SPAs, mobile apps, and microservices.

\n

In this blog, we’ll walk you through what is JWT, why use it, and how to implement JWT authentication using LoginRadius.

\n

You’ll learn what JWT is, why it’s effective, and how it works in real-world applications. We'll cover both integration methods (IDX and Direct API), generating your signing key, managing sessions, storing the JWT token securely, and applying best practices throughout.

\n

Whether you're a developer, product manager, or IAM architect, this guide offers a complete foundation for implementing JWT token authentication into your application stack.

\n

What is JWT?

\n

JSON Web Token (JWT) is an open standard (RFC 7519) used to transmit information securely between parties as a JSON object. It’s compact, self-contained, and digitally signed, making it a reliable format for authentication and authorization across modern applications.

\n

A JWT consists of three parts:

\n
    \n
  1. Header – Contains metadata like the type of token and signing algorithm (e.g., HS256).
    2. \n
  2. Payload – Stores the actual data or “claims,” such as user ID, roles, and token expiry.
    3. \n
  3. Signature – A cryptographic hash that ensures the token hasn’t been tampered with.
    4. \n
\n

Example of a token structure:

\n

<base64Header>.<base64Payload>.<signature>

\n

Why Use JWT?

\n\n

How JWT Works?

\n

\"Flowchart

\n
    \n
  1. A user logs in with credentials.
    2. \n
  2. Your app (or identity provider like LoginRadius) issues a signed JWT.
    3. \n
  3. The client stores the token and sends it with each request (usually in the Authorization header).
    4. \n
  4. The server validates the token’s signature and claims.
    5. \n
  5. If valid, access is granted — without any session stored on the backend.
    6. \n
\n

JWT simplifies identity verification, especially when you're building apps that talk to APIs or need to scale without centralized session storage.

\n

JWT Authentication with LoginRadius: Overview

\n

LoginRadius provides robust support for JWT (JSON Web Token) authentication, which allows for flexible and secure access control across different digital platforms. Whether you're building a fully custom identity flow or using a pre-built interface, the platform supports various integration approaches depending on your architecture.

\n

If you're looking to understand how to implement JWT token authentication effectively, LoginRadius offers two primary implementation models that cater to different levels of customization and control:

\n

1. IDX Implementation – JWT through a Hosted Login Page

\n

The IDX-hosted login approach enables secure, standards-compliant, JWT-based authentication without requiring you to build a custom login interface. This is a strategic option for fast, compliant, and user-friendly deployments.

\n\n

Configuration Steps:

\n
    \n
  1. Enable JWT Login
    2. \n
  2. Go to authentication configuration settings and enable JWT Login in the Admin Console.
    3. \n
\n

\"Screenshot

\n
    \n
  1. Specify your signing algorithm and expiry policy, and define your JWT Secret Key.
    2. \n
  2. Input a secure JWT signing key.
    3. \n
  3. Specify token expiry duration (e.g., 15–60 minutes)
    4. \n
  4. Select the desired algorithm —HS256 for symmetric signing (same key signs and verifies)
    5. \n
  5. RS256 for asymmetric signing, where LoginRadius securely stores the private key used to sign the JWT.
    6. \n
  6. Your app or backend service uses the public key to validate the token signature.
    7. \n
  7. LoginRadius provides a JWKS (JSON Web Key Set) endpoint to dynamically fetch and rotate public keys, ensuring trust without key exposure.
    8. \n
  8. Update IDX Template for Callback
    9. \n
  9. Modify your IDX login page template to retrieve the JWT post-login. You can access the token via redirect URL parameters or secure JavaScript callbacks.
    10. \n
\n

Example Response:

\n

{

\n

\"access_token\": \"eyJhbGciOiJIUzI1NiIsInR...\",

\n

\"expires_in\": 1800

\n

}

\n

This integration approach works best for all teams that want effective identity workflows without the complexity of building proprietary login screens, something that is crucial for customer portals, onboarding of mobile applications, and even managing access for business partners.

\n

2. Direct API Implementation – Self Managed Login

\n

If you’re building a custom login UI or working in a headless environment, LoginRadius lets you generate and handle JWTs directly through its Authentication APIs. Here’s how you can programmatically perform token authentication using the classic method:

\n\n

Configuration Steps:

\n

Step 1: Authenticate via API:

\n\n

Include the user’s credentials (email + password) in the request body.

\n

Step 2: Get JWT in Response

\n\n

{

\n

\"access_token\": \"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...\",

\n

\"expires_in\": 3600

\n

}

\n

Step 3: JWT Decoding and Validation

\n\n

Step 4: Set Custom Claims (Optional)

\n

With LoginRadius, it is possible to customize the payload to include user roles and/or any additional metadata. You can set custom JWT claims on the Admin Console.

\n

With this method, you have complete customization over login flows while using LoginRadius to issue signed JWTs for user session management.

\n

NOTE- With either method, LoginRadius ensures that JWTs are securely signed, optionally short-lived, and compatible with standard token validation libraries, making integration seamless for everyone.

\n

To get started with JWT implementation, you can read our complete developer documentation.

\n

Hosted Login vs Direct API

\n

\"Illustration

\n

What is Session Management and How It Works with JWT

\n

Session management is how your app keeps track of a user after they log in so they don’t have to prove who they are with every request.

\n

In traditional apps, sessions are stored on the server using session IDs. Every time a request comes in, the server checks that session ID to verify the user.

\n

In modern apps, especially SPAs and APIs, JWTs are used to manage sessions without needing server-side storage; this is called stateless session management. The token itself carries the user’s identity, roles, and expiration details. As long as the token is valid, the user stays logged in.

\n

Good session management ensures:

\n\n

How LoginRadius Handles Session Management with JWT:

\n
    \n
  1. \n

    User Logs In

    \n\n
    2. \n
  2. \n

    Client Stores the Token

    \n\n
    3. \n
  3. \n

    Access Token Expiry

    \n\n
    4. \n
  4. \n

    Token Renewal

    \n\n
    5. \n
  5. Logout and Token Revocation Strategy
    6. \n
\n

When the user logs out, both the access token and refresh token should be cleared from client storage.

\n\n

Combining these strategies gives you greater control over token misuse and enables a robust, enterprise-grade logout flow.

\n

\"illustration

\n

How to Store JWT Tokens?

\n

When you implement JWT-based authentication, the client (browser or mobile app) needs a way to store the access token and, optionally, the refresh token after they are issued by the authentication server. This stored token is then attached to every subsequent request to prove the user's identity.

\n

Choosing where to store the JWT is a crucial security decision. The most common storage options are:

\n\n

Each option has trade-offs between security, accessibility, and persistence, and the right choice depends on your application's architecture and threat model.

\n

Recommended Storage Strategy

\n\n

Best Practices for Storing JWTs

\n
    \n
  1. Never store sensitive tokens (like refresh tokens) in localStorage or sessionStorage.
    2. \n
  2. Use Secure and HttpOnly flags with cookies to prevent JavaScript access and ensure transmission only over HTTPS.
    3. \n
  3. Set the SameSite=Strict or Lax attribute on cookies to protect against CSRF.
    4. \n
  4. Use short-lived access tokens and rotate refresh tokens regularly.
    5. \n
  5. Implement CSP (Content Security Policy) to reduce XSS risk.
    6. \n
  6. Avoid storing any tokens in frontend code (e.g., hardcoded in JS files).
    7. \n
\n

Conclusion

\n

JWT authentication with LoginRadius offers a modern, stateless approach to managing sessions across distributed systems. The IDX integration is ideal for rapid deployment, while the Direct API model is best for organizations needing deep customization and integration flexibility.

\n

With robust token signing, refresh capabilities, and centralized control, LoginRadius provides a future-ready foundation for secure, scalable identity architecture. Contact us to know more about JWT authentication and implementation guide.

\n

FAQs

\n

1. What is JWT authentication used for?

\n

A: JWT authentication securely verifies user identities, enabling stateless session management across web, mobile apps, and microservices without server-side session storage.

\n

2. How does LoginRadius simplify JWT integration?

\n

A: LoginRadius simplifies JWT integration by offering hosted IDX login pages and direct API-based authentication methods, enabling rapid deployment and deep customization.

\n

3. Is JWT authentication secure?

\n

A: Yes, JWT authentication is secure when implemented with best practices like short-lived tokens, secure storage methods, signature validation, and refresh token rotation.

\n

4. Can JWT tokens be revoked with LoginRadius?

\n

A: Yes, LoginRadius allows revocation of JWT refresh tokens explicitly, and supports strategies like short-lived tokens and key rotation to manage token lifecycles securely.

\n","frontmatter":{"date":"April 15, 2025","updated_date":null,"description":"Discover JWT (JSON Web Token) authentication, its advantages, and how to integrate it seamlessly using LoginRadius' hosted IDX and Direct API methods for secure, scalable identity management.","title":"JWT Authentication with LoginRadius: Quick Integration Guide","tags":["JWT","JSON Web Token","Authentication","Authorization"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":0.7782101167315175,"src":"/static/4cedb7829f98208cbc6d5a9aea4e983d/58556/how-to-integrate-jwt.webp","srcSet":"/static/4cedb7829f98208cbc6d5a9aea4e983d/61e93/how-to-integrate-jwt.webp 200w,\n/static/4cedb7829f98208cbc6d5a9aea4e983d/1f5c5/how-to-integrate-jwt.webp 400w,\n/static/4cedb7829f98208cbc6d5a9aea4e983d/58556/how-to-integrate-jwt.webp 800w,\n/static/4cedb7829f98208cbc6d5a9aea4e983d/1cc9f/how-to-integrate-jwt.webp 896w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Kundan Singh","github":null,"avatar":null}}}},"pageContext":{"limit":6,"skip":150,"currentPage":26,"type":"//engineering//","numPages":53,"pinned":"5c425581-f474-5ae9-abe7-cf5342db2aaa"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}