How To Represent A Data For Collaborative Filtering

Recommendation systems have a wide range of applications beyond the domains. Building a proficient recommender system suitable to the business organization requirement is ever a claiming. A good organization tin only be developed when there is a good understanding of its working. In this article, we volition talk over how to build a recommender system, especially collaborative filtering based, from scratch. We will start with the random data and build a recommender system to generate recommendations. The major points to be discussed in this article are listed beneath.

Table of contents

What is collaborative filtering?
Use of correlation
Implementation of item-based collaborative filtering
Implementation of user-based collaborative filtering

Let'southward start with understanding collaborative filtering.

What is collaborative filtering?

Collaborative filtering can be considered equally a technique to provide recommendations in a recommendation system or engine. In a basic sense, we can say that it is a way to find similarities between users and items. Utilizing it we can summate ratings based on ratings of similar users or like items.

Recommendation systems based on collaborative filtering can exist categorized in the post-obit ways:

Item-based: This blazon of recommendation system helps in finding similarities between the items or products. This is done past generating data of the number of users who bought two or more than items together and if the organization finds a high correlation and so it assumes similarity between products. For example, in that location are ii products 10 and Y that are highly correlated when a user buys X, the arrangement recommends ownership Y also.
User-based: This type of system helps in finding like users based on their nature of item choice. For example, ane user uses a helmet, knee joint guard, and elbow guard, and the 2nd uses only a helmet and elbow guard at the fourth dimension of bike riding the user-based recommendation organization will recommend the second user apply a knee guard.

In this commodity, we volition endeavour to understand collaborative filtering from scratch. Offset, nosotros will create an example of data and we will effort to find similarities between items. Finding similarity between items is related to finding the correlation between items based on the information that nosotros have. Earlier going for implementation nosotros are required to understand what is the correlation.

Employ of correlation

Correlation can be considered as the human relationship between two variables. This can be of three types positive, negative or neutral. If two variables are positively correlated then we can say changes in ane variable in a positive or negative direction tin can provide a change in the second variable in a positive or negative direction.

If the correlation is negative and so a alter in ane variable can crusade a alter in the reverse direction. If the variables are neutrally correlated and so changes in 1 variable practice not cause a alter in the other. The measurement of correlation can be done using the correlation coefficient.

Calculation of correlation coefficient can be done past start computing the covariance of the variable then dividing by the covariance quantity by the product of those variables' standard deviations.

Mathematically,

Where,

r = correlation coefficient

x_i = values of ten variable in a sample

x = hateful of the values of the x variable

y_i = values of y variable in a sample

y = hateful of the values of the y variable

There are many types of correlation coefficients used in statistical analysis, nosotros mainly apply Pearson correlation for recommendation systems because it is a measure of the forcefulness and direction of the linear relationship between two variables. Let's move toward the implementation of a recommendation system.

Implementation of item-based collaborative filtering

1. Importing library

          import pandas every bit pd import numpy every bit np import matplotlib.pyplot as plt

two. Dataset

In this article, we are going to implement a recommender organization using the collaborative filtering approach for that purpose we volition piece of work on simple information. Let'south say that we have some users, products, and ratings of that production given by the user. Nosotros can make such a data set using the following codes;

          data2 = {'user_id':[1, 2, 3, 1, two],          'product_id':[1, two, 1,ii,3],         'product_name':['product_1', 'product_2', 'product_1','product_2','product_3'],          'rating':[three,3,3,2,2]          }   items_df = pd.DataFrame(data2) items_df

Output:

Here nosotros can see that we have data of iii users and 3 products.

3. Pivot tabular array

Permit'due south create a pivot table using this information based on user_id and product_name.

          pin = pd.pivot_table(items_df,values='rating',columns='product_name',index='user_id') pivot

Output:

Here in the to a higher place output, nosotros can see our pivot table. This table format can exist used for computing correlation. Every bit the correlation will be college we can employ them as our recommendation.

Generating recommendation

To sympathise the process clearly, nosotros have used a very simple dataset and nosotros tin can say by seeing the above table that products i, ii, and 3 have like ratings and production ane has got two reviews. So in that location may be a possibility of products 2 and 3 to be recommended with product one. Allow's check our results.

          print('recommended product with product_2:') print( pivot.corr()['product_2'].sort_values(ascending=False).iloc[ane:two])

Output:

Using the in a higher place lines of codes, we calculate the correlation between products and sort the values. Then nosotros printed 1 value and found that our organisation is recommending us to buy or use product 2 with product 1.

Implementation of user-based collaborative filtering

In the higher up section, nosotros take gone through the process of making data and pivot tables. In this section, we will use like data for implementing user-based collaborative filtering.

1. Pivot tabular array

Allow's first with making a pivot table for user-based collaborative filtering. For this purpose, we are required to inverse our older pivot tabular array which ways now nosotros are making a pivot table based on users as columns.

          pivot1 = pd.pivot_table(items_df,values='rating',columns='user_id',alphabetize='product_name') pivot1

Output:

In the to a higher place table, we can see that nosotros take user-id as a column and products as a row.

2. Generating recommendation

In this department, we will find like users based on their provided ratings. So that we can filter out users and can give similar recommendations of different items or we tin also give recommendations to a user based on similar user history.

          print('similar users to user_2:') print( pivot1.corr()[2].sort_values(ascending=True).iloc[ane:2])

Output:

In the to a higher place output, we can see that user 1st is more similar to user 2nd, and information technology is because they have provided almost similar ratings in our main dataset.

Final words

In this commodity, we take gone through the bones intuitions behind making recommendation systems using collaborative filtering techniques and nosotros learned this approach from scratch. More than advanced coverage on collaborative filtering can be found hither where we can see how information technology tin exist performed with a large dataset.

The codes used in the above implementation tin can be constitute here.