Breaking News: Grepper is joining You.com. Read the official announcement!

searching for fuzzy duplicates dataframe pandas

Mahmoud Shams El Din answered on March 7, 2024 Popularity 1/10 Helpfulness 1/10

answer searching for fuzzy duplicates dataframe pandas

searching for fuzzy duplicates dataframe pandas

Comment

import pandas as pd
from fuzzywuzzy import fuzz

# Custom function for finding the fuzzy duplicates
def find_fuzzy_duplicates(dataframe, column, threshold):
    duplicates = []
    # Iterate over each row in the DataFrame
    for i in range(len(dataframe)):
        # Compare the current row with subsequent rows
        for j in range(i+1, len(dataframe)):
            # Calculate the similarity score using fuzz.ratio
            similarity = fuzz.ratio(dataframe[column][i], dataframe[column][j])
            # Check if the similarity score meets the threshold
            if similarity >= threshold:
                # Add the duplicate rows to the duplicates list
                duplicates.append(dataframe.iloc[[i, j]])
    # Concatenate the list of DataFrames into a single DataFrame
    if duplicates:
      duplicates_df = pd.concat(duplicates)
      return duplicates_df
    else:
      return pd.DataFrame()

# Create a DataFrame
data = {
      'StudentName': ['Mark', 'Ali', 'Bob', 'John', 'Johny', 'Mark'],
      'Score': [45, 65, 76, 44, 39, 45]
}
df = pd.DataFrame(data)

# Define the threshold value
threshold = 70

# Find fuzzy duplicates in the 'StudentName' column
fuzzy_duplicates = find_fuzzy_duplicates(df, 'StudentName', threshold)
print("Fuzzy duplicates:")
print(fuzzy_duplicates.to_string(index=False))

Popularity 1/10 Helpfulness 1/10 Language python

Source: stackabuse.com

Tags: dataframe fuzzy pandas python

Link to this answer
Share Copy Link

Contributed on Mar 07 2024

Mahmoud Shams El Din

0 Answers Avg Quality 2/10

searching for fuzzy duplicates dataframe pandas

Contents

More Related Answers

searching for fuzzy duplicates dataframe pandas

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.