Identifying and Removing Duplicates in Python with pandas

Finding Duplicate Rows

Pandas provides two main methods for identifying duplicate rows in a DataFrame:

1. duplicated() method: This method returns a Boolean Series indicating whether each row is a duplicate (True) or not (False). You can control how duplicates are identified using the keep parameter:

   • keep='first': Marks duplicates as True except for the first occurrence.
   • keep=False (default): Marks all duplicates as True.

   Here's an example:

   import pandas as pd
   
   data = {'Name': ['Alice', 'Bob', 'Charlie', 'Alice', 'Bob'],
           'Age': [25, 30, 22, 25, 30]}
   df = pd.DataFrame(data)
   
   duplicates = df.duplicated(keep=False)  # Find all duplicates
   print(duplicates)
   

   This will output:

   0    False
   1    False
   2    False
   3     True
   4     True
   dtype: bool
   

2. drop_duplicates() method: This method creates a new DataFrame by removing duplicate rows based on specified columns. You can control the following aspects:

   • subset (optional): A list of column names to consider for identifying duplicates. If omitted, all columns are used.
   • keep (same as in duplicated()): Controls which duplicates are kept (first, last, or none).

   df_no_duplicates = df.drop_duplicates()
   print(df_no_duplicates)
   

   This will output a DataFrame containing only the unique rows (without duplicates):

     Name  Age
   0  Alice   25
   1    Bob   30
   2  Charlie   22
   

Key Points:

• Use duplicated() to identify which rows are duplicates and potentially create a list of duplicate indices.
• Use drop_duplicates() to create a new DataFrame with duplicates removed.
• Specify columns using subset in drop_duplicates() if you only want to consider duplicates based on specific columns.

Additional Considerations:

• If your DataFrame has a hierarchical index, you can use the keep='level' option in duplicated() to keep duplicates at a specific level.
• For very large DataFrames, consider using the inplace=True option in drop_duplicates() to modify the DataFrame in-place, potentially improving performance.

By effectively using these methods, you can efficiently manage duplicate rows in your pandas DataFrames.

Finding All Duplicates (duplicated() method):

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'Alice', 'David'],
        'Age': [25, 30, 22, 25, 28]}
df = pd.DataFrame(data)

# Find all duplicates (including the first occurrence)
duplicates = df.duplicated(keep=False)
print("All duplicates (including first occurrence):")
print(duplicates)

# Find duplicates except for the first occurrence
duplicates_except_first = df.duplicated(keep='first')
print("\nDuplicates except for the first occurrence:")
print(duplicates_except_first)

# Find duplicates except for the last occurrence
duplicates_except_last = df.duplicated(keep='last')
print("\nDuplicates except for the last occurrence:")
print(duplicates_except_last)

All duplicates (including first occurrence):
0    False
1    False
2    False
3     True
4    False
dtype: bool

Duplicates except for the first occurrence:
0    False
1    False
2    False
3     True
4    False
dtype: bool

Duplicates except for the last occurrence:
0    False
1    False
2    False
3     True
4     True
dtype: bool

# Remove duplicates based on all columns
df_no_duplicates = df.drop_duplicates()
print("\nDataframe without duplicates (all columns):")
print(df_no_duplicates)

# Remove duplicates based on specific columns ('Name')
df_no_duplicates_name = df.drop_duplicates(subset=['Name'])
print("\nDataframe without duplicates ('Name' column):")
print(df_no_duplicates_name)

Dataframe without duplicates (all columns):
   Name  Age
0  Alice   25
1    Bob   30
2  Charlie   22
4   David   28

Dataframe without duplicates ('Name' column):
   Name  Age
0  Alice   25
1    Bob   30
2  Charlie   22
4   David   28

These examples demonstrate how to use duplicated() and drop_duplicates() with different keep options and subset parameters to achieve various results.

Using a loop and set:

This method iterates through the DataFrame and adds unique elements to a set. If an element already exists in the set, it's considered a duplicate. This approach is generally less efficient for large DataFrames compared to vectorized methods like duplicated().

def find_duplicates_loop(df):
  unique_elements = set()
  duplicates = []
  for index, row in df.iterrows():
    if tuple(row.tolist()) not in unique_elements:
      unique_elements.add(tuple(row.tolist()))
    else:
      duplicates.append(index)
  return duplicates

data = {'Name': ['Alice', 'Bob', 'Charlie', 'Alice', 'David'],
        'Age': [25, 30, 22, 25, 28]}
df = pd.DataFrame(data)

duplicates = find_duplicates_loop(df.copy())
print("Duplicates using loop and set:")
print(duplicates)

Using groupby() and counting occurrences:

This method groups rows by their values and checks if the count is greater than 1 to identify duplicates. It's slightly less efficient than duplicated() but can be useful if you need additional information like the count of duplicates.

duplicates = df.groupby(df.columns.tolist()).filter(lambda x: x.shape[0] > 1)
print("Duplicates using groupby and count:")
print(duplicates)

Remember that these alternate methods can be less performant for large DataFrames. For most cases, duplicated() and drop_duplicates() are the preferred solutions due to their optimized vectorized operations. Choose the approach that best suits your specific needs and dataset size.