Remove Duplicates from list in python

In this tutorial we will be focusing on how to remove the duplicates in the python list. We will be trying out different ways to deduplicate the elements of the list in python. one of them being to preserve the order after removing duplicates using set() method and other is to deduplicate the list without preserving order.

• Remove Duplicates from a Python list (preserving the order).
• Remove Duplicates from a Python list (preserving the order) using list comprehension
• Remove Duplicates from python list (preserving order) using collections.OrderedDict.fromkeys()
• Deduplicate python list (preserving order) using List comprehension and enumerate()
• Deduplicate Python list (not preserving the order) using a set()

Remove Duplicates from a Python list (preserving the order):

We will be manually removing duplicate list as shown below, First an empty list is created and with the help of for loop we will be appending the element to the list which are not already in the list as shown below, so it will result in the list which duplicates are removed and the order is preserved

# Remove Duplicates from a Python list (preserving the order) using a set()

duplicated_list = [1,1,4,3,2,4,1,2,3,4]

print ("The original list is : " +  str(duplicated_list))

# using naive method

# to remove duplicated

# from list

res = []

for i in duplicated_list:

if i not in res:

res.append(i)


print ("The list after removing duplicates : " + str(res))

so, the output will be,

output:

Remove Duplicates from a Python list (preserving the order) using list comprehension:

We will be removing duplicate list using the list comprehension, First an empty list is created and with the list comprehension we will be appending the element to the list which are not already in the list as shown below, so it will result in the list which duplicates are removed and the order is preserved

# Remove Duplicates from a Python list  (preserving the order) using list comprehension

duplicated_list = [1,1,4,3,2,4,1,2,3,4]

print ("The original list is : " +  str(duplicated_list))


# using list comprehension

# to remove duplicated

# from list

res = []

[res.append(x) for x in duplicated_list if x not in res]

print ("The list after removing duplicates : " + str(res))

so, the output will be,

output:

Remove Duplicates from python list (preserving order) using collections.OrderedDict.fromkeys(): 

We will be removing duplicate list in python using ordered dictionary from collections, we will be passing the duplicate list to the ordered dictionary it will result in the list which duplicates are removed and the order is preserved

# Remove Duplicates from python list (preserving order) using collections.OrderedDict.fromkeys()

from collections import OrderedDict

# initializing list

duplicated_list = [1,1,4,3,2,4,1,2,3,4]

print ("The original list is : " +  str(duplicated_list))

# using collections.OrderedDict.fromkeys()

# to remove duplicated

# from list

res = list(OrderedDict.fromkeys(duplicated_list))

print ("The list after removing duplicates : " + str(res))

so, the output will be,

output:

Deduplicate python list (preserving order) using List comprehension and enumerate()

We will be deduplicating the python list using enumerate inside the list comprehension, we will be passing the duplicate list to enumerate() inside the list comprehension, which will result in the list which duplicates are removed and the order is preserved

# Remove Duplicates from python list (preserving order) using List comprehension + enumerate()

duplicated_list = [1,1,4,3,2,4,1,2,3,4]

print ("The original list is : " +  str(duplicated_list))


# using list comprehension + enumerate()

# to remove duplicated

# from list

res = [i for n, i in enumerate(duplicated_list) if i not in duplicated_list[:n]]

print ("The list after removing duplicates : " + str(res))

so, the output will be,

output:

Deduplicate a Python list (not preserving the order) using a set():

We will be deduplicating the python list using simple set() function, we will be passing the duplicate list to set() which will result in the set which duplicates are removed and the order is not preserved and will be sorted in the increasing order by default and finally the set is converted to list() as shown below.

# # Remove Duplicates from a Python list  (not preserving the order) using a set()

duplicated_list = [1,1,4,3,2,4,1,2,3,4]

print ("The original list is : " +  str(duplicated_list))

deduplicated_list = list(set(duplicated_list))

print("The list after removing duplicates : " + str(deduplicated_list))

so, the output will be,

output: