Home > python > Remove duplicates from a list AND keep the original order of the elements

Remove duplicates from a list AND keep the original order of the elements

Problem
You have a list of elements. You want to remove the duplicates but you want to keep the original order of the elements too. Example:

input: apple, fruit, dog, fruit, cat, apple, dog, cat

output: apple, fruit, dog, cat

Solution

def remove_duplicates(li):
    my_set = set()
    res = []
    for e in li:
        if e not in my_set:
            res.append(e)
            my_set.add(e)
    #
    return res

The trick list(set(li)) is not acceptable in this case because elements are unordered in a set.

Advertisements
  1. January 15, 2015 at 02:17

    Why use my_set? You could use result by changing line 5 to ‘if e not in res:’.

    • January 15, 2015 at 08:31

      That would work but would be slow. When you check if an element is in a list, Python uses linear search. The bigger the list is, the slower it becomes. However, if you use a set, you get the answer much faster since a set uses a hash technique. So I use a set here to speed the process up.

  1. No trackbacks yet.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: