Deleting duplicate rows in excel

One of the most common data cleaning tasks in Excel includes removing duplicate or redundant records/rows.

While removing whole rows that are duplicate is straightforward, it’s a little trickier when you’re trying to remove entire duplicate rows based on one or more columns.

In this tutorial, we are going to look at three easy ways to remove duplicate rows based on one column in Excel

Using the ‘Remove Duplicates’ Feature

This method is quite straightforward and the most commonly used way to remove duplicate rows in Excel.

Suppose you have a dataset as shown below, and you want to remove all the duplicate records based on Column A.

In this data, you can see that I have multiple instances of ‘Sleeping Bag’ and ‘Karaoke machine’ in column A, and I only want to retain the first row of each instance and get rid of all the others.

To remove all duplicate rows from our sample dataset (shown in the figure above), follow the steps listed below:

1. Select the entire dataset, along with the column headers.
2. From the Data tab, under the Data Tools group select the Remove Duplicates button.

1. This will open the Remove Duplicates dialog box.

1. If your selection in step 1 included column headers, then make sure the ‘My data has headers’ checkbox is checked.

1. Click on the Unselect All button, to uncheck all the checkboxes under ‘Columns’.

1. Now check the column based on which you want to remove the duplicate rows. In our example, we want to remove duplicate rows based on Product Name, so we will simply check the box next to its heading.

1. Click OK to close the Remove Duplicates dialog box.
2. You should see a notification telling you how many duplicate rows were removed and how many unique rows have been retained, as shown in the image below:

1. Click OK

After applying the above steps on our sample dataset, here’s how the dataset should look:

Notice from the above image that exactly one instance corresponding to each product has been retained while all other duplicate rows have been removed.

In other words, this method does not get rid of all duplicate rows. It keeps just one copy and removes all others.

Note: If you want to remove duplicates based on more than one column, you can check the boxes next to the columns you want to include in step 6.

Suppose you have a dataset as shown below, and you want to remove all the duplicate records based on Column A.

You can do this easily using a short VBA code as well. This method is useful when you have to do this quite often and don’t want to follow many steps.

So instead, you can have the code and add it to the Quick Access Toolbar, so that you can access it with a single click.

If you are comfortable with a little coding, then you can use this method to remove duplicate rows based on a single column.

Even if you’re not that keen on coding, you can just copy-paste the following code:

Sub Delete_duplicate_rows()
Dim Rng As Range
Set Rng = Selection
Rng.RemoveDuplicates Columns:=Array(1), Header:=xlYes
End Sub

This code uses a VBA built-in command for removing duplicates in list-objects.

It takes the selected range, as well as the columns that you want to base the duplicate removal on.

In the above code, we specified this as Column 1 or the Product Names column. The code then removes all the rows from the range that contain duplicate product names.

Note: You cannot undo changes made by this VBA script, so we suggest you keep a backup copy of your dataset before running the code.

To enter the above code, copy it and paste it in your developer window.

Here’s how to do this:

1. From the Developer tab, select Visual Basic.

1. Once your VBA window opens, you will see all your files and folders in the Project Explorer on the left side.

1. Make sure ‘ThisWorkbook’ is selected under the VBA project with the same name as your Excel workbook.
2. Click Insert->Module. A new module window should open up.

1.  Now you can start coding. Copy the above script and paste into the module window.

1. Close the VBA window. 

If you can’t see the Developer ribbon, from the File menu, go to Options. Select Customize Ribbon and check the Developer option from Main Tabs. Finally, Click OK.

Your macro is now ready to use.

Note: If you want to remove duplicates based on more than one column, you can specify the column numbers in the last line of the code. So if you want to search based on columns 1 and 2, your last line would be:

Rng.RemoveDuplicates Columns:=Array(1,2), Header:=xlYes

Running the Macro

To run your macro, do the following:

1. Select the range of cells that you want to work with. In our case, select A1:C9
2. Select the Developer tab.
3. Click on the Macros button (under the Code group).

1. This will open the Macro window, where you will find the names of all the macros that you have created so far.
2. Select the macro (or module) named ‘Delete_duplicate_rows’.

1. Click Run.
2. Click OK.

All the rows containing duplicates should now be removed.

Explanation of the Code

The above code simply took the selected range of cells and assigned it to a variable, Rng.

The RemoveDuplicates command then took this range, searched for duplicates in Column 1 (as we had specified Columns:=Array(1)), and then removed all copies of rows containing the same value in Column 1.

Note: We specified Header:=xlYes to tell Excel that the first row of our selected range contains column headings.

Using Filters and the COUNTIF Function to Remove Duplicate Rows based on one Column

You can also use a formula to help you find duplicate values in your data. This method involves two steps. 

Suppose you have the same dataset as shown below, and you want to remove all the duplicate records based on Column A.

First, we will use the COUNTIF function to count the first occurrence of a product as 1, its second occurrence as 2, and so on.

Then, we will use this result to filter out those rows that occur for the second time or more.

The result is the set of all duplicate rows. We can then delete these visible rows and remove the filter to obtain rows containing only unique product names.

The COUNTIF function helps count cells in a range that satisfy a given condition. The syntax for the function is as follows:

= COUNTIF (range, condition) 

Here,

• range is the range of cells containing the data you want the function to work on (or count)
• condition is the condition that you want satisfied in order to include a cell in the count.

So, if you want to find the number of times the value at cell reference A2 appears in the range A2:A9, use the function as follows:

=COUNTIF(A2:A9,A2)

But if we want to count the number of times the value appears in the range A2 to the current row, then we can use the function as follows:

=COUNTIF($A$2:$A2,A2)

Note: We locked the starting cell reference in the first parameter with ‘$’ because we don’t want this reference to change when the formula is copied to the other cells.

Here, we want the cell reference $A$2 to remain constant, irrespective of which row the formula is copied to. In this way, we can count how many times the value appears up to the current row. 

This will ensure that the first occurrence of the cell value is counted as 1, the second occurrence is counted as 2, and so on.

To remove all duplicate rows from our sample dataset, follow the steps listed below:

1. Create a new column with the heading ‘Count’. Next to the dataset.

1. In the first cell of the column, type the formula: =COUNTIF($A$2:$A2,A2).
2. Copy this formula to the rest of the cells in the column by dragging down the cell’s fill handle. This will display the number of times each product name appears in column A.

1. Now we need to filter the dataset to show only the duplicate rows.Select the entire dataset, along with the column headers.
2. From the Data tab, under the Sort & Filter group select the Filter button.

1. You should now see arrows next to each column’s heading.
2. Click on the arrow next to the ‘Count’ heading.

1. From the dropdown menu that appears, select Number Filters->greater than.

1. This will open the Custom Autofilter dialog box. Type ‘1’ in the input box next to “is greater than”.

1. Click OK.
2. You should see all the duplicate rows only.

1. Delete these filtered rows by selecting them, right-clicking and choosing Delete Rows from the context menu that appears.

1. Remove the filter now by clicking on the Filter button again (under the Data tab).

1. You should now be left with all the unique rows of the original dataset.

You can now get rid of the Count column.

In this tutorial, we looked at three ways in which you can remove duplicate rows based on one or more columns.

You can feel free to choose the method that best suits your requirement at hand.

We hope this was helpful. 

Other Excel tutorials you may also like:

• How to Find Duplicates in Excel (Conditional Formatting/ Count If/ Filter)
• How to Remove Blank Columns in Excel? (Formula + VBA)
• Duplicate Sheet in Excel (Shortcuts + VBA)
• How to Count Negative Numbers in Excel
• How to Count How Many Times a Word Appears in Excel
• How to Select Rows with Specific Text in Excel
• How to Delete Filtered Rows in Excel (with and without VBA)
• Get Unique Values from a Column in Excel
• How to Count Unique Values in Excel (Formulas)

Duplicate values in your data can be a big problem! It can lead to substantial errors and over estimate your results.

But finding and removing them from your data is actually quite easy in Excel.

In this tutorial, we are going to look at 7 different methods to locate and remove duplicate values from your data.

Video Tutorial

What Is A Duplicate Value?

Duplicate values happen when the same value or set of values appear in your data.

For a given set of data you can define duplicates in many different ways.

In the above example, there is a simple set of data with 3 columns for the Make, Model and Year for a list of cars.

1. The first image highlights all the duplicates based only on the Make of the car.
2. The second image highlights all the duplicates based on the Make and Model of the car. This results in one less duplicate.
3. The second image highlights all the duplicates based on all columns in the table. This results in even less values being considered duplicates.

The results from duplicates based on a single column vs the entire table can be very different. You should always be aware which version you want and what Excel is doing.

Find And Remove Duplicate Values With The Remove Duplicates Command

Removing duplicate values in data is a very common task. It’s so common, there’s a dedicated command to do it in the ribbon.

Select a cell inside the data which you want to remove duplicates from and go to the Data tab and click on the Remove Duplicates command.

Excel will then select the entire set of data and open up the Remove Duplicates window.

1. You then need to tell Excel if the data contains column headers in the first row. If this is checked, then the first row of data will be excluded when finding and removing duplicate values.
2. You can then select which columns to use to determine duplicates. There are also handy Select All and Unselect All buttons above you can use if you’ve got a long list of columns in your data.

When you press OK, Excel will then remove all the duplicate values it finds and give you a summary count of how many values were removed and how many values remain.

This command will alter your data so it’s best to perform the command on a copy of your data to retain the original data intact.

Find And Remove Duplicate Values With Advanced Filters

There is also another way to get rid of any duplicate values in your data from the ribbon. This is possible from the advanced filters.

Select a cell inside the data and go to the Data tab and click on the Advanced filter command.

This will open up the Advanced Filter window.

1. You can choose to either to Filter the list in place or Copy to another location. Filtering the list in place will hide rows containing any duplicates while copying to another location will create a copy of the data.
2. Excel will guess the range of data, but you can adjust it in the List range. The Criteria range can be left blank and the Copy to field will need to be filled if the Copy to another location option was chosen.
3. Check the box for Unique records only.

Press OK and you will eliminate the duplicate values.

Advanced filters can be a handy option for getting rid of your duplicate values and creating a copy of your data at the same time. But advanced filters will only be able to perform this on the entire table.

Find And Remove Duplicate Values With A Pivot Table

Pivot tables are just for analyzing your data, right?

You can actually use them to remove duplicate data as well!

You won’t actually be removing duplicate values from your data with this method, you will be using a pivot table to display only the unique values from the data set.

First, create a pivot table based on your data. Select a cell inside your data or the entire range of data ➜ go to the Insert tab ➜ select PivotTable ➜ press OK in the Create PivotTable dialog box.

With the new blank pivot table add all fields into the Rows area of the pivot table.

You will then need to change the layout of the resulting pivot table so it’s in a tabular format. With the pivot table selected, go to the Design tab and select Report Layout. There are two options you will need to change here.

1. Select the Show in Tabular Form option.
2. Select the Repeat All Item Labels option.

You will also need to remove any subtotals from the pivot table. Go to the Design tab ➜ select Subtotals ➜ select Do Not Show Subtotals.

You now have a pivot table that mimics a tabular set of data!

Pivot tables only list unique values for items in the Rows area, so this pivot table will automatically remove any duplicates in your data.

Find And Remove Duplicate Values With Power Query

Power Query is all about data transformation, so you can be sure it has the ability to find and remove duplicate values.

Select the table of values which you want to remove duplicates from ➜ go to the Data tab ➜ choose a From Table/Range query.

Remove Duplicates Based On One Or More Columns

With Power Query, you can remove duplicates based on one or more columns in the table.

You need to select which columns to remove duplicates based on. You can hold Ctrl to select multiple columns.

Right click on the selected column heading and choose Remove Duplicates from the menu.

You can also access this command from the Home tab ➜ Remove Rows ➜ Remove Duplicates.

= Table.Distinct(#"Previous Step", {"Make", "Model"})

If you look at the formula that’s created, it is using the Table.Distinct function with the second parameter referencing which columns to use.

Remove Duplicates Based On The Entire Table

To remove duplicates based on the entire table, you could select all the columns in the table then remove duplicates. But there is a faster method that doesn’t require selecting all the columns.

There is a button in the top left corner of the data preview with a selection of commands that can be applied to the entire table.

Click on the table button in the top left corner ➜ then choose Remove Duplicates.

= Table.Distinct(#"Previous Step")

If you look at the formula that’s created, it uses the same Table.Distinct function with no second parameter. Without the second parameter, the function will act on the whole table.

Keep Duplicates Based On A Single Column Or On The Entire Table

In Power Query, there are also commands for keeping duplicates for selected columns or for the entire table.

Follow the same steps as removing duplicates, but use the Keep Rows ➜ Keep Duplicates command instead. This will show you all the data that has a duplicate value.

Find And Remove Duplicate Values Using A Formula

You can use a formula to help you find duplicate values in your data.

First you will need to add a helper column that combines the data from any columns which you want to base your duplicate definition on.

= [@Make] & [@Model] & [@Year]

The above formula will concatenate all three columns into a single column. It uses the ampersand operator to join each column.

= TEXTJOIN("", FALSE , CarList[@[Make]:[Year]])

If you have a long list of columns to combine, you can use the above formula instead. This way you can simply reference all the columns as a single range.

You will then need to add another column to count the duplicate values. This will be used later to filter out rows of data that appear more than once.

= COUNTIFS($E$3:E3, E3)

Copy the above formula down the column and it will count the number of times the current value appears in the list of values above.

If the count is 1 then it’s the first time the value is appearing in the data and you will keep this in your set of unique values. If the count is 2 or more then the value has already appeared in the data and it is a duplicate value which can be removed.

Add filters to your data list.

• Go to the Data tab and select the Filter command.
• Use the keyboard shortcut Ctrl + Shift + L.

Now you can filter on the Count column. Filtering on 1 will produce all the unique values and remove any duplicates.

You can then select the visible cells from the resulting filter to copy and paste elsewhere. Use the keyboard shortcut Alt + ; to select only the visible cells.

Find And Remove Duplicate Values With Conditional Formatting

With conditional formatting, there’s a way to highlight duplicate values in your data.

Just like the formula method, you need to add a helper column that combines the data from columns. The conditional formatting doesn’t work with data across rows, so you’ll need this combined column if you want to detect duplicates based on more than one column.

Then you need to select the column of combined data.

To create the conditional formatting, go to the Home tab ➜ select Conditional Formatting ➜ Highlight Cells Rules ➜ Duplicate Values.

This will open up the conditional formatting Duplicate Values window.

1. You can select to either highlight Duplicate or Unique values.
2. You can also choose from a selection of predefined cell formats to highlight the values or create your own custom format.

Warning: The previous methods to find and remove duplicates considers the first occurrence of a value as a duplicate and will leave it intact. However, this method will highlight the first occurrence and will not make any distinction.

With the values highlighted, you can now filter on either the duplicate or unique values with the filter by color option. Make sure to add filters to your data. Go to the Data tab and select the Filter command or use the keyboard shortcut Ctrl + Shift + L.

1. Click on the filter toggle.
2. Select Filter by Color in the menu.
3. Filter on the color used in the conditional formatting to select duplicate values or filter on No Fill to select unique values.

You can then select just the visible cells with the keyboard shortcut Alt + ;.

Find And Remove Duplicate Values Using VBA

There is a built in command in VBA for removing duplicates within list objects.

Sub RemoveDuplicates() Dim DuplicateValues As Range Set DuplicateValues = ActiveSheet.ListObjects("CarList").Range DuplicateValues.RemoveDuplicates Columns:=Array(1, 2, 3), Header:=xlYes End Sub

The above procedure will remove duplicates from an Excel table named CarList.

Columns:=Array(1, 2, 3)

The above part of the procedure will set which columns to base duplicate detection on. In this case it will be on the entire table since all three columns are listed.

Header:=xlYes

The above part of the procedure tells Excel the first row in our list contains column headings.

You will want to create a copy of your data before running this VBA code, as it can’t be undone after the code runs.

Conclusions

Duplicate values in your data can be a big obstacle to a clean data set.

Thankfully, there are many options in Excel to easily remove those pesky duplicate values.

So, what’s your go to method to remove duplicates?

=UNIQUE (array, [by_col], [exactly_once])

In case you want to highlight rows, you can combine all row values in one cell using the =CONCAT formula; if you would like to learn more about this function, read this article on the Microsoft support page.

This post will guide you how to remove duplicate rows from a Microsoft Spreadsheet.

• Remove Duplicate Rows with Remove Duplicates Command
• Remove Duplicate Rows with Advanced Filter
• Remove Duplicate Rows with Formula

Assuming that you have a list of data in range A1:B6 in which contain duplicate rows, and you want to remove them and just keep the unique row. This post will show your three methods to remove duplicate rows.

Remove Duplicate Rows with Remove Duplicates Command

The easiest way of removing duplicate rows from a selected range is to use the Remove Duplicates command. Just do the following steps:

Step1: select the range that you want to remove duplicate rows.

Step2: go to Data tab in the Excel Ribbon, and click Remove Duplicates command under Data Tools group. And the Remove Duplicates dialog will open.

Step3: checked all column options under Columns list box, and click Ok button. (Note: the dialog box will allow you to select which columns that your range that you want to be included)

Step4: you would see a prompt box and it will inform you that how many rows is removed and how many unique rows is remaining.

Remove Duplicate Rows with Advanced Filter

You can also use Advanced Filter feature to filter unique rows in a Microsoft Excel Spreadsheet and copy the last result to a new range. Just do the following steps:

Step1: select the range that you want to remove the duplicate rows from.

Step2: go to Data tab, and click Advanced command under Sort & Filter group. And the Advanced Filter dialog will appear.

Step3: select the option Copy to another location in the Advanced Filter dialog box. You need to make sure that the selected range has been entered into the List range text box. And then select one blank cell in the Copy to list box as the new location. Make sure to check the Unique records only box. Click Ok button.

Step4: the newly range has been created without duplicate rows.

Remove Duplicate Rows with Formula

You can also use Excel formulas to accomplish the same result. And you can concatenate all columns into one column, and you can find the duplicates values in the combined column. And then you can use another column based on the COUNTIF function to calculate the number of occurrences of each value in another column. Then filter the count number that is greater than 1. And just delete those filtered rows. It should be duplicate rows. Let’s see the below steps:

Step1: select a single cell adjacent to your data, such as: C1. Then enter the following formula into cell C1, and press Enter key. Then you need to copying this formula down all other rows to apply this formula.

=A1&B1

Step2: the contents of columns A-B have been concatenated into column C, and then you need to find the duplicates in the combined column C with another formula based on the COUNTIF function. Select another single cell adjacent to the column C. such as: cell D1, enter the following formula, and copying this formula down all other rows.

=COUNTIF(C$1:C1,C1)

Step3: keep to select column D, and go to Data tab, click Filter button under Sort &Filter group. And one filter arrow will be added into the cell D1.

Step4: click on the Filter arrow in cell D1, and select rows that are not equal to 1, it means that uncheck the value 1. Click Ok button.

Step5: you would see that the first occurrence of every row is hidden. And only duplicate rows are displayed. Then select all filtered rows, and right click on it, click Delete Rows from the popup menu list.

Step6: remove the filter from the column D.

• Excel COUNTIF function
  The Excel COUNTIF function will count the number of cells in a range that meet a given criteria. This function can be used to count the different kinds of cells with number, date, text values, blank, non-blanks, or containing specific characters.etc.= COUNTIF (range, criteria)…