pandas drop_duplicates: subset, keep and latest versions

Before removing repeated records, I define what one record represents. Two identical export lines may be repeated deliveries of the same event. Two purchases by the same customer may be entirely valid events. Their visual similarity does not settle that distinction.

DataFrame.drop_duplicates() selects rows using specified columns and a retention rule. There is no top-level pd.drop_duplicates() function. In these examples, I call the method on a particular table. The documentation describes its parameters and defaults.

I ran every block below sequentially with Python 3.14.4, pandas 3.0.6 and 3.1.0, and NumPy 2.5.4, without PyArrow. Both environments produced the displayed results. The data is small and synthetic; these examples are not a performance benchmark for large files.

Keep the first, last or none of the repeated rows

I will create a five-row table. The pair A, 10 appears twice, as does B, 20. The row C, 30 has no copy. Printing the indexes makes it easy to identify exactly which occurrences remain.

# example: keep-options
import pandas as pd

rows = pd.DataFrame({
    "account": ["A", "B", "A", "C", "B"],
    "amount": [10, 20, 10, 30, 20],
})
print(rows.drop_duplicates().index.tolist())
print(rows.drop_duplicates(keep="last").index.tolist())
print(rows.drop_duplicates(keep=False).index.tolist())
print(len(rows))

[0, 1, 3]
[2, 3, 4]
[3]
5

The default keep="first" retains the first occurrence of each combination. With keep="last", the last occurrences survive, but their order still follows their positions in the input: indexes 2, 3, 4. The method does not group accounts alphabetically or arrange the result as 2, 4, 3.

With keep=False, every occurrence of a repeated combination disappears. Only account C remains. This can help separate records without conflicts, but it also removes the first valid copy. Before using it, I check that this is the intended analytical outcome. The original rows still contains five rows because these calls returned separate results.

Subset defines the comparison key

Next, I will change the final amount for account B to 25. Those two account rows now differ in their contents. Comparing complete rows retains both, whereas comparing account alone selects just one.

# example: subset-and-audit
changed = rows.copy()
changed.loc[4, "amount"] = 25
print(changed.drop_duplicates().index.tolist())
print(changed.drop_duplicates(subset=["account"]).index.tolist())
conflicts = changed.loc[changed.duplicated(subset=["account"], keep=False)]
print(conflicts.index.tolist())

[0, 1, 3, 4]
[0, 1, 3]
[0, 1, 2, 4]

Here, subset=["account"] means accepting the first amount for each account. The amount column remains in the result even though it takes no part in the comparison. If the different amounts represent separate purchases, this operation loses valid data. If they represent balance revisions, I still need a rule for choosing the current version.

To inspect every participant in a conflict, I used the duplicated method with keep=False. In practice, I examine these rows before approving deduplication. A composite key is a list, such as subset=["account", "amount"]. Adding any column changes the definition of equality.

The latest record needs sorting and a tie rule

“Last” refers to the current table order. On its own, keep="last" does not inspect timestamps or select the largest value. In the next example, I specify a rule: choose the latest timestamp for each account, then the largest revision_id when timestamps tie.

# example: latest-revision
versions = pd.DataFrame({
    "account": ["A", "A", "B", "A"],
    "updated_at": pd.to_datetime([
        "2026-10-20", "2026-10-22", "2026-10-21", "2026-10-22"
    ]),
    "revision_id": [10, 11, 12, 13],
    "amount": [100, 120, 80, 125],
})
assert versions[["account", "updated_at", "revision_id"]].notna().all().all()
assert versions["revision_id"].is_unique
ordered = versions.sort_values(["account", "updated_at", "revision_id"])
latest = ordered.drop_duplicates(subset=["account"], keep="last")
print(latest[["account", "revision_id", "amount"]].to_dict("records"))

[{'account': 'A', 'revision_id': 13, 'amount': 125}, {'account': 'B', 'revision_id': 12, 'amount': 80}]

Revision 13 wins for A. The two latest revisions share a date, so the additional identifier resolves the tie. I check that identifiers are unique and ordering keys are present before selecting rows. Without those assumptions, the result might depend on accidental import order or on where missing timestamps are placed.

In an actual process, the largest identifier must have a meaningful interpretation. A random UUID does not inherently identify a newer revision. When the source provides no revision order, I record the conflict for investigation. The sort_values documentation explains sorting options; deciding which record wins belongs to the rules of the dataset.

A missing key does not identify the same customer

In the following explicitly typed Int64 column, the two pd.NA values belong to the same duplicate group when comparing customer_id. Mechanically, one of those rows survives. Analytically, I have no evidence that they describe the same person.

# example: missing-keys
customers = pd.DataFrame({
    "customer_id": pd.Series([1, 1, pd.NA, pd.NA, 2], dtype="Int64"),
    "record": ["a", "b", "c", "d", "e"],
})
print(customers.drop_duplicates(subset=["customer_id"]).index.tolist())
known = customers["customer_id"].notna()
resolved = pd.concat([
    customers.loc[known].drop_duplicates(subset=["customer_id"]),
    customers.loc[~known],
]).sort_index()
print(resolved.index.tolist())

[0, 2, 4]
[0, 2, 3, 4]

The second version deduplicates known identifiers and keeps both rows without a key. Here, sort_index() restores input order because the example has a unique, increasing index. A different index would require a separate position column. I also do not assume that the text "NULL", an empty string and an actual missing value are automatically equivalent. I establish typing and normalization rules first.

Indexes, ignore_index and the returned result

The index takes no part in row comparison. Two identical rows with different index labels are still duplicates. Conversely, repeated index labels do not cause different rows to disappear.

# example: index-and-result
same = pd.DataFrame({"value": [7, 7]}, index=[100, 200])
different = pd.DataFrame({"value": [7, 8]}, index=[100, 100])
print(same.drop_duplicates().index.tolist())
print(different.drop_duplicates()["value"].tolist())
clean = rows.drop_duplicates(ignore_index=True)
print(clean.index.tolist())
print(clean["account"].tolist())
print(len(rows), len(clean))

[100]
[7, 8]
[0, 1, 2]
['A', 'B', 'C']
5 3

ignore_index=True assigns consecutive labels to the result. It changes neither the comparison rule nor the selected record. When the index contains an identifier needed for auditing, I preserve it or move it into a named column first. If it should form part of the deduplication key, it must become one of the compared columns.

I prefer assigning clean = rows.drop_duplicates(...). Older code may use inplace=True, which changes the object and returns None; assigning that result back to rows loses the reference to the table. In pandas 3.1.0, this method’s inplace parameter was deprecated, with removal planned for 4.0. The release notes and PDEP-8 section describe the change. I do not treat that parameter as a memory-saving guarantee.

Check the result before accepting it

I compare the number of records before and after the operation and account for the removed positions. Then I inspect conflicts and verify that applying the same rule again leaves the result unchanged. That stability is useful, but it cannot prove that the key is correct: an incorrect rule can also produce a stable result.

For a list of distinct values from one column, I use pandas unique. For their count, I use pandas nunique. I choose drop_duplicates when the result should contain complete retained rows and I can justify why each one was selected.