pandas replace: values, regex and checking changes

In pandas, replace changes values according to a match. It is useful when a source uses several labels for the same category, stores a special code for missing measurements or needs selected strings corrected. I start by defining the scope: exactly what should change, in which column, and how will I recognize a correct result? A short instruction can affect considerably more data than its author intended.

All fixtures below are synthetic, and each block runs independently. I executed the examples with Python 3.14.4, pandas 3.0.6 and NumPy 2.5.4. They require neither external files nor customer records. We start with ordinary value matching, then examine column scope, regex and validation. I preserve the input so that it can be compared with the result.

Replace one specific value first

Call the method on a Series or DataFrame. There is no separate pd.replace function. The two basic arguments identify the value to find and its replacement. By default, the method returns a new object. Assign that result to a variable or to the appropriate table column if you want to use it in subsequent work.

# example: scalar
import pandas as pd

source = pd.Series([0, -1, 4, -1], dtype="Int64")
result = source.replace(-1, pd.NA)
print(source.tolist())
print(result.tolist(), str(result.dtype))

The original list is [0, -1, 4, -1]; the result is [0, <NA>, 4, <NA>] with dtype Int64. Here, we have agreed that -1 means a missing measurement. Zero is a valid observation and remains unchanged. In another domain, negative numbers may be valid too. Their meaning comes from the data contract, rather than the appearance of the values.

An explicit nullable dtype accommodates integers and missing values in one column. This does not mean every replacement is compatible with every dtype. Inserting text into a strictly numeric column may fail. Decide on the intended value model before cleaning instead of relying on an accidental type expansion during replacement.

A dictionary expresses several replacements

A dictionary is readable when several old labels need agreed names. Its keys describe input values and its values describe their replacements. You do not need a separate call for every category. Ordinary replace preserves values that do not match dictionary keys, which matters when the category mapping is incomplete.

# example: mapping
import pandas as pd

source = pd.Series(["new", "done", "hold", "new"], dtype="string")
result = source.replace({"new": "open", "done": "closed"})
print(result.tolist())
print(source.tolist())

The result is ['open', 'closed', 'hold', 'open'], while the original still contains the old labels. hold has not become missing. If the process permits only a fixed set of categories, check the resulting set separately. Preserving an unknown label is predictable method behavior; it does not automatically mean the reporting rules accept that label.

When mapping keys are also outputs of other replacements, consider the intended sequence. One mapping and several consecutive calls can produce different outcomes. A step replacing new with open, followed by another replacing open with active, also affects values created by the first step. Avoid accidentally constructing a chain of corrections when you intended one translation of the original categories.

Restrict changes to the correct columns

A global df.replace is convenient but can affect identical values in columns with different meanings. A zero may mean an unknown status in one column and a valid order count in another. A nested dictionary attaches a separate mapping to each column name. Alternatively, select a single Series and assign its transformed result back.

# example: columns
import pandas as pd

source = pd.DataFrame({"status": ["N", "D", "N"], "note": ["N", "D", "keep"]})
result = source.replace({"status": {"N": "new", "D": "done"}})
print(result["status"].tolist())
print(result["note"].tolist())

The statuses become ['new', 'done', 'new'], while the notes remain ['N', 'D', 'keep']. This fixture deliberately repeats old codes in the notes. That makes it possible to see whether the rule respects the column boundary. A test looking only at statuses would not detect an overly broad replacement elsewhere in the table.

The DataFrame.replace documentation describes nested mappings and argument forms. I choose the form that makes the intention visible. A few additional characters specifying a column are easier to review than a global expression whose consequences must be reconstructed from the complete table. This is particularly useful for identifiers and free-form notes.

Replace and map handle unknown values differently

With an ordinary dictionary, Series.map builds its output from that mapping. A value absent from the keys becomes missing. replace substitutes matches and retains the rest. Both behaviors are useful, but they meet different requirements. Decide whether an unknown label should survive, be rejected or be flagged for investigation before selecting the method.

# example: map-versus-replace
import pandas as pd

source = pd.Series(["N", "D", "X"], dtype="string")
rules = {"N": "new", "D": "done"}
replaced = source.replace(rules)
mapped = source.map(rules)
print(replaced.tolist())
print(mapped.iloc[:2].tolist(), bool(pd.isna(mapped.iloc[2])))

replace returns ['new', 'done', 'X']. The first two mapped values are ['new', 'done'], and the third is missing. The map documentation also describes exceptions involving dictionary subclasses with custom missing-key behavior. We use an ordinary dictionary here, so the result is unambiguous.

I do not automatically repair every missing result from map with fillna(source). That can be appropriate, but it can also hide an incomplete mapping or overwrite a missing value the mapping deliberately produced. Count unknown values first and decide the rule. If the goal is simply to preserve all unmatched categories, replace expresses that directly.

Define the scope of a regex match

Ordinary string replacement matches the complete value. With regex=True, a pattern may match only part of a string. Anchors help restrict the rule to the whole cell. This fixture compares a single letter, a word ending in that letter and another word containing it. Those cases reveal the difference between whole-value and substring matching.

# example: regex
import pandas as pd

source = pd.Series(["A", "BA", "CAT", "B"], dtype="string")
exact = source.replace({r"^A$": "X"}, regex=True)
fragment = source.replace({r"A": "X"}, regex=True)
print(exact.tolist())
print(fragment.tolist())

The first result is ['X', 'BA', 'CAT', 'B']; the second is ['X', 'BX', 'CXT', 'B']. For these inputs, ^A$ matches the complete value, whereas A also matches fragments. A raw string prefixed with r makes backslashes easier to express in more complex patterns. It does not turn off regular expression semantics.

Regex replacements operate on text. They are not a way to convert every numeric value in a table. To replace numeric 1, use numeric 1 or a numeric condition. Passing the string "1" does not establish that every displayed one will change. Finding no matches often raises no exception, which is why I check match counts before and after the operation.

Use str.replace for string fragments

When the task concerns strings in one column, Series.str.replace often makes the purpose clearer. For a literal substring, I set regex=False. A dot has special meaning in a regular expression, but a literal dot in a filename is simply a character. An explicit parameter saves the reader from guessing the intended interpretation.

# example: literal-text
import pandas as pd

source = pd.Series(["report.v1.csv", "plain", pd.NA], dtype="string")
result = source.str.replace(".", "_", regex=False)
print(result.tolist())

The result is ['report_v1_csv', 'plain', <NA>]. The missing value remains missing. The str.replace documentation also covers limiting replacements within each string. Do not confuse that with limiting how many rows change across a table. These restrictions operate at different levels of the data.

Before cleaning names in bulk, determine whether a character is merely formatting. A dot may separate document versions, a space may belong to an identifier and a minus sign may describe a numeric value. Text replacement does not know those meanings. Preserving the original column makes it easier to explain why two distinct strings ended up with the same normalized representation.

Method and limit were removed from replace

Older instructions show method="pad", method="bfill" or limit passed to replace. Do not use them with pandas 3. Their removal is described in the pandas 3.0 changes. If the task is to propagate values into missing positions, consider ffill or bfill separately, after defining order and group boundaries.

For a requirement such as “change the first two matches,” use a mask. eq identifies matches and the cumulative sum counts them in current row order. Combine both conditions: checking the cumulative sum alone would also select earlier rows that do not match at all. The removed limit parameter is unnecessary for expressing this requirement.

# example: first-matches
import pandas as pd

source = pd.Series([0, -1, 4, -1, -1], dtype="Int64")
hits = source.eq(-1).fillna(False)
selected = hits & hits.cumsum().le(2)
result = source.copy()
result.loc[selected] = pd.NA
print(selected.tolist())
print(result.tolist())

The mask is [False, True, False, True, False], and the result is [0, <NA>, 4, <NA>, -1]. The third match survives. This rule applies to the whole series in its current order. For the first two events per customer by date, define stable sorting and separate counting within groups. Position in an arbitrary file does not establish an event’s time.

Assign explicitly and check the scope

Avoid chained df["status"].replace(..., inplace=True). With Copy-on-Write, this is not a correct way to modify the parent table. Assigning df["status"] = df["status"].replace(...) makes the destination explicit. When selecting rows, build a mask and use loc, while checking index alignment on both sides of the assignment.

# example: audit
import pandas as pd

source = pd.DataFrame({"id": ["001", "002", "003"], "status": ["N", "D", "X"]})
result = source.copy()
result["status"] = result["status"].replace({"N": "new", "D": "done"})
changed = source["status"].ne(result["status"])
unknown = ~result["status"].isin(["new", "done"])
print(int(changed.sum()), result.loc[unknown, "id"].tolist())
print(result["id"].equals(source["id"]), len(result) == len(source))

The output 2 ['003'] identifies two changes and the record with an unknown status. The two True values confirm that identifiers and row count were preserved. That provides more evidence than inspecting the first few rows. This fixture has no missing statuses; nullable inputs require a separate comparison rule for two missing values and for transitions between missing and present values.

Validate what the result means

Before applying rules, record category frequencies, missing counts and values outside the mapping. Compare the same measures afterward. When combining new and pending into one category, fewer distinct categories may be expected. When combining identifiers, the same effect could mean losing the distinction between objects. A technically successful replacement does not establish that later aggregation remains meaningful.

Repeatability matters too. A category normalization rule will often be expected to produce the same result when applied twice. Replacing the fragment A with AA does not have that property: a second run makes the string longer again. For a recurring process, test the rules on an already cleaned sample. If multiple passes are intentional, document them as distinct transformation stages.

Do not discard unknown categories merely to make a quality check report no problems. Keep a list for investigation and count affected records. The person maintaining the dictionary can then decide whether a label is a typo, a valid new category or a source error. replace executes an agreed decision; it does not make that decision from spelling similarity or frequency.

When combining files, retain record provenance where useful. The same old label may mean different things in two systems. A global mapping would then be too broad: select rules by source or first agree on a shared data contract. Checking provenance is easier than reconstructing later why an identical code was interpreted in two different ways.

Prepare mappings without losing information

A mapping should be reviewable independently of its implementation. For each old value, record the new label and a brief reason. If two people supply conflicting destinations for the same key, combining them into a dictionary can conceal the conflict. A later assignment replaces the earlier value for that key. Detect disagreements before constructing the mapping used on the table.

Case and whitespace introduce similar questions. N, n and N are different strings. You can include all variants in the mapping or introduce an earlier normalization stage that strips spaces and standardizes case. That stage changes data too, so document it explicitly. Ignoring case is not automatically appropriate for usernames, product codes or other identifiers.

A status pipeline might retain three columns: the original value, its normalized spelling and its mapped category. That separates formatting corrections from classification decisions. If a report reader questions the number of open cases, you can identify exactly which labels were assigned to that category. There is no need to guess whether the discrepancy came from importing, trimming spaces or mapping.

The order of regex rules deserves attention as well. A broad pattern may change text intended for a later, more specific step. For separate transformations, record the expected result after each stage. Avoid regex for a simple category dictionary when whole-value matching is sufficient. A smaller set of possible matches makes the rule easier to review and reduces surprising edge cases.

With mixed inputs, inspect actual value types rather than display alone. The number one and a string containing the digit one can look identical in a table preview without behaving identically under ordinary matching. Establish why the types differ first. Converting the whole table to text may standardize its appearance while damaging dates, missing values and subsequent numeric calculations.

Missing-value checks require another decision: should two missing values, before and after replacement, count as a change? Usually they should not. A transition from the sentinel -1 to a missing value should count, however. Comparing nullable series can produce missing entries in the mask. Define conditions for presence and equality separately, then test a small fixture covering each transition.

Add an empty series and a series with no matches to the test set. The first should preserve an empty result with the appropriate dtype; the second should remain unchanged. Add a fixture where every record matches too. These cases expose assumptions that there must always be both a matching value and a value left untouched.

Exercise: test the boundaries of your rule

Compare operation order too. First combine two categories and count them; then count the original categories and add the relevant results. A correct mapping should produce the same destination count through both routes. If the results differ, inspect missing values, unknown labels and column scope. This independent aggregation check can reveal a mistake that is difficult to see in an individual row.

Write down concrete expected values before executing the transformation. Four records are small enough to count manually. Do not construct the expected result by repeating the same replacement in another piece of code, because both versions may repeat the same mistake. A small fixture with deliberate cases provides a better reference than a large random collection with no known answer.

Extend the column example with N in another column, an unknown status and a missing value. Write down the expected result before running the code. Compare a global replacement with the mapping attached specifically to status. Identify which differences are intentional and which reveal excessive scope. Do not judge correctness solely by the absence of an exception.

For the mask example, try limits of zero, one and more than the number of matches. Add a missing value before the first match. Check that only matching positions increase the counter and that the source remains unchanged. Finally, combine an import through read_json with one clearly defined replacement rule. This creates a small process whose result you can trace back to its input.