Pandas to_numeric: numeric conversion, errors, and downcasting in pandas 3

Does a column look numeric while addition fails, or do values disappear when you clean it? I start with two questions: what is actually stored in the cells, and which values are allowed to count as invalid. pd.to_numeric() handles conversion. It cannot decide whether the word missing represents an absent measurement, a broken export, or zero.

I’ll use small synthetic datasets to show conversion and keep track of values that fail. I executed the examples with Python 3.14.4, pandas 3.0.6, and NumPy 2.5.4, without PyArrow. Run the blocks in order in one script or notebook. Variable names and executable code are identical in the Polish and English versions so that you can compare the results directly.

What to_numeric converts

pd.to_numeric(arg, errors="raise", downcast=None) accepts inputs including a scalar, a list, or a Series. A Series produces a Series with its index preserved; a list produces an array. A whole two-dimensional DataFrame is not a valid argument. With a table, I select specific columns instead of asking the function to guess their meaning.

The two error settings matter most here. errors="raise" stops conversion when a value cannot be parsed, while errors="coerce" replaces that value with a missing value. With the default NumPy types used in the first examples, that marker is NaN. Nullable types can use pd.NA. This is the technical representation; the reason for the missing value still needs separate tracking.

The result does not always have the same dtype. A Series of integer strings can produce int64, while fractions or missing values represented by NaN often lead to float64. Input that is already numeric may retain its existing dtype. I inspect the result’s dtype instead of inferring it from a preview containing a few values without decimal places. The to_numeric documentation describes the supported arguments.

Conversion with an error audit

Our data contains three valid values, one invalid string, and one value that was missing from the start. Explicit dtype="object" lets us display the original None alongside strings without depending on automatic string dtype inference.

# example: basic
import pandas as pd

raw = pd.Series(["12", "20.5", "bad", None, "0"], name="raw", dtype="object")
numeric = pd.to_numeric(raw, errors="coerce")
invalid = raw.notna() & numeric.isna()
audit = pd.DataFrame({"raw": raw, "numeric": numeric, "invalid": invalid})
print(audit.to_string(index=False))
print(numeric.dtype)
print("invalid:", int(invalid.sum()), "missing:", int(raw.isna().sum()))

 raw  numeric  invalid
  12     12.0    False
20.5     20.5    False
 bad      NaN     True
None      NaN    False
   0      0.0    False
float64
invalid: 1 missing: 1

The key expression is raw.notna() & numeric.isna(). It identifies a row that had a value before conversion but no longer has one afterward. That makes bad a conversion failure, while None is not a new failure. Using only numeric.isna().sum() would return two and combine different situations.

I keep raw intact. That lets me return to the source value, show it to the person responsible for the file, and investigate whether the problem affects one entry or the whole column. For a larger dataset, a record identifier and a sample of rejected values stored in a controlled location can be enough. Diagnosing the problem does not require printing the entire export into logs.

If the source contract promises valid numbers, I prefer to stop processing:

# example: raise
try:
    pd.to_numeric(raw, errors="raise")
except ValueError:
    print("Invalid numeric value; inspect the source column")

Invalid numeric value; inspect the source column

The printed message is mine, not the literal pandas exception message. I catch a specific ValueError, without hiding arbitrary program failures. A production process still needs an explicit decision to stop the job or set the data aside for repair. A print statement alone is not a complete error handling strategy.

coerce is useful when I deliberately accept a partially valid file. I add a counter and an acceptance rule, such as rejecting the import when invalid values exceed an agreed threshold. The actual threshold depends on the use case. There is no universal percentage that makes sense in every project.

Why errors ignore no longer works

Older tutorials show errors="ignore". That option was removed in pandas 3, as recorded in the 3.0 release notes. Do not treat it as a third supported mode.

# example: ignore-removed
try:
    pd.to_numeric(raw, errors="ignore")
except ValueError:
    print("errors='ignore' is not supported in pandas 3")

errors='ignore' is not supported in pandas 3

If you actually need to retain the original column after a failed conversion, use raise inside your own try/except and decide what the error branch should do. Returning numbers on some runs and text on others under the same column name makes subsequent analysis harder. I prefer a separate result column and an explicit validation status. Readers of the report can then tell whether the operation succeeded.

Downcasting and nullable types

downcast attempts to use a smaller dtype when the values permit it. Small integers make the effect easy to see:

# example: downcast-integer
whole = pd.to_numeric(pd.Series(["12", "20", "0"]), downcast="integer")
print(whole.tolist())
print(whole.dtype)

[12, 20, 0]
int8

int8 can hold these three values. A larger value could require a wider type. I do not interpret this result as a fixed contract for all future files.

Now try the same option with a fraction and missing values:

# example: downcast-fraction
fractional = pd.to_numeric(raw, errors="coerce", downcast="integer")
print(fractional.tolist())
print(fractional.dtype)

[12.0, 20.5, nan, nan, 0.0]
float64

The result stays float64. downcast="integer" does not mean truncating 20.5 to 20. I do not remove a fraction merely to reduce the column’s storage size. First I establish whether fractional values are permitted in this domain, and only then consider memory and the target dtype.

For integer values with missing entries, I can deliberately choose a nullable backend:

# example: nullable
counts = pd.to_numeric(
    pd.Series(["12", None, "0"]),
    errors="raise", downcast="integer", dtype_backend="numpy_nullable",
)
print(counts.tolist())
print(counts.dtype)

[12, <NA>, 0]
Int8

Int8, with a capital I, is a pandas extension dtype supporting pd.NA. Lowercase int8 is a different dtype and cannot represent this missing value. The result [12, <NA>, 0] preserves the distinction between an unknown count and a genuine zero. The nullable integer guide explains this representation.

There is also dtype_backend="pyarrow", which requires the separate PyArrow library. I did not execute that variant in the stated environment, and the test results above do not cover it. When choosing a backend, check compatibility with subsequent operations and exports as well as the dtype name. The PyArrow integration documentation sets out its separate requirements.

Decimal commas and thousands separators

to_numeric is not a parser for every local number format. It will not automatically interpret 12,75 as 12.75. I first establish the source convention. The contract below allows a decimal comma, ordinary spaces between complete groups of thousands, an optional sign, and integers without separators. It does not allow periods, currency symbols, or empty strings.

# example: locale
local_raw = pd.Series(
    ["1 234,50", "12,75", None, "1,234.50", "-2 000,00", "1 2,00", ""],
    dtype="string",
)
text = local_raw.str.strip()
valid_format = text.str.fullmatch(
    r"[+-]?(?:\d{1,3}(?: \d{3})+|\d+)(?:,\d+)?"
).fillna(False)
normalised = text.where(valid_format).str.replace(" ", "", regex=False).str.replace(
    ",", ".", regex=False
)
local_numeric = pd.to_numeric(normalised, errors="raise", dtype_backend="numpy_nullable")
local_invalid = local_raw.notna() & ~valid_format
print(local_numeric.tolist())
print(local_invalid.tolist())

[1234.5, 12.75, <NA>, <NA>, -2000.0, <NA>, <NA>]
[False, False, False, True, False, True, True]

The pattern rejects 1,234.50 because it uses a different format, and 1 2,00 because its thousands group has the wrong length. The final empty string is also invalid under this contract. The original None remains missing rather than becoming a format error. If empty strings mean missing values in your source, introduce that rule deliberately before validation and preserve the original information.

I do not globally remove every comma and period. The string 1,234 can represent different numbers under different conventions. Here it means 1.234 because the comma is decimal. With mixed conventions, establish where the rows came from first; a single character replacement cannot resolve ambiguity.

This pattern does not cover a nonbreaking space or a narrow nonbreaking space. If your export uses those characters, identify them and add specific normalization to the contract. The current example is not a universal importer for European numeric formats.

A consistent CSV can sometimes be handled at import time. read_csv provides decimal and thousands parameters, described in the CSV import documentation. Configuring them does not remove the need to inspect the resulting dtype and look for exceptions in the column.

Multiple columns and identifiers

Not every string of digits represents a measurement. Identifier 0012 needs its leading zeros. I select only two columns for conversion:

# example: columns
orders = pd.DataFrame({
    "id": ["0012", "0013"],
    "quantity": ["2", "bad"],
    "amount": ["12.5", "8"],
})
columns = ["quantity", "amount"]
converted = orders.copy()
converted[columns] = orders[columns].apply(pd.to_numeric, errors="coerce")
print(converted.to_string(index=False))
print((orders[columns].notna() & converted[columns].isna()).sum().to_dict())

  id  quantity  amount
0012       2.0    12.5
0013       NaN     8.0
{'quantity': 1, 'amount': 0}

apply passes each selected Series to to_numeric. I retain the original table and count new missing values separately for each column. That tells me the problem affects quantity, while amount converted successfully.

I do not fill every missing result with zero at the end of a method chain. In this example, doing so would assert that the incorrectly recorded quantity equals zero. We do not know that. Similarly, I do not convert phone numbers, postal codes, or long identifiers simply because they look numeric. If you are not going to add them or compare them as quantities, text is often the appropriate representation.

Infinity and precision

Successful parsing does not necessarily produce a valid business value:

# example: finite
import numpy as np

special = pd.to_numeric(pd.Series(["3", "inf", "-inf", "bad"]), errors="coerce")
print(special.tolist())
print(np.isfinite(special).tolist())

[3.0, inf, -inf, nan]
[True, False, False, False]

inf is accepted as infinity. If a column must contain finite measurements, I need another check using np.isfinite, and often checks for range, units, and permitted signs. errors="coerce" does not perform those checks for me.

Large numbers deserve attention too. This example intentionally demonstrates loss of precision after a conversion to floating point:

# example: precision
identifier = "9007199254740993"
exact_integer = pd.to_numeric(pd.Series([identifier]))
approximate_float = pd.to_numeric(pd.Series([identifier, "0.5"]))
print(int(exact_integer.iloc[0]))
print(int(approximate_float.iloc[0]))

9007199254740993
9007199254740992

The first Series holds the exact number in an integer dtype. The second must also accommodate a fraction, and float64 cannot represent the specified integer exactly. This is another reason to avoid turning identifiers into measurements. If exact decimal arithmetic is a requirement, choose an appropriate data model; downcast alone provides no such guarantee.

Empty data and an exercise

After successful conversion, I still check how many observations remain:

# example: empty
empty = pd.to_numeric(pd.Series([], dtype="object"), errors="coerce")
all_missing = pd.to_numeric(pd.Series([None, None], dtype="object"), errors="coerce")
print(len(empty), all_missing.isna().tolist())
print(all_missing.sum(), all_missing.sum(min_count=1))

0 [True, True]
0.0 nan

The empty Series has zero elements. The all-missing Series contains no measurements, even though its default sum() returns 0.0. With min_count=1, at least one nonmissing value is required, so the result here is missing. Do not confuse that situation with adding valid zeros.

Try a small exercise. For input "7", "oops", None, "0", "2.5", predict the number of new missing values and the resulting dtype with downcast="integer". Then run the code:

# example: exercise
practice = pd.Series(["7", "oops", None, "0", "2.5"], dtype="object")
answer = pd.to_numeric(practice, errors="coerce", downcast="integer")
print(answer.tolist())
print(int((practice.notna() & answer.isna()).sum()))
print(answer.dtype)

[7.0, nan, nan, 0.0, 2.5]
1
float64

There is one new failure, the original missing value is a separate situation, and the fraction keeps the result in a floating dtype. Once you can explain those three points, you can add conversion to a larger process without guessing what disappeared along the way. A possible next step is deliberate missing-value handling with fillna, after establishing what the missing values mean.