Reading files in Python: text, CSV and import checks

Reading a file in Python can take one line. Working out what actually arrived in memory is often more interesting. An identifier loses its leading zeros, accented characters look strange, and an empty field quietly turns into zero. The program finishes without an error, which makes it tempting to move straight to the chart. I prefer to check the import before asking the data to explain anything.

I will move from a small text file to a CSV import with explicit column and value checks. All data is synthetic. The examples use Python 3.14’s standard library, with a pandas alternative checked in version 3.0.6 near the end. Run the Python blocks in order in one script. There is no missing Titanic download or mysterious file that only exists on the tutorial author’s desktop.

First establish where Python looks for the file

A relative path is resolved against the process’s current working directory. That directory does not have to be the folder containing your script. An editor, notebook and terminal can start the same code from different locations. If you get FileNotFoundError, check the working directory and filename before changing unrelated encoding settings or reinstalling a library.

I use Path from pathlib to construct paths. Joining a directory and filename with the slash operator reduces separator mistakes. In an ordinary Python string, \t represents a tab and \n represents a newline. That matters when copying Windows paths. A raw string can help, but passing an already interpreted string to Path will not undo its escape sequences.

For this exercise, I put everything in a separate temporary directory. Only our sample files are created there, so running the exercise does not overwrite a real export. Keep the workspace object until the final block, which removes the directory and its contents. In an actual project, replace that location with the agreed input directory rather than relying on wherever your editor happens to start.

# example: workspace
from pathlib import Path
from tempfile import TemporaryDirectory

workspace = TemporaryDirectory(prefix="file-reading-")
root = Path(workspace.name)
text_path = root / "notes.txt"
text_path.write_text("Łódź\nKraków\n", encoding="utf-8")
print(text_path.name, text_path.is_file())

The result is notes.txt True. Writing the sample is setup for the exercise; the reading steps do not modify it. The pathlib documentation covers path and file operations. An is_file() check is useful for diagnosis, but it cannot guarantee a later read will succeed: a file can disappear, or the process might lack permission to open it.

Open text with with and an explicit encoding

open returns a file object. Mode r reads text, while rb reads bytes. Here I want text, and I specify encoding="utf-8" because that is how I created the sample. Encoding describes the file. It is not a universal repair switch. An older export may use a different encoding, which you should confirm from its source or documented format.

# example: text
with text_path.open("r", encoding="utf-8") as handle:
    contents = handle.read()
    print(repr(contents))
    print(handle.closed)
print(handle.closed)

You will see 'Łódź\nKraków\n', followed by False and True. repr exposes the newline characters that ordinary printing would display as line breaks. The file is open inside the block and closed after leaving it. It also closes if the block raises an exception. The contents variable still holds the text that was read into memory.

Manual open and close calls make it easy to miss a cleanup path. A context manager gives that responsibility a clear boundary. Python’s file-reading tutorial also explains shorter ways to read a small file. I use with here because it makes the lifetime of the open resource visible while we learn the basic operations.

I do not immediately add errors="ignore" when decoding fails. Ignoring undecodable bytes can remove parts of names or identifiers while leaving output that looks plausible. First establish where the file came from and how it was encoded. If replacement characters are an intentional compromise, record that choice and inspect the affected records. A successful read and a faithful read are separate checks.

Choose between read, readline and iteration

Calling read() without an argument reads all remaining contents. That is convenient for a short note and potentially expensive for a large file. readline() reads one line, while iterating over the file visits successive lines. Three calls to readline() do not become a complete import merely because the original demonstration happened to contain three lines of data.

# example: lines
with text_path.open(encoding="utf-8") as handle:
    first = handle.readline()
    remaining = list(handle)
    after_end = handle.read()
print(first.rstrip("\n"))
print([line.rstrip("\n") for line in remaining])
print(repr(after_end))

The output is Łódź, then ['Kraków'], then ''. Reading advances the current position in the file. Once you reach the end, another read does not automatically start again. If I need a second pass, I reopen the file or deliberately reposition the stream. For a beginner’s script, reopening often makes the sequence of operations easier to follow.

I use rstrip("\n") to remove the trailing newline. Plain strip() would remove other leading and trailing whitespace too, which can change fixed-width data. Cleaning should follow the format. A trailing space may be irrelevant in a friendly display, but preserving it while investigating a malformed identifier can be exactly what reveals the problem.

The example turns remaining lines into a list so that the result is easy to inspect. For a large file, process each line inside the loop without collecting everything. Memory usage depends on the whole pipeline. Iterating over a stream provides little benefit if you immediately append every result to another large in-memory list.

Parse CSV instead of splitting on commas

CSV is text with rules for representing records. A delimiter can appear inside a quoted field, and a quoted field can contain a newline. That is why split(",") does not replace a CSV parser. It may work for a convenient two-column example and fail as soon as an exported comment contains punctuation or a line break.

Let’s create three synthetic records from a task queue. task_id is an identifier, city a name, minutes a measured duration and note free text. A duration of zero is a valid measurement here. An empty duration means no measurement is available. I include both deliberately, because an import should preserve those meanings before a calculation chooses how to use them.

# example: csv-fixture
import csv

csv_path = root / "tasks.csv"
with csv_path.open("w", encoding="utf-8", newline="") as handle:
    writer = csv.writer(handle, delimiter=";")
    writer.writerow(["task_id", "city", "minutes", "note"])
    writer.writerows([
        ["001", "Łódź", "12", "call; follow-up"],
        ["002", "Kraków", "0", ""],
        ["003", "Gdańsk", "", "two\nlines"],
    ])

Our delimiter is a semicolon, and the first note contains a semicolon too. csv.writer correctly quotes that field. The final note occupies two physical text lines while still belonging to one record. Counting file lines therefore will not tell you how many tasks are present. Count the records produced by the parser, with the header handled according to the format.

Using newline="" leaves newline handling to the csv module. The csv documentation recommends this, including for multiline quoted fields. It does not mean deleting newlines from values. Encoding and delimiters describe different properties of the input, so correcting one cannot automatically fix a mistake in the other.

Preserve identifiers and check the header

DictReader uses the first record as field names and returns dictionaries for subsequent records. I can then refer to task_id instead of a mysterious column number. Ordinary reading through csv leaves field values as strings. That gives me a useful point at which to decide explicitly which fields should become numbers and which should remain text.

# example: csv-read
with csv_path.open(encoding="utf-8", newline="") as handle:
    reader = csv.DictReader(handle, delimiter=";")
    expected_columns = ["task_id", "city", "minutes", "note"]
    if reader.fieldnames != expected_columns:
        raise ValueError("Unexpected columns")
    rows = list(reader)
print(len(rows), rows[0]["task_id"])
print(rows[0]["note"])
print(repr(rows[2]["note"]))

The output is 3 001, followed by call; follow-up and 'two\nlines'. The identifier kept its leading zeros. The parser did not mistake the semicolon inside the note for a column boundary or split the multiline note into two tasks. These small checks tell us more about a faithful import than merely displaying the first record and assuming the rest looks similar.

Comparing the header lists requires a specific column order here. That is a deliberate contract for this exercise. If a source may reorder columns, validate the agreed names and their uniqueness instead. Do not accept arbitrary headers simply because most of the report still runs. Duplicate column names can lose information when records are represented as dictionaries, so checking names is part of protecting the input’s structure.

Convert values and validate their meaning

int() does not know whether a negative duration makes sense for this task. It also cannot decide whether an empty string means zero, a missing observation or a broken export. I set the rules explicitly: an identifier is required, duration is a nonnegative integer, and an empty duration remains None. The script should not invent measurements on behalf of the source.

# example: validation
validated = []
for record_number, row in enumerate(rows, start=1):
    if None in row or any(value is None for value in row.values()):
        raise ValueError(f"Wrong field count in record {record_number}")
    if not row["task_id"]:
        raise ValueError(f"Missing task_id in record {record_number}")
    raw_minutes = row["minutes"]
    minutes = None if raw_minutes == "" else int(raw_minutes)
    if minutes is not None and minutes < 0:
        raise ValueError(f"Negative minutes in record {record_number}")
    validated.append({**row, "minutes": minutes})

known = [row["minutes"] for row in validated if row["minutes"] is not None]
print(len(validated), sum(known), len(validated) - len(known))

The result 3 12 1 means three tasks, twelve known minutes and one missing measurement. Error messages use the data-record number, not the physical file line. That distinction matters for multiline CSV. Testing None in row detects additional fields, while None values expose fields missing from a record that has too few values.

I distinguish a missing number from a value that evaluates to false. After conversion, if row["minutes"] would reject both None and a valid zero. The filter therefore uses is not None. My article on variables, data types and operators explains the underlying distinctions in more detail, including why truthiness is not a complete data-quality rule.

In this small exercise, a bad record stops processing. A larger workflow can separate valid records from rejected ones, but then its report needs rejection counts and reasons. Silently dropping problematic rows produces a convenient impression of completeness. Whether one bad record should stop an entire import depends on the process; knowing what was excluded should never depend on luck.

When I switch to pandas.read_csv

When the next step is filtering, grouping or joining a table, a DataFrame can be more convenient than a list of dictionaries. I then use pandas.read_csv, still specifying delimiter, encoding and types. Import options should follow the data contract. Automatic inference helps explore an unfamiliar file, but it may be unsuitable for a report that runs repeatedly against changing input.

# example: pandas-csv
import pandas as pd

df = pd.read_csv(
    csv_path,
    sep=";",
    encoding="utf-8",
    dtype={"task_id": "string", "city": "string", "minutes": "Int64", "note": "string"},
    keep_default_na=False,
    na_values={"minutes": [""]},
)
print(df["task_id"].tolist())
print(int(df["minutes"].sum()), int(df["minutes"].isna().sum()))
print(df.loc[1, "note"] == "")

The results are ['001', '002', '003'], 12 1 and True. The capitalized Int64 dtype supports missing integers. I disable the default collection of missing-value markers and recognize an empty string only in the duration column. An empty note remains an empty note. This stops one rule from spreading across fields with different meanings.

That configuration is a choice for this input. Another source may use empty strings to mean missing cities as well. Decide before importing. The read_csv reference documents dtype, keep_default_na, na_values and chunked reading. A dtype declaration still does not check negative durations, identifier uniqueness or agreement between the exported and imported record counts.

The older article demonstrated np.recfromcsv. NumPy removed that function in version 2.0, as documented in its migration guide. genfromtxt is an option for numerical arrays. With textual identifiers and free-form notes, I choose csv or pandas here. Adding another library is unnecessary when the existing tools fit the structure of the exercise.

Large files and failed imports need separate decisions

Reading an entire CSV into a list or a complete DataFrame requires memory for the result. Its in-memory size can differ substantially from the file’s size on disk. For larger inputs, consider iterating over records or using pandas chunksize. You still need to decide which calculations work per chunk and how to combine their results without changing their meaning.

Sums and record counts can be added across chunks. Means require retaining both totals and counts; averaging chunk averages can give the wrong answer. Duplicate detection must consider identifiers seen in earlier chunks as well. Splitting input into pieces solves part of a memory problem, but it does not design the analytical process for you.

When an import fails, I ask separate questions: does the file exist, can it be opened, can the bytes be decoded, does the parser understand the structure, and do values satisfy the rules? FileNotFoundError, UnicodeDecodeError and a numeric conversion error describe different stages. A broad exception handler that prints only “something went wrong” removes the detail needed to repair the right stage.

Checks before reusing the importer

For the next export, compare its coverage as well as its column names. Check the period, record count and whether the file is a complete snapshot or an increment since the previous run. Two files with identical headers can cover different scopes. Combining overlapping snapshots may double a result, and the file-reading function has no way to recognize that mistake.

Decide whether task identifiers must be unique. Our example treats them as identifiers of individual tasks, but the validation code does not yet enforce uniqueness. You could track previously seen identifiers and compare their count with the record count. If repeated identifiers instead represent successive events for the same task, removing duplicates would lose legitimate information. Establish what one row represents first.

A header that looks correct can contain an unexpected byte-order mark at the beginning of a UTF-8 export. For a confirmed source of that kind, utf-8-sig handles the mark while decoding. Do not repair the symptom by blindly deleting the first character of every header. That workaround will corrupt an ordinary file when its producer stops adding the mark.

Inspect actual contents rather than trusting the extension. A file called .csv may contain an error page returned by a server, while a .txt file may contain a valid tab-separated table. Changing the extension does not convert either structure or encoding. Record the expected delimiter, encoding, columns and missing-value rules beside the code so that another person can diagnose a specific record and rule instead of reporting that “Python cannot read the file”.

Exercise: break the data and observe the checks

Write down the expected result before running the code. Compare it with the actual result and explain each difference using the input records.

Replace 12 in the sample with twelve, then try -1 separately. The first case should fail during integer conversion; the second should fail the nonnegative-duration rule. Next, remove an identifier and add an extra field to one record. Each problem should be detected before the total is calculated. Rerun the whole script after each change so that old notebook state cannot disguise the result.

Also test valid edge cases: an empty note, zero minutes and a file containing only the header. The last case has a valid structure but no data records. Whether that is acceptable for a daily report depends on the process. Distinguish “there were no tasks” from “the export never arrived”. A CSV-reading function cannot establish that business meaning by itself.

# example: cleanup
workspace.cleanup()
print(root.exists())

The final result is False: the exercise directory has been removed. You now have a small import whose behavior you can explain using individual records. Continue with list indexing and slicing or a complete first pandas report. I first want to know that the imported values mean the same thing as the values in the source.