Why treating Delta tables like Parquet files is costing you reliability — and how PySpark can fix it

You might think reading and writing Delta tables with PySpark is just loading and saving files, but Delta Lake changes the game by bringing ACID transactions, schema evolution, and time travel to your data pipelines. Suddenly, what seemed like a simple task becomes an opportunity to build robust, scalable, and maintainable data systems.

When I first started using Delta Lake, I was surprised how naturally it fit into Spark workflows. Priya on my team once joked, “It’s like Spark finally grew up.” Yet, many still treat Delta tables like plain Parquet files—missing a chance to harness its full power. Let’s walk through the essential ways you can read and write Delta tables using PySpark, and take advantage of all the features Delta Lake offers.

Reading Delta Tables with PySpark

Reading a Delta table is straightforward if you know the right format to specify. PySpark doesn’t automatically guess Delta format, so you have to explicitly tell it.

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName('ReadDeltaTable') \
    .config('spark.jars.packages', 'io.delta:delta-core_2.12:2.3.0') \
    .getOrCreate()

# Read a Delta table from a given path
delta_df = spark.read.format('delta').load('/path/to/delta/table')
delta_df.show()

Here, the key is .format(‘delta’). Without it, Spark treats the data as regular Parquet files, losing Delta’s transactional consistency and schema handling. This simple call loads the latest snapshot of your Delta table.

You can even perform time travel by specifying a version or timestamp if you want to inspect historical data:

historical_df = spark.read.format('delta') \\
    .option('versionAsOf', 5) \\
    .load('/path/to/delta/table')

This flexibility means Delta tables aren’t just storage—they’re reliable data sources you can query safely ever after.

Writing and Appending Data to Delta Tables with PySpark

Writing to Delta tables unlocks not only data persistence but schema evolution and multiple write modes. The most common modes are overwrite and append.

Appending is simple, but what’s really neat is Delta’s support for schema evolution. Imagine you have a new column in your data; with schema evolution enabled, Delta can merge the new schema without dropping the old one.

from pyspark.sql import SparkSession
from pyspark.sql import Row

spark = SparkSession.builder \
    .appName('WriteDeltaTable') \
    .config('spark.jars.packages', 'io.delta:delta-core_2.12:2.3.0') \
    .getOrCreate()

# Create a sample DataFrame
new_data = spark.createDataFrame([
    Row(id=3, name='Alice', age=30)
])

# Append data to the Delta table with schema evolution enabled
new_data.write.format('delta') \\
    .mode('append') \\
    .option('mergeSchema', 'true') \\
    .save('/path/to/delta/table')

The .option(‘mergeSchema’, ‘true’) tells Delta to reconcile differences between the existing schema and the new data’s schema. This saves you from manually altering tables, and makes data ingestion pipelines more resilient to change.

Use .mode(‘overwrite’) if you want to replace the entire table content:

df.write.format('delta').mode('overwrite').save('/path/to/delta/table')

But be cautious with overwrite, especially in shared environments.

How to Upsert Data in Delta Tables Using PySpark

Upserts—updating existing rows and inserting new ones—are classic headaches in data engineering. Delta Lake’s merge API makes this painless.

from pyspark.sql import SparkSession
from delta.tables import DeltaTable

spark = SparkSession.builder \
    .appName('DeltaMergeExample') \
    .config('spark.jars.packages', 'io.delta:delta-core_2.12:2.3.0') \
    .getOrCreate()

# Prepare new data to upsert
updatesDF = spark.createDataFrame([
    (1, 'John Doe', 35),
    (4, 'Jane Smith', 29)
], ['id', 'name', 'age'])

# Load the existing Delta table
deltaTable = DeltaTable.forPath(spark, '/path/to/delta/table')

# Perform merge (upsert) based on matching 'id'
deltaTable.alias('t') \\
    .merge(updatesDF.alias('s'), 't.id = s.id') \\
    .whenMatchedUpdate(set={
        'name': 's.name',
        'age': 's.age'
    }) \\
    .whenNotMatchedInsert(values={
        'id': 's.id',
        'name': 's.name',
        'age': 's.age'
    }) \\
    .execute()

Delta’s merge API lets you express this logic declaratively: when the id matches, update the record; otherwise, insert it. This is a huge upgrade over custom merge logic or complicated batch scripts.

How to Efficiently Read and Write Delta Tables with PySpark

• Always include the Delta Lake package in your Spark session: io.delta:delta-core_2.12:2.3.0 (or your specific version). Without it, Delta format won’t be recognized.
• Use .format(‘delta’) consistently to make your intent clear.
• Use append with schema evolution (option(‘mergeSchema’, ‘true’)) to avoid painful migrations.
• Leverage Delta’s upsert capabilities to handle incremental updates gracefully.
• Explore time travel options (option(‘versionAsOf’, <version>)) to debug or audit historical data.
• Use overwrite mode carefully, especially with partitioned tables or in production environments.
• Incorporate Delta’s delete API (deltaTable.delete(condition)) for data clean-up instead of cumbersome batch jobs.
• Monitor your Delta tables regularly for file compaction and optimize performance with OPTIMIZE commands if your environment supports it.

What Goes Wrong When Reading and Writing Delta Tables with PySpark

• Ignoring the Delta format when reading or writing leads to inconsistent data or errors that are hard to trace. Parquet syntax and Delta syntax look similar but behave very differently.
• Trying to append data with incompatible schemas without enabling schema evolution throws errors or silently corrupts your table state.
• Using overwrite mode without understanding your table’s partitioning or transactional context can cause data loss or partial overwrites.
• Not including the Delta Lake package in your Spark session will cause runtime failures, often frustrating to diagnose.
• Failing to use the merge API for upserts means reinventing error-prone and inefficient logic outside Delta Lake’s transactional guarantees.

There’s a subtlety to Delta Lake. It’s not just another file format but a powerful transactional layer on top of your data lake. Treat it as such.

I’ve seen teams stumble trying to handle concurrent writes or schema changes without leveraging Delta’s built-in features. One simple merge operation replaced days of brittle scripting for us. Delta tables changed the way we think about data reliability.

*”We do not learn from experience… we learn from reflecting on experience.”* — John Dewey

When you treat Delta tables not as static files but living, versioned entities, your data pipelines become more resilient and maintainable. Like a good conversation, reading and writing Delta tables with PySpark is about listening to what your data is telling you, and responding with precision.

Keep your Spark session clean, your schema flexible, and your merges sharp. The data will thank you. 🚀📊✨

References

• Related reading: https://brontowise.com/category/spark/