How Do You Check Data Type in PySpark?

How Do You Check Data Type in PySpark?

In PySpark, it is often necessary to check the data type of different columns in a DataFrame. This information helps us understand the structure of our data and make informed decisions during data processing and analysis. In this tutorial, we will explore various methods to check the data type in PySpark.

Method 1: Using the printSchema() Function

The easiest way to check the data type in PySpark is by using the printSchema() function. This function displays the schema of a DataFrame, including column names and their corresponding data types. Let’s consider an example:

from pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder.getOrCreate()

# Read a CSV file into a DataFrame
data = spark.read.csv("data.csv", header=True)

# Print the schema
data.printSchema()

The output will look like this:

root
 |-- name: string
 |-- age: integer
 |-- salary: double

We can see that the “name” column has a string data type, while both “age” and “salary” have different data types, namely integer and double, respectively.

Method 2: Using the dtypes Property

An alternative method to check data types is by using the dtypes property. It returns an array of tuples containing column names and their corresponding data types.

# Get column names and their data types
column_types = data.dtypes

for column_name, column_type in column_types:
    print(f"Column '{column_name}' has data type: {column_type}")

The output will be:

Column 'name' has data type: string
Column 'age' has data type: int
Column 'salary' has data type: double

Method 3: Using the describe() Function

The describe() function provides summary statistics for columns in a DataFrame, including the data type. However, this method is more suitable for numeric columns as it provides statistical information.

# Describe the DataFrame
data.describe().show()

+-------+----+------------------+
|summary|name|               age|
+-------+----+------------------+
|  count|   5|                 5|
|   mean|null|              32.4|
| stddev|null|8.514693182963014 |
|min    |Jack|                25|
|max    |Tom |                42|
+-------+----+------------------+

In this case, we can infer that the “name” column has a string data type based on the summary statistics table.

Conclusion

In this tutorial, we explored multiple methods to check the data type in PySpark. The printSchema() function provides a comprehensive view of the entire DataFrame schema, while the dtypes property allows us to iterate over individual columns and their respective data types. Additionally, the describe() function provides summary statistics that can be used to determine the data type of numeric columns.

By utilizing these methods, you can easily check the data type in PySpark and gain a better understanding of your data for further analysis and processing.