ChatGPT解决这个技术问题 Extra ChatGPT

PySpark: How to fillna values in dataframe for specific columns?

I have the following sample DataFrame:

a    | b    | c   | 

1    | 2    | 4   |
0    | null | null| 
null | 3    | 4   |

And I want to replace null values only in the first 2 columns - Column "a" and "b":

a    | b    | c   | 

1    | 2    | 4   |
0    | 0    | null| 
0    | 3    | 4   |

Here is the code to create sample dataframe:

rdd = sc.parallelize([(1,2,4), (0,None,None), (None,3,4)])
df2 = sqlContext.createDataFrame(rdd, ["a", "b", "c"])

I know how to replace all null values using:

df2 = df2.fillna(0)

And when I try this, I lose the third column:

df2 = df2.select(df2.columns[0:1]).fillna(0)
You can find more options here: stackoverflow.com/a/65811297/38368

E
Endre Both
df.fillna(0, subset=['a', 'b'])

There is a parameter named subset to choose the columns unless your spark version is lower than 1.3.1


wouldn't this give a new df with only one column? anyway to do this in place
@Fizi in Spark, Dataframe is immutable, which means it's impossible to do this in place.
s
scottlittle

Use a dictionary to fill values of certain columns:

df.fillna( { 'a':0, 'b':0 } )

This is a better answer because it does not matter wether it is one or many values being filled in.
@ChrisMarotta Does the values type of all selected columns have to be of same type? Could it also be possible: df.fillna( { 'a':0, 'b':'2022-12-01' } ) where column a is of numeric type, and b is of date type?
@nam, I suggest you fire up a pyspark terminal and find out