Wsrtjtyk

I am facing a simple error with the code below.

My objective is to use simpleimputer to plug missing values of different datatypes in one shot.

When i try to do that, the fit_transform seems to be not work as expected.
When dtype argument is not used, the code works just fine, but the resulting dataframe loses its data type information. When i include the dtype list in the arguments, i am seeing the below error. You should be able to simulate the error by just copying and pasting in your local.

import pandas as pd

import numpy as np

from sklearn.impute import SimpleImputer



import sklearn

print(sklearn.__version__)



0.21.dev0



data = [['Alex','NJ',21,5.10],['Mary','NY',20,np.nan],['Sam',np.nan,np.nan,6.3]]

df = pd.DataFrame(data,columns=['Name','State','Age','Height'])



df.dtypes

Name       object

State      object

Age       float64

Height    float64

dtype: object                 



imp = SimpleImputer(strategy="most_frequent")



#df = pd.DataFrame(imp.fit_transform(df),columns=df.columns)   <<<<----- This works just fine

#df

#Name   State   Age Height

#0  Alex    NJ  21  5.1

#1  Mary    NY  20  5.1

#2  Sam NJ  20  6.3

#df.dtypes

#Name      object

#State     object

#Age       object

#Height    object

#dtype: object

The below statement fails - with the error listed below ( I am trying to preserve dtypes during imputing process)

df = pd.DataFrame(imp.fit_transform(df),columns=df.columns,dtype=df.dtypes)

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-23-e9780979921f> in <module>()

      7 

      8 imp = SimpleImputer(strategy="most_frequent")

----> 9 df = pd.DataFrame(imp.fit_transform(df),columns=df.columns,dtype=df.dtypes)



~AppDataLocalContinuumanaconda3libsite-packagespandascoreframe.py in __init__(self, data, index, columns, dtype, copy)

    337             data = {}

    338         if dtype is not None:

--> 339             dtype = self._validate_dtype(dtype)

    340 

    341         if isinstance(data, DataFrame):



~AppDataLocalContinuumanaconda3libsite-packagespandascoregeneric.py in _validate_dtype(self, dtype)

    166 

    167         if dtype is not None:

--> 168             dtype = pandas_dtype(dtype)

    169 

    170             # a compound dtype



~AppDataLocalContinuumanaconda3libsite-packagespandascoredtypescommon.py in pandas_dtype(dtype)

   2020     # which we safeguard against by catching them earlier and returning

   2021     # np.dtype(valid_dtype) before this condition is evaluated.

-> 2022     if dtype in [object, np.object_, 'object', 'O']:

   2023         return npdtype

   2024     elif npdtype.kind == 'O':



~AppDataLocalContinuumanaconda3libsite-packagespandascoregeneric.py in __nonzero__(self)

   1574         raise ValueError("The truth value of a {0} is ambiguous. "

   1575                          "Use a.empty, a.bool(), a.item(), a.any() or a.all()."

-> 1576                          .format(self.__class__.__name__))

   1577 

   1578     __bool__ = __nonzero__



ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

If you want to preserve the dtype, I recommend using pandas to find the mode and then call fillna:

df = df.fillna(df.agg(lambda x: pd.Series.mode(x)[0], axis=0))

print(df)

   Name State   Age  Height

0  Alex    NJ  21.0     5.1

1  Mary    NY  20.0     5.1

2   Sam    NJ  20.0     6.3



print(df.dtypes)

Name       object

State      object

Age       float64

Height    float64

dtype: object

Alternatively, use astype and pass a dictionary:

df = pd.DataFrame(

     imp.fit_transform(df), columns=df.columns

).astype(df.dtypes.to_dict())



print(df)

   Name State   Age  Height

0  Alex    NJ  21.0     5.1

1  Mary    NY  20.0     5.1

2   Sam    NJ  20.0     6.3



print(df.dtypes)

Name       object

State      object

Age       float64

Height    float64

dtype: object

Explicit astype call is needed because, as per the documentation, only a single dtype can be passed to the pd.DataFrame constructor.

?pd.DataFrame

...

dtype : dtype, default None

 |      Data type to force. Only a single dtype is allowed.