Pandas 0.25.0 is Here. What is New? Named aggregation, explode() and sparse dataframe

If you are like me, you might have missed that the fantastic Pandas team has released the new version Pandas 0.25.0.

As one would expect, there are quite a few new things in Pandas 0.25.0. A couple of new enhancements are around pandas’ groupby aggregation. Here are a few new things that look really interesting.

To get started with pandas version 0.25.0, install

python3 -m pip install --upgrade pandas

And load the new version of pandas.

import pandas as pd
# make sure the version is pandas 0.25.0
pd.__version__

Named Aggregation with groupby

One of the interesting updates is a new groupby behavior, known as “named aggregation”. This helps naming the output columns when applying multiple aggregation functions to specific columns.

animals = pd.DataFrame({'kind': ['cat', 'dog', 'cat', 'dog'],
                        'height': [9.1, 6.0, 9.5, 34.0],
                        'weight': [7.9, 7.5, 9.9, 198.0]})

For example, if we want to compute both minimum and maximum values of height for each aniumal kind and keep them as resulting column, we can use pd.NamedAgg function as follows.

animals.groupby("kind").agg(
    min_height=pd.NamedAgg(column='height', aggfunc='min'),
    max_height=pd.NamedAgg(column='height', aggfunc='max'))

And we would get

     min_height  max_height  
kind                                        
cat          9.1         9.5           
dog          6.0        34.0     

In addition to explicitly using pd.NameddAgg() function, we can also providethe desired columns names as the **kwargs to .agg. However, the values of **kwargs should be tuples where the first element is the column selection, and the second element is the aggregation function to apply.

We will get the same result as above using the following code

animals.groupby("kind").agg(
   min_height=('height', 'min'),
   max_height=('height', 'max'))

Explode function to split list-like values to separate rows

Another interesting function in Pandas 0.25.0 is explode() method available for both Series and DataFrame objects.

For example, you might have a dataframe with a column, whose values contain multiple items separated by a delimiter. Basically, the values of the column are like a list. Sometimes you might want the elements of list to be a separate row.

This new explode() function is sort of like the new separate_rows() function in tidyverse.

Here is an example of dataframe with comma separated string in a column. And how explode can be useful in splitting them in to a separate row.

df = pd.DataFrame([{'var1': 'a,b,c', 'var2': 1},
                   {'var1': 'd,e,f', 'var2': 2}])

    var1  var2
0  a,b,c     1
1  d,e,f     2

And we can split the comma separated column values as rows.

df.assign(var1=df.var1.str.split(',')).explode('var1')

  var1  var2
0    a     1
0    b     1
0    c     1
1    d     2
1    e     2
1    f     2

SparseDataFrame is Deprecated

Another interesting change is Pandas’ SparseDataFrame subclass (and SparseSeries) is deprecated. Instead, the DataFrame function can directly take sparse values as input.

Instead of using SparseDataFrame to create a sparse dataframe like

# Old Way
pd.SparseDataFrame({"A": [0, 1]})

in the new version of pandas, one would use

# New Way
pd.DataFrame({"A": pd.SparseArray([0, 1])})

Similarly, there is a new way for dealing with sparse matrix in Pandas.

Instead of the old approach

# Old way
from scipy import sparse
mat = sparse.eye(3)
df = pd.SparseDataFrame(mat, columns=['A', 'B', 'C'])

the new version of Pandas offers

# New way
df = pd.DataFrame.sparse.from_spmatrix(mat, columns=['A', 'B', 'C'])