#pandas — Public Fediverse posts

Using PyArrow dtypes in #Python #Pandas isn't always faster:

s_pyarr = Series(nums, dtype='int64[pyarrow]')
s_np = Series(nums, dtype='int64')

%timeit s_pyarr.mean() # 21.5ms
%timeit s_np.mean() # 47.3ms

%timeit s_pyarr.nlargest(10) # 667ms
%timeit s_np.nlargest(10) # 663ms

At this zoo, the food is as cute as the animals.

PyArrow strings in #Python #Pandas are smaller than Python strings. But they're also far faster:

s_pyarr = Series(alice)
s_py = Series(alice, dtype='object')

%timeit s_pyarr.str.len() # 106 µs
%timeit s_py.str.len() # 1.6 ms

In many examples, PyArrow was far faster.

Man sings to zoo animals, cat inserted into artwork, dog rats on owner, and panda massage.

Every Wednesday, Bamboo Weekly offers new #Python #Pandas data-analysis challenges. The next day, you get my solutions.

This week? I want to see *your* solutions, in the first-ever Bamboo Weekly Community Contest!

There are three ways to enter. And it's 100% free. So why not?

More info is at https://www.bambooweekly.com/contest/ .

This week's challenge is at https://www.bambooweekly.com/bamboo-weekly-184-parmesan-cheese/

Want PyArrow dtypes in your #Python #Pandas data frame?

df = pd.read_csv(filename, dtype_backend='pyarrow')

The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones.

Note: This is still experimental... but it's also the future.

If the dtype of your #Python #Pandas series is "object", use memory_usage(deep=True):

s_arr.memory_usage() # 161,148 -- PyArrow doesn't need "deep"
s_py.memory_usage() # 102,236 -- pointer size, not string size!
s_py.memory_usage(deep=True) # 684,477 -- actual string size

Using #Python #Pandas 3? Strings use PyArrow, not Pandas 2's Python strings (dtype "object"):

filename = 'alice'
alice = open(filename).read().split()
s_arr = Series(alice)
s_py = Series(alice, dtype='object')

s_arr.memory_usage() # 161148
s_py.memory_usage(deep=True) #684477

In #Python #Pandas 3, you can use PyArrow dtypes — which are nullable:

s = Series([10, 20, 30, 40], dtype='int64[pyarrow]')
s.loc[2] = pd.NA

What is s?

0 10
1 20
2 <NA> # pd.NA, not np.nan
3 40
dtype: int64[pyarrow] # see? Not float!

Assign either np.nan or pd.NA to a #Python #Pandas series with a NumPy dtype, and it'll be np.nan, a float.

Which means the entire series has a dtype of float:

s = Series([10, 20, 30, 40])
s.loc[2] = pd.NA

Result:

0 10.0
1 20.0
2 NaN
3 40.0
dtype: float64

If your dtype is too small, operations on your #Python #Pandas series will fail:

s = Series([10, 20, 30], dtype='int8')
s + 100

Returns:

0 110
1 120
2 -126 # 🤯
dtype: int8

This does give an error:

s + 500
OverflowError: Python integer 500 out of bounds for int8

Reading a CSV into a #Python #Pandas data frame? Use the "dtype" keyword arg and a dict to specify dtypes, and avoid the int64/float64/str defaults:

df = pd.read_csv(filename, dtype={'VendorID':'int8',
'passenger_count':'int8', 'RateCodeID':'int8',
'payment_type':'int8'})

Want to change the dtype of a #Python #Pandas series? You can't — at least, not by assigning:

df['passenger_count'].dtype = 'int32' # Error!

Instead, use "astype" to get a new series, and replace the old one:

df['passenger_count'] = df['passenger_count'].astype('int32')

Get the dtype of a #Python #Pandas series with dtype:

df['trip_distance'].dtype

Get the dtypes of all columns in a data frame with "dtypes", which returns a series:

df.dtypes

How many of each dtype? Use value_counts:

df.dtypes.value_counts()

Narwhals provides a lightweight and extensible compatibility layer between various DataFrame libraries: https://www.python4data.science/en/latest/performance/parallelise-pandas.html#narwhals
#Python #pandas #DataFrame

Want to get a quick look at a #Python #Pandas data frame you just created?

Option 1, use df.head(n) to look at the first n rows:

df.head(5) # first 5 rows

Option 2, use df.sample(n) to look at n randomly selected rows:

df.sample(5) # 5 random rows

Want to read a zipped CSV file into a #Python #Pandas data frame: Just pass the filename to read_csv:

df = pd.read_csv('data.zip')

The file can contain a single CSV file, with an extension of .gz, .bz2, .zip, .xz, .zst, .tar, .tar.gz, .tar.xz, or .tar.bz2.

Most #Pandas exercises use toy data sets. Bamboo Weekly uses data from coal plants, earthquakes, and Netflix views -- real (messy) data, with badly named columns.

500+ exercises with full answers, now free. Every issue is open after 2 years.

More info: https://BambooWeekly.com

What sheets are in an Excel document you're about to read into #Python #Pandas?

You can check with:

pd.ExcelFile(filename).sheet_names

You'll get a list of Python strings back.

Which is faster in #Python #Pandas, | or isin?

I prefer isin; it's clearer to write and read. Plus, fewer worries about parentheses.

But is it faster? Depends on the column type! isin edges out | on strings, but | wins on ints. Which surprised me.

Bottom line: Use %%timeit to check. Don't just guess!

Want rows in a #Python #Pandas data frame that might have several values? Another way (besides the | I showed yesterday) is the "isin" method:

(
df
.loc[ pd.col('passenger_count').isin([5, 7]) ]
)

We'll compare isin vs. | speed tomorrow. But I find this far more readable.

Want rows in a #Python #Pandas data frame that might have several values? You can use the | operator, but be sure to use () to avoid precedence issues:

(
df
.loc[((pd.col('passenger_count') == 5) |
(pd.col('passenger_count') == 7))]
)

Шиномонтаж методом Симпсона: как красивая статистика чуть не убила премиум-сервис

Что делать, если средний чек премиального автосервиса оказался в 4 раза меньше планового? Первое «очевидное» решение руководства — немедленно избавиться от дешёвой мелочёвки вроде сезонного шиномонтажа и хранения колес, которая перегружает мастеров и портит красивую статистику. Звучит логично? Абсолютно. Вот только глубокий дата-анализ показал, что такое «оптимизаторское» решение лишило бы компанию доброй трети валовой выручки. В этой статье мы разбираем реальный кейс анализа массива из сотен тысяч заказов за 10 лет работы крупного автодилера. Разберем обработку данных в Python (Pandas/Seaborn), столкнемся с Парадоксом Симпсона в действии и рассчитаем честный LTV клиентов. Вы узнаете, как сезонная переобувка работает в роли «троянского коня» и почему клиенты с шиномонтажом за свой жизненный цикл приносят компании в 3.1 раза больше денег.

https://habr.com/ru/articles/1066368/

#анализ_данных #python #pandas #ltv #бизнесанализ #продуктовая_аналитика #визуализация_данных #сегментация_клиентов #парадокс_симпсона

Stacking loc to filter a #Python #Pandas dataframe? Order can matter:

(
df
.loc[ pd.col('total_amount') > 25 ]
.loc[ pd.col('passenger_count') > 0 ]
) # 64.5 ms

(
df
.loc[ pd.col('passenger_count') > 0 ]
.loc[ pd.col('total_amount') > 25 ]
) # 95.8 ms -- 30% slower

Filtering rows in a #Python #Pandas data frame? Use .loc:

(
df
.loc[ pd.col('passenger_count') > 1 ]
)

pd.col refers to the previous line's returned data frame. So we can stack them:

(
df
.loc[ pd.col('passenger_count') > 1 ]
.loc[ pd.col('total_amount') > 50 ]
)

📢 In case you missed it: "Not an island: bringing compression to the tabular ecosystem".

Most compression libraries ask you to move in — learn their API, convert your data, leave your tools behind. We think that's backwards: compression should be a fast, compact layer *underneath* the tools you already use.

Blosc2 4.9.1: DuckDB, Polars, PyArrow and pandas 3 read a CTable directly via Arrow's PyCapsule protocol.

📖 https://blosc.org/posts/not-an-island-tabular-ecosystem/

#Arrow #DuckDB #Pandas #Compression #OpenSource