#pandas — Public Fediverse posts

Oh look, another "revolutionary" tech blog post telling us 🐼 #pandas should go extinct because some edgy developer discovered #Polars and DuckDB—how original! 🙄 Next, we'll be told that using Apache Arrow is like discovering fire. 🔥 Stop the presses; we've got a genius on our hands! 🎉
https://eddie.codes/posts/pandas-should-go-extinct/ #techblog #drama #edgydev #extinction #DuckDB #HackerNews #ngated

Pandas Should Go Extinct

https://eddie.codes/posts/pandas-should-go-extinct/

Comments: https://news.ycombinator.com/item?id=49668198

#HackerNews #Pandas #Should #Go #Extinct #conservation #debate #wildlife #ethics #extinction #pandas

Утечка на 3.5 часа вперёд: как модель обманывала саму себя полтора месяца — и как мы это поймали

Разрыв между «отлично работает на истории» и «сливает в реальности» — классика ML на временных рядах. В нашем случае модель заглядывала в будущее на 3.5 часа через некорректный ресемплинг 4-часовых свечей. Разбираем анатомию утечки, математику позиционного теста для её детекции и делимся сниппетом защиты от подобных ошибок.

https://habr.com/ru/articles/1080972/

#data_leakage #pandas #resample #lookahead_bias #time_series #алготрейдинг #lightgbm #валидация_данных

How are the schools in your country?

The results from PISA 2025 are out, ranking education systems in dozens of countries.

In the latest Bamboo Weekly, we use #Python #Pandas to better understand some of this data.

Level up your data-analysis skills every Wednesday: https://BambooWeekly.com

Profile raw CSVs from ETL or APIs: this 282-line pandas script flags outliers via z-score (std thresh) and outputs an HTML report. Intermediate level, ready to run. https://www.valtersit.com/python/pandas-dataframe-profiler-and-outlier-detector/ #python #data #pandas

Ugh -- the US and Canada are in a trade war.

But what does each country export to the other? How much? Which states/provinces trade the most? And what do they trade, anyway?

The latest Bamboo Weekly poses 5 #Python #Pandas challenges about US/Canada trade data.

Try it out: https://BambooWeekly.com

Another PyArrow advantage in #Python #Pandas: Changing dtypes

s_pyarrow = Series([10, 50], dtype='int8[pyarrow]')
s_numpy = Series([10, 50], dtype='int8')

s_pyarrow * 100 # all good, returns int64 dtype
s_numpy * 100 # retains int8 dtype, some negative numbers!

Using PyArrow dtypes in #Python #Pandas isn't always faster:

s_pyarr = Series(nums, dtype='int64[pyarrow]')
s_np = Series(nums, dtype='int64')

%timeit s_pyarr.mean() # 21.5ms
%timeit s_np.mean() # 47.3ms

%timeit s_pyarr.nlargest(10) # 667ms
%timeit s_np.nlargest(10) # 663ms

At this zoo, the food is as cute as the animals.

PyArrow strings in #Python #Pandas are smaller than Python strings. But they're also far faster:

s_pyarr = Series(alice)
s_py = Series(alice, dtype='object')

%timeit s_pyarr.str.len() # 106 µs
%timeit s_py.str.len() # 1.6 ms

In many examples, PyArrow was far faster.

Man sings to zoo animals, cat inserted into artwork, dog rats on owner, and panda massage.

Every Wednesday, Bamboo Weekly offers new #Python #Pandas data-analysis challenges. The next day, you get my solutions.

This week? I want to see *your* solutions, in the first-ever Bamboo Weekly Community Contest!

There are three ways to enter. And it's 100% free. So why not?

More info is at https://www.bambooweekly.com/contest/ .

This week's challenge is at https://www.bambooweekly.com/bamboo-weekly-184-parmesan-cheese/

Want PyArrow dtypes in your #Python #Pandas data frame?

df = pd.read_csv(filename, dtype_backend='pyarrow')

The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones.

Note: This is still experimental... but it's also the future.

If the dtype of your #Python #Pandas series is "object", use memory_usage(deep=True):

s_arr.memory_usage() # 161,148 -- PyArrow doesn't need "deep"
s_py.memory_usage() # 102,236 -- pointer size, not string size!
s_py.memory_usage(deep=True) # 684,477 -- actual string size

Using #Python #Pandas 3? Strings use PyArrow, not Pandas 2's Python strings (dtype "object"):

filename = 'alice'
alice = open(filename).read().split()
s_arr = Series(alice)
s_py = Series(alice, dtype='object')

s_arr.memory_usage() # 161148
s_py.memory_usage(deep=True) #684477

In #Python #Pandas 3, you can use PyArrow dtypes — which are nullable:

s = Series([10, 20, 30, 40], dtype='int64[pyarrow]')
s.loc[2] = pd.NA

What is s?

0 10
1 20
2 <NA> # pd.NA, not np.nan
3 40
dtype: int64[pyarrow] # see? Not float!

Assign either np.nan or pd.NA to a #Python #Pandas series with a NumPy dtype, and it'll be np.nan, a float.

Which means the entire series has a dtype of float:

s = Series([10, 20, 30, 40])
s.loc[2] = pd.NA

Result:

0 10.0
1 20.0
2 NaN
3 40.0
dtype: float64

If your dtype is too small, operations on your #Python #Pandas series will fail:

s = Series([10, 20, 30], dtype='int8')
s + 100

Returns:

0 110
1 120
2 -126 # 🤯
dtype: int8

This does give an error:

s + 500
OverflowError: Python integer 500 out of bounds for int8

Reading a CSV into a #Python #Pandas data frame? Use the "dtype" keyword arg and a dict to specify dtypes, and avoid the int64/float64/str defaults:

df = pd.read_csv(filename, dtype={'VendorID':'int8',
'passenger_count':'int8', 'RateCodeID':'int8',
'payment_type':'int8'})

Want to change the dtype of a #Python #Pandas series? You can't — at least, not by assigning:

df['passenger_count'].dtype = 'int32' # Error!

Instead, use "astype" to get a new series, and replace the old one:

df['passenger_count'] = df['passenger_count'].astype('int32')

Get the dtype of a #Python #Pandas series with dtype:

df['trip_distance'].dtype

Get the dtypes of all columns in a data frame with "dtypes", which returns a series:

df.dtypes

How many of each dtype? Use value_counts:

df.dtypes.value_counts()

Narwhals provides a lightweight and extensible compatibility layer between various DataFrame libraries: https://www.python4data.science/en/latest/performance/parallelise-pandas.html#narwhals
#Python #pandas #DataFrame

Want to get a quick look at a #Python #Pandas data frame you just created?

Option 1, use df.head(n) to look at the first n rows:

df.head(5) # first 5 rows

Option 2, use df.sample(n) to look at n randomly selected rows:

df.sample(5) # 5 random rows

Want to read a zipped CSV file into a #Python #Pandas data frame: Just pass the filename to read_csv:

df = pd.read_csv('data.zip')

The file can contain a single CSV file, with an extension of .gz, .bz2, .zip, .xz, .zst, .tar, .tar.gz, .tar.xz, or .tar.bz2.

Most #Pandas exercises use toy data sets. Bamboo Weekly uses data from coal plants, earthquakes, and Netflix views -- real (messy) data, with badly named columns.

500+ exercises with full answers, now free. Every issue is open after 2 years.

More info: https://BambooWeekly.com

What sheets are in an Excel document you're about to read into #Python #Pandas?

You can check with:

pd.ExcelFile(filename).sheet_names

You'll get a list of Python strings back.

Which is faster in #Python #Pandas, | or isin?

I prefer isin; it's clearer to write and read. Plus, fewer worries about parentheses.

But is it faster? Depends on the column type! isin edges out | on strings, but | wins on ints. Which surprised me.

Bottom line: Use %%timeit to check. Don't just guess!