#pandas — Public Fediverse posts

Assign either np.nan or pd.NA to a #Python #Pandas series with a NumPy dtype, and it'll be np.nan, a float.

Which means the entire series has a dtype of float:

s = Series([10, 20, 30, 40])
s.loc[2] = pd.NA

Result:

0 10.0
1 20.0
2 NaN
3 40.0
dtype: float64

Assign either np.nan or pd.NA to a #Python #Pandas series with a NumPy dtype, and it'll be np.nan, a float.

Which means the entire series has a dtype of float:

s = Series([10, 20, 30, 40])
s.loc[2] = pd.NA

Result:

0 10.0
1 20.0
2 NaN
3 40.0
dtype: float64

If your dtype is too small, operations on your #Python #Pandas series will fail:

s = Series([10, 20, 30], dtype='int8')
s + 100

Returns:

0 110
1 120
2 -126 # 🤯
dtype: int8

This does give an error:

s + 500
OverflowError: Python integer 500 out of bounds for int8

If your dtype is too small, operations on your #Python #Pandas series will fail:

s = Series([10, 20, 30], dtype='int8')
s + 100

Returns:

0 110
1 120
2 -126 # 🤯
dtype: int8

This does give an error:

s + 500
OverflowError: Python integer 500 out of bounds for int8

Reading a CSV into a #Python #Pandas data frame? Use the "dtype" keyword arg and a dict to specify dtypes, and avoid the int64/float64/str defaults:

df = pd.read_csv(filename, dtype={'VendorID':'int8',
'passenger_count':'int8', 'RateCodeID':'int8',
'payment_type':'int8'})

Reading a CSV into a #Python #Pandas data frame? Use the "dtype" keyword arg and a dict to specify dtypes, and avoid the int64/float64/str defaults:

df = pd.read_csv(filename, dtype={'VendorID':'int8',
'passenger_count':'int8', 'RateCodeID':'int8',
'payment_type':'int8'})

Want to change the dtype of a #Python #Pandas series? You can't — at least, not by assigning:

df['passenger_count'].dtype = 'int32' # Error!

Instead, use "astype" to get a new series, and replace the old one:

df['passenger_count'] = df['passenger_count'].astype('int32')

Want to change the dtype of a #Python #Pandas series? You can't — at least, not by assigning:

df['passenger_count'].dtype = 'int32' # Error!

Instead, use "astype" to get a new series, and replace the old one:

df['passenger_count'] = df['passenger_count'].astype('int32')

Get the dtype of a #Python #Pandas series with dtype:

df['trip_distance'].dtype

Get the dtypes of all columns in a data frame with "dtypes", which returns a series:

df.dtypes

How many of each dtype? Use value_counts:

df.dtypes.value_counts()

Get the dtype of a #Python #Pandas series with dtype:

df['trip_distance'].dtype

Get the dtypes of all columns in a data frame with "dtypes", which returns a series:

df.dtypes

How many of each dtype? Use value_counts:

df.dtypes.value_counts()

Narwhals provides a lightweight and extensible compatibility layer between various DataFrame libraries: https://www.python4data.science/en/latest/performance/parallelise-pandas.html#narwhals
#Python #pandas #DataFrame

Narwhals provides a lightweight and extensible compatibility layer between various DataFrame libraries: https://www.python4data.science/en/latest/performance/parallelise-pandas.html#narwhals
#Python #pandas #DataFrame

Want to get a quick look at a #Python #Pandas data frame you just created?

Option 1, use df.head(n) to look at the first n rows:

df.head(5) # first 5 rows

Option 2, use df.sample(n) to look at n randomly selected rows:

df.sample(5) # 5 random rows

Want to get a quick look at a #Python #Pandas data frame you just created?

Option 1, use df.head(n) to look at the first n rows:

df.head(5) # first 5 rows

Option 2, use df.sample(n) to look at n randomly selected rows:

df.sample(5) # 5 random rows

Want to read a zipped CSV file into a #Python #Pandas data frame: Just pass the filename to read_csv:

df = pd.read_csv('data.zip')

The file can contain a single CSV file, with an extension of .gz, .bz2, .zip, .xz, .zst, .tar, .tar.gz, .tar.xz, or .tar.bz2.

Want to read a zipped CSV file into a #Python #Pandas data frame: Just pass the filename to read_csv:

df = pd.read_csv('data.zip')

The file can contain a single CSV file, with an extension of .gz, .bz2, .zip, .xz, .zst, .tar, .tar.gz, .tar.xz, or .tar.bz2.

Most #Pandas exercises use toy data sets. Bamboo Weekly uses data from coal plants, earthquakes, and Netflix views -- real (messy) data, with badly named columns.

500+ exercises with full answers, now free. Every issue is open after 2 years.

More info: https://BambooWeekly.com

Most #Pandas exercises use toy data sets. Bamboo Weekly uses data from coal plants, earthquakes, and Netflix views -- real (messy) data, with badly named columns.

500+ exercises with full answers, now free. Every issue is open after 2 years.

More info: https://BambooWeekly.com

What sheets are in an Excel document you're about to read into #Python #Pandas?

You can check with:

pd.ExcelFile(filename).sheet_names

You'll get a list of Python strings back.

What sheets are in an Excel document you're about to read into #Python #Pandas?

You can check with:

pd.ExcelFile(filename).sheet_names

You'll get a list of Python strings back.

Which is faster in #Python #Pandas, | or isin?

I prefer isin; it's clearer to write and read. Plus, fewer worries about parentheses.

But is it faster? Depends on the column type! isin edges out | on strings, but | wins on ints. Which surprised me.

Bottom line: Use %%timeit to check. Don't just guess!

Which is faster in #Python #Pandas, | or isin?

I prefer isin; it's clearer to write and read. Plus, fewer worries about parentheses.

But is it faster? Depends on the column type! isin edges out | on strings, but | wins on ints. Which surprised me.

Bottom line: Use %%timeit to check. Don't just guess!

Want rows in a #Python #Pandas data frame that might have several values? Another way (besides the | I showed yesterday) is the "isin" method:

(
df
.loc[ pd.col('passenger_count').isin([5, 7]) ]
)

We'll compare isin vs. | speed tomorrow. But I find this far more readable.

Want rows in a #Python #Pandas data frame that might have several values? Another way (besides the | I showed yesterday) is the "isin" method:

(
df
.loc[ pd.col('passenger_count').isin([5, 7]) ]
)

We'll compare isin vs. | speed tomorrow. But I find this far more readable.

Want rows in a #Python #Pandas data frame that might have several values? You can use the | operator, but be sure to use () to avoid precedence issues:

(
df
.loc[((pd.col('passenger_count') == 5) |
(pd.col('passenger_count') == 7))]
)

Want rows in a #Python #Pandas data frame that might have several values? You can use the | operator, but be sure to use () to avoid precedence issues:

(
df
.loc[((pd.col('passenger_count') == 5) |
(pd.col('passenger_count') == 7))]
)

Шиномонтаж методом Симпсона: как красивая статистика чуть не убила премиум-сервис

Что делать, если средний чек премиального автосервиса оказался в 4 раза меньше планового? Первое «очевидное» решение руководства — немедленно избавиться от дешёвой мелочёвки вроде сезонного шиномонтажа и хранения колес, которая перегружает мастеров и портит красивую статистику. Звучит логично? Абсолютно. Вот только глубокий дата-анализ показал, что такое «оптимизаторское» решение лишило бы компанию доброй трети валовой выручки. В этой статье мы разбираем реальный кейс анализа массива из сотен тысяч заказов за 10 лет работы крупного автодилера. Разберем обработку данных в Python (Pandas/Seaborn), столкнемся с Парадоксом Симпсона в действии и рассчитаем честный LTV клиентов. Вы узнаете, как сезонная переобувка работает в роли «троянского коня» и почему клиенты с шиномонтажом за свой жизненный цикл приносят компании в 3.1 раза больше денег.

https://habr.com/ru/articles/1066368/

#анализ_данных #python #pandas #ltv #бизнесанализ #продуктовая_аналитика #визуализация_данных #сегментация_клиентов #парадокс_симпсона

Stacking loc to filter a #Python #Pandas dataframe? Order can matter:

(
df
.loc[ pd.col('total_amount') > 25 ]
.loc[ pd.col('passenger_count') > 0 ]
) # 64.5 ms

(
df
.loc[ pd.col('passenger_count') > 0 ]
.loc[ pd.col('total_amount') > 25 ]
) # 95.8 ms -- 30% slower

Stacking loc to filter a #Python #Pandas dataframe? Order can matter:

(
df
.loc[ pd.col('total_amount') > 25 ]
.loc[ pd.col('passenger_count') > 0 ]
) # 64.5 ms

(
df
.loc[ pd.col('passenger_count') > 0 ]
.loc[ pd.col('total_amount') > 25 ]
) # 95.8 ms -- 30% slower

Filtering rows in a #Python #Pandas data frame? Use .loc:

(
df
.loc[ pd.col('passenger_count') > 1 ]
)

pd.col refers to the previous line's returned data frame. So we can stack them:

(
df
.loc[ pd.col('passenger_count') > 1 ]
.loc[ pd.col('total_amount') > 50 ]
)

Filtering rows in a #Python #Pandas data frame? Use .loc:

(
df
.loc[ pd.col('passenger_count') > 1 ]
)

pd.col refers to the previous line's returned data frame. So we can stack them:

(
df
.loc[ pd.col('passenger_count') > 1 ]
.loc[ pd.col('total_amount') > 50 ]
)

📢 In case you missed it: "Not an island: bringing compression to the tabular ecosystem".

Most compression libraries ask you to move in — learn their API, convert your data, leave your tools behind. We think that's backwards: compression should be a fast, compact layer *underneath* the tools you already use.

Blosc2 4.9.1: DuckDB, Polars, PyArrow and pandas 3 read a CTable directly via Arrow's PyCapsule protocol.

📖 https://blosc.org/posts/not-an-island-tabular-ecosystem/

#Arrow #DuckDB #Pandas #Compression #OpenSource

#pandas (the software) should be forbidden. I will not elaborate.

We just finished another day at #EuroPython2026 in Krakow. But the fun keeps coming -- either at a social event, or using #Python #Pandas to analyze Krakow tourism data from the Polish government!

Level up your data-analysis skills every Wednesday: https://BambooWeekly.com

We just finished another day at #EuroPython2026 in Krakow. But the fun keeps coming -- either at a social event, or using #Python #Pandas to analyze Krakow tourism data from the Polish government!

Level up your data-analysis skills every Wednesday: https://BambooWeekly.com

I'm in Krakow for #EuroPython, and as such, Bamboo Weekly is about tourism to Krakow. Get data from the Polish government, and answer data-analysis questions with #Python #Pandas!

More info: https://BambooWeekly.com

I'm in Krakow for #EuroPython, and as such, Bamboo Weekly is about tourism to Krakow. Get data from the Polish government, and answer data-analysis questions with #Python #Pandas!

More info: https://BambooWeekly.com

#NVIDIA published a blog post where they present GQE, a GPU-based query engine. Querying data from databases with GPU accelleration is beyond cool, and will certainly optimize the storage requirements for #bigdata due to enabling for more efficient compression algorithms. Here is the blog post:

Designing GPU-Accelerated Query Engines with NVIDIA GQE

#bigdata #databases #polars #pandas #datascience

#NVIDIA published a blog post where they present GQE, a GPU-based query engine. Querying data from databases with GPU accelleration is beyond cool, and will certainly optimize the storage requirements for #bigdata due to enabling for more efficient compression algorithms. Here is the blog post:

Designing GPU-Accelerated Query Engines with NVIDIA GQE

#bigdata #databases #polars #pandas #datascience

Europe has been pretty hot lately -- and in the latest Bamboo Weekly, we used #Python #Pandas, GeoPandas, Marimo, and Plotly to find out just *how* hot.

Every week, get better at data analysis: https://BambooWeekly.com

Europe has been pretty hot lately -- and in the latest Bamboo Weekly, we used #Python #Pandas, GeoPandas, Marimo, and Plotly to find out just *how* hot.

Every week, get better at data analysis: https://BambooWeekly.com

Europe has been very hot for the last few weeks. How hot? In the latest Bamboo Weekly, we use #Python #Pandas to find out, using APIs and GeoPandas. Plus, we use Marimo for interactive plots.

Level up your data skills at https://BambooWeekly.com ... and stay cool!

Europe has been very hot for the last few weeks. How hot? In the latest Bamboo Weekly, we use #Python #Pandas to find out, using APIs and GeoPandas. Plus, we use Marimo for interactive plots.

Level up your data skills at https://BambooWeekly.com ... and stay cool!

#AIP:
"
".. extrem lichtschwache Zwerggalaxie wurde in der Nähe der Andromedagalaxie (M31) erstmals beobachtet. .. Studie legt nahe, dass die Galaxie, benannt And XXXVI, eine der lichtschwächsten Satellitengalaxien im Umfeld von Andromeda ist. ..
"
".. Alter von etwa 12,5 Milliarden Jahren .. erstaunlich arm an schweren Elementen .."

https://www.aip.de/de/news/galaxy-around-andromeda/

29.6.2026

#Andromeda #AndXXXVI #Astronomie #DM #DunkleMaterie #Galaxie #GTC #Kosmologie #PAndAS #Standardmodell #Universum #Zwerggalaxie

Trying to use .loc to retrieve a slice in #Python #Pandas? If the index repeats, you need to sort it:

s = Series([10, 20, 30], index=list('aba'))

s.loc['a':'b'] # KeyError: "Cannot get left slice bound for non-unique label: 'a'"

s.sort_index().loc['a':'b'] # works!

Retrieve from a #Python #Pandas data frame with .loc, which takes 2 arguments:

1. Row selector: index, list of indexes, or a mask index (i.e., booleans)
2. Optional column selector: column name or a list of column names

.loc uses [] and not (), so you can use slices!