#pandas — Public Fediverse posts

Ugh -- the US and Canada are in a trade war.

But what does each country export to the other? How much? Which states/provinces trade the most? And what do they trade, anyway?

The latest Bamboo Weekly poses 5 #Python #Pandas challenges about US/Canada trade data.

Try it out: https://BambooWeekly.com

Another PyArrow advantage in #Python #Pandas: Changing dtypes

s_pyarrow = Series([10, 50], dtype='int8[pyarrow]')
s_numpy = Series([10, 50], dtype='int8')

s_pyarrow * 100 # all good, returns int64 dtype
s_numpy * 100 # retains int8 dtype, some negative numbers!

Using PyArrow dtypes in #Python #Pandas isn't always faster:

s_pyarr = Series(nums, dtype='int64[pyarrow]')
s_np = Series(nums, dtype='int64')

%timeit s_pyarr.mean() # 21.5ms
%timeit s_np.mean() # 47.3ms

%timeit s_pyarr.nlargest(10) # 667ms
%timeit s_np.nlargest(10) # 663ms

At this zoo, the food is as cute as the animals.

PyArrow strings in #Python #Pandas are smaller than Python strings. But they're also far faster:

s_pyarr = Series(alice)
s_py = Series(alice, dtype='object')

%timeit s_pyarr.str.len() # 106 µs
%timeit s_py.str.len() # 1.6 ms

In many examples, PyArrow was far faster.

Man sings to zoo animals, cat inserted into artwork, dog rats on owner, and panda massage.

Every Wednesday, Bamboo Weekly offers new #Python #Pandas data-analysis challenges. The next day, you get my solutions.

This week? I want to see *your* solutions, in the first-ever Bamboo Weekly Community Contest!

There are three ways to enter. And it's 100% free. So why not?

More info is at https://www.bambooweekly.com/contest/ .

This week's challenge is at https://www.bambooweekly.com/bamboo-weekly-184-parmesan-cheese/

Want PyArrow dtypes in your #Python #Pandas data frame?

df = pd.read_csv(filename, dtype_backend='pyarrow')

The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones.

Note: This is still experimental... but it's also the future.

If the dtype of your #Python #Pandas series is "object", use memory_usage(deep=True):

s_arr.memory_usage() # 161,148 -- PyArrow doesn't need "deep"
s_py.memory_usage() # 102,236 -- pointer size, not string size!
s_py.memory_usage(deep=True) # 684,477 -- actual string size

Using #Python #Pandas 3? Strings use PyArrow, not Pandas 2's Python strings (dtype "object"):

filename = 'alice'
alice = open(filename).read().split()
s_arr = Series(alice)
s_py = Series(alice, dtype='object')

s_arr.memory_usage() # 161148
s_py.memory_usage(deep=True) #684477

In #Python #Pandas 3, you can use PyArrow dtypes — which are nullable:

s = Series([10, 20, 30, 40], dtype='int64[pyarrow]')
s.loc[2] = pd.NA

What is s?

0 10
1 20
2 <NA> # pd.NA, not np.nan
3 40
dtype: int64[pyarrow] # see? Not float!

Assign either np.nan or pd.NA to a #Python #Pandas series with a NumPy dtype, and it'll be np.nan, a float.

Which means the entire series has a dtype of float:

s = Series([10, 20, 30, 40])
s.loc[2] = pd.NA

Result:

0 10.0
1 20.0
2 NaN
3 40.0
dtype: float64

If your dtype is too small, operations on your #Python #Pandas series will fail:

s = Series([10, 20, 30], dtype='int8')
s + 100

Returns:

0 110
1 120
2 -126 # 🤯
dtype: int8

This does give an error:

s + 500
OverflowError: Python integer 500 out of bounds for int8

Reading a CSV into a #Python #Pandas data frame? Use the "dtype" keyword arg and a dict to specify dtypes, and avoid the int64/float64/str defaults:

df = pd.read_csv(filename, dtype={'VendorID':'int8',
'passenger_count':'int8', 'RateCodeID':'int8',
'payment_type':'int8'})

Want to change the dtype of a #Python #Pandas series? You can't — at least, not by assigning:

df['passenger_count'].dtype = 'int32' # Error!

Instead, use "astype" to get a new series, and replace the old one:

df['passenger_count'] = df['passenger_count'].astype('int32')

Get the dtype of a #Python #Pandas series with dtype:

df['trip_distance'].dtype

Get the dtypes of all columns in a data frame with "dtypes", which returns a series:

df.dtypes

How many of each dtype? Use value_counts:

df.dtypes.value_counts()

Narwhals provides a lightweight and extensible compatibility layer between various DataFrame libraries: https://www.python4data.science/en/latest/performance/parallelise-pandas.html#narwhals
#Python #pandas #DataFrame

Want to get a quick look at a #Python #Pandas data frame you just created?

Option 1, use df.head(n) to look at the first n rows:

df.head(5) # first 5 rows

Option 2, use df.sample(n) to look at n randomly selected rows:

df.sample(5) # 5 random rows

Want to read a zipped CSV file into a #Python #Pandas data frame: Just pass the filename to read_csv:

df = pd.read_csv('data.zip')

The file can contain a single CSV file, with an extension of .gz, .bz2, .zip, .xz, .zst, .tar, .tar.gz, .tar.xz, or .tar.bz2.

Most #Pandas exercises use toy data sets. Bamboo Weekly uses data from coal plants, earthquakes, and Netflix views -- real (messy) data, with badly named columns.

500+ exercises with full answers, now free. Every issue is open after 2 years.

More info: https://BambooWeekly.com

What sheets are in an Excel document you're about to read into #Python #Pandas?

You can check with:

pd.ExcelFile(filename).sheet_names

You'll get a list of Python strings back.

Which is faster in #Python #Pandas, | or isin?

I prefer isin; it's clearer to write and read. Plus, fewer worries about parentheses.

But is it faster? Depends on the column type! isin edges out | on strings, but | wins on ints. Which surprised me.

Bottom line: Use %%timeit to check. Don't just guess!

Want rows in a #Python #Pandas data frame that might have several values? Another way (besides the | I showed yesterday) is the "isin" method:

(
df
.loc[ pd.col('passenger_count').isin([5, 7]) ]
)

We'll compare isin vs. | speed tomorrow. But I find this far more readable.

Want rows in a #Python #Pandas data frame that might have several values? You can use the | operator, but be sure to use () to avoid precedence issues:

(
df
.loc[((pd.col('passenger_count') == 5) |
(pd.col('passenger_count') == 7))]
)

Шиномонтаж методом Симпсона: как красивая статистика чуть не убила премиум-сервис

Что делать, если средний чек премиального автосервиса оказался в 4 раза меньше планового? Первое «очевидное» решение руководства — немедленно избавиться от дешёвой мелочёвки вроде сезонного шиномонтажа и хранения колес, которая перегружает мастеров и портит красивую статистику. Звучит логично? Абсолютно. Вот только глубокий дата-анализ показал, что такое «оптимизаторское» решение лишило бы компанию доброй трети валовой выручки. В этой статье мы разбираем реальный кейс анализа массива из сотен тысяч заказов за 10 лет работы крупного автодилера. Разберем обработку данных в Python (Pandas/Seaborn), столкнемся с Парадоксом Симпсона в действии и рассчитаем честный LTV клиентов. Вы узнаете, как сезонная переобувка работает в роли «троянского коня» и почему клиенты с шиномонтажом за свой жизненный цикл приносят компании в 3.1 раза больше денег.

https://habr.com/ru/articles/1066368/

#анализ_данных #python #pandas #ltv #бизнесанализ #продуктовая_аналитика #визуализация_данных #сегментация_клиентов #парадокс_симпсона

Stacking loc to filter a #Python #Pandas dataframe? Order can matter:

(
df
.loc[ pd.col('total_amount') > 25 ]
.loc[ pd.col('passenger_count') > 0 ]
) # 64.5 ms

(
df
.loc[ pd.col('passenger_count') > 0 ]
.loc[ pd.col('total_amount') > 25 ]
) # 95.8 ms -- 30% slower

Filtering rows in a #Python #Pandas data frame? Use .loc:

(
df
.loc[ pd.col('passenger_count') > 1 ]
)

pd.col refers to the previous line's returned data frame. So we can stack them:

(
df
.loc[ pd.col('passenger_count') > 1 ]
.loc[ pd.col('total_amount') > 50 ]
)

#AIP:
"
".. extrem lichtschwache Zwerggalaxie wurde in der Nähe der Andromedagalaxie (M31) erstmals beobachtet. .. Studie legt nahe, dass die Galaxie, benannt And XXXVI, eine der lichtschwächsten Satellitengalaxien im Umfeld von Andromeda ist. ..
"
".. Alter von etwa 12,5 Milliarden Jahren .. erstaunlich arm an schweren Elementen .."

https://www.aip.de/de/news/galaxy-around-andromeda/

29.6.2026

#Andromeda #AndXXXVI #Astronomie #DM #DunkleMaterie #Galaxie #GTC #Kosmologie #PAndAS #Standardmodell #Universum #Zwerggalaxie

От «обезьяньей» работы к Smart-анализу: как выполнить предобработку данных для моделей

От «обезьяньей» работы к Smart-анализу: как правильно готовить данные для моделей. Что такое Exploratory Data Analysis и как избежать основных ошибок при его выполнении.

https://habr.com/ru/articles/975082/

#pandas #sklearn #data_science #exploratory_data_analysis #machine_learning #numpy #statistics #feature_engineering

If I had to explain my job... #IOER_FDz .. I'd just show this workflow from the last two days. It's a great example of building reproducible, automated documentation. 🚀

1. Start: A project schema, drafted by colleagues in Excel.

2. Problem: Excel isn't reproducible and is hard to use as a single source of truth for a database.
Solution: I migrated the schema to the universal Protobuf format, making it version-controlled and language-neutral. [1]

3. Problem: Protobuf definitions aren't easy for everyone to read.
Solution: Using an existing #CI/#CD workflow from [3] (cheers @mcnesium), I put the schema on a documentation website. Now it's accessible and legible. [4]

4. Problem: Colleagues still needed a familiar template for data collection.
Solution: Two Python scripts using #pandas now auto-generates both `.xlsx` and open-source `.ods` schema [5] and templates [6] directly from the Protobuf single-point-of-truth, and make all of these available for download, too.

5. Problem: The text-based schema wasn't visual enough.
Solution: Added a script [7] to the CI that transforms the Protobuf files into a #MermaidJS class diagram, rendered directly on the docs site. [8]

6. Problem: The diagram was too narrow in the site's layout.
Solution: The script now generates a second, wide-format version of the diagram for a dedicated fullscreen view. [9]

7. Problem: The static diagram was hard to edit or restyle.
Solution: The CI now generates a shareable link to the Mermaid Live Editor [10]. It reads our latest diagram file, compresses it (using pako/zlib), and bakes it into the URL. Thanks to a code snippet from a friendly stranger on GitHub [11], anyone can now open the *latest* version of the schema in the editor with a single click.

From a static Excel file to version-controlled, multi-format, visual, and interactive documentation, all fully automated. That's the job.

#Automation #GitLab #Python #Protobuf #OpenData #Documentation

@ioer

[1]: https://gitlab.vgiscience.de/casereports/rwlschema
[2]: https://docs.casereports.fdz.ioer.info/
[3]: https://lbsn.vgiscience.org
[4]: https://docs.casereports.fdz.ioer.info/protobuf/
[5]: https://gitlab.vgiscience.de/casereports/rwlschema/-/blob/main/resources/generate_schema.py?ref_type=heads
[6]: https://gitlab.vgiscience.de/casereports/rwlschema/-/blob/main/resources/generate_template.py?ref_type=heads
[7]: https://gitlab.vgiscience.de/casereports/rwlschema/-/blob/main/resources/generate_proto_uml.sh?ref_type=heads
[8]: https://docs.casereports.fdz.ioer.info/konzept/
[9]: https://docs.casereports.fdz.ioer.info/uml/fullscreen.html
[10]: https://mermaidlive.com/play
[11]: https://github.com/mermaid-js/mermaid-live-editor/discussions/1291#discussioncomment-6837936

Looking for something to do after lunch? There are a bunch of open spaces tarting at 2:00 PM:

Room 308: AI for data science
Room 309: CTF tips & tricks
Room 315: Free-threaded #Python
Room 316: #WheelNext - Let's re-invent the Wheel
Room 318: All things tabular (#Pandas, #Polars, @duckdb

https://us.pycon.org/2025/schedule/open-spaces/

#PyConUS #PyConUS2025 #PyConUSOpenSpaces

But how hard did you try... if you squint, maybe there's a digit in there somewhere and with a little decluttering we could get to an integer.. from there, it's almost a double... give it a go... #typesafety - talk to your kids about it before someone shows them #pandas

#Bales2023FilmChallenge March 16: a #panda spotted in a #movie for #NationalPandaDay

In ON THE MARRIAGE BROKER #JOKE AS CITED BY #SIGMUNDFREUD IN “WIT AND ITS RELATION TO THE UNCONSCIOUS” OR CAN THE AVANT-GARDE ARTIST BE WHOLED?” (1977) Owen Land explores meaning, wit, and #WordPlay, and manages to unite the #marketing of #umeboshi #plums in a wide variety of vessels, the brokering of brides, and pandas discussing #Freud in all of the above contexts.

#film #OwenLand #AvantGarde #Pandas #ShortFilm #FilmMastodon #CineMastodon @film https://letterboxd.com/12pt9/list/bales2023filmchallenge/

2001-2022 #ChicagoCrimes data loaded from a #parquet file and summarized with #pandas #dataFrames and simple #Altair charts 📊📈 in a #JupyterNotebook:

👀 https://github.com/RandomFractals/chicago-crimes/blob/main/docs/images/chicago-crimes-altair-charts-ipynb.gif

📓 https://github.com/RandomFractals/chicago-crimes/blob/main/notebooks/chicago-crimes-altair.ipynb

#dataNotebooks 📚 ...

So, I ported my old #ChicagoCrimes #pandas data wrestling with #matplotlib plots to new repo. See this #JupyterNotebook with many crime data summaries and plots from 2001 to present:

🧐 https://github.com/RandomFractals/chicago-crimes/blob/main/notebooks/chicago-crimes-pandas.ipynb

#dataNotebooks 📚 ...

Kia ora tātou,大家好, howdy 👋

I was quite content just lurking on The Other Site. Mastodon needs all the posts it can get though, so you may hear about (no promises):

- #NZPol stuff: #Union, #ClimateJustice, #LGBTQ, #Disability, #Chinese diaspora stuff
- Work stuff: #Python, #Pandas, #Julia, #EResearch, #opensource #DigitalHumanities
- Life: #Cycling, my progress in learning #Sailing, te reo #Māori; other people's #Cats and #Dogs

Send me #Hashtags!
#Introduction 🌈