#pandas — Public Fediverse posts

Oh look, another "revolutionary" tech blog post telling us 🐼 #pandas should go extinct because some edgy developer discovered #Polars and DuckDB—how original! 🙄 Next, we'll be told that using Apache Arrow is like discovering fire. 🔥 Stop the presses; we've got a genius on our hands! 🎉
https://eddie.codes/posts/pandas-should-go-extinct/ #techblog #drama #edgydev #extinction #DuckDB #HackerNews #ngated

Oh look, another "revolutionary" tech blog post telling us 🐼 #pandas should go extinct because some edgy developer discovered #Polars and DuckDB—how original! 🙄 Next, we'll be told that using Apache Arrow is like discovering fire. 🔥 Stop the presses; we've got a genius on our hands! 🎉
https://eddie.codes/posts/pandas-should-go-extinct/ #techblog #drama #edgydev #extinction #DuckDB #HackerNews #ngated

Oh look, another "revolutionary" tech blog post telling us 🐼 #pandas should go extinct because some edgy developer discovered #Polars and DuckDB—how original! 🙄 Next, we'll be told that using Apache Arrow is like discovering fire. 🔥 Stop the presses; we've got a genius on our hands! 🎉
https://eddie.codes/posts/pandas-should-go-extinct/ #techblog #drama #edgydev #extinction #DuckDB #HackerNews #ngated

Oh look, another "revolutionary" tech blog post telling us 🐼 #pandas should go extinct because some edgy developer discovered #Polars and DuckDB—how original! 🙄 Next, we'll be told that using Apache Arrow is like discovering fire. 🔥 Stop the presses; we've got a genius on our hands! 🎉
https://eddie.codes/posts/pandas-should-go-extinct/ #techblog #drama #edgydev #extinction #DuckDB #HackerNews #ngated

Oh look, another "revolutionary" tech blog post telling us 🐼 #pandas should go extinct because some edgy developer discovered #Polars and DuckDB—how original! 🙄 Next, we'll be told that using Apache Arrow is like discovering fire. 🔥 Stop the presses; we've got a genius on our hands! 🎉
https://eddie.codes/posts/pandas-should-go-extinct/ #techblog #drama #edgydev #extinction #DuckDB #HackerNews #ngated

Pandas Should Go Extinct

https://eddie.codes/posts/pandas-should-go-extinct/

Comments: https://news.ycombinator.com/item?id=49668198

#HackerNews #Pandas #Should #Go #Extinct #conservation #debate #wildlife #ethics #extinction #pandas

Pandas Should Go Extinct

https://eddie.codes/posts/pandas-should-go-extinct/

Comments: https://news.ycombinator.com/item?id=49668198

#HackerNews #Pandas #Should #Go #Extinct #conservation #debate #wildlife #ethics #extinction #pandas

Pandas Should Go Extinct

https://eddie.codes/posts/pandas-should-go-extinct/

Comments: https://news.ycombinator.com/item?id=49668198

#HackerNews #Pandas #Should #Go #Extinct #conservation #debate #wildlife #ethics #extinction #pandas

Pandas Should Go Extinct

https://eddie.codes/posts/pandas-should-go-extinct/

Comments: https://news.ycombinator.com/item?id=49668198

#HackerNews #Pandas #Should #Go #Extinct #conservation #debate #wildlife #ethics #extinction #pandas

Pandas Should Go Extinct

https://eddie.codes/posts/pandas-should-go-extinct/

Comments: https://news.ycombinator.com/item?id=49668198

#HackerNews #Pandas #Should #Go #Extinct #conservation #debate #wildlife #ethics #extinction #pandas

Утечка на 3.5 часа вперёд: как модель обманывала саму себя полтора месяца — и как мы это поймали

Разрыв между «отлично работает на истории» и «сливает в реальности» — классика ML на временных рядах. В нашем случае модель заглядывала в будущее на 3.5 часа через некорректный ресемплинг 4-часовых свечей. Разбираем анатомию утечки, математику позиционного теста для её детекции и делимся сниппетом защиты от подобных ошибок.

https://habr.com/ru/articles/1080972/

#data_leakage #pandas #resample #lookahead_bias #time_series #алготрейдинг #lightgbm #валидация_данных

Утечка на 3.5 часа вперёд: как модель обманывала саму себя полтора месяца — и как мы это поймали

Разрыв между «отлично работает на истории» и «сливает в реальности» — классика ML на временных рядах. В нашем случае модель заглядывала в будущее на 3.5 часа через некорректный ресемплинг 4-часовых свечей. Разбираем анатомию утечки, математику позиционного теста для её детекции и делимся сниппетом защиты от подобных ошибок.

https://habr.com/ru/articles/1080972/

#data_leakage #pandas #resample #lookahead_bias #time_series #алготрейдинг #lightgbm #валидация_данных

Утечка на 3.5 часа вперёд: как модель обманывала саму себя полтора месяца — и как мы это поймали

Разрыв между «отлично работает на истории» и «сливает в реальности» — классика ML на временных рядах. В нашем случае модель заглядывала в будущее на 3.5 часа через некорректный ресемплинг 4-часовых свечей. Разбираем анатомию утечки, математику позиционного теста для её детекции и делимся сниппетом защиты от подобных ошибок.

https://habr.com/ru/articles/1080972/

#data_leakage #pandas #resample #lookahead_bias #time_series #алготрейдинг #lightgbm #валидация_данных

How are the schools in your country?

The results from PISA 2025 are out, ranking education systems in dozens of countries.

In the latest Bamboo Weekly, we use #Python #Pandas to better understand some of this data.

Level up your data-analysis skills every Wednesday: https://BambooWeekly.com

Profile raw CSVs from ETL or APIs: this 282-line pandas script flags outliers via z-score (std thresh) and outputs an HTML report. Intermediate level, ready to run. https://www.valtersit.com/python/pandas-dataframe-profiler-and-outlier-detector/ #python #data #pandas

Ugh -- the US and Canada are in a trade war.

But what does each country export to the other? How much? Which states/provinces trade the most? And what do they trade, anyway?

The latest Bamboo Weekly poses 5 #Python #Pandas challenges about US/Canada trade data.

Try it out: https://BambooWeekly.com

Another PyArrow advantage in #Python #Pandas: Changing dtypes

s_pyarrow = Series([10, 50], dtype='int8[pyarrow]')
s_numpy = Series([10, 50], dtype='int8')

s_pyarrow * 100 # all good, returns int64 dtype
s_numpy * 100 # retains int8 dtype, some negative numbers!

Using PyArrow dtypes in #Python #Pandas isn't always faster:

s_pyarr = Series(nums, dtype='int64[pyarrow]')
s_np = Series(nums, dtype='int64')

%timeit s_pyarr.mean() # 21.5ms
%timeit s_np.mean() # 47.3ms

%timeit s_pyarr.nlargest(10) # 667ms
%timeit s_np.nlargest(10) # 663ms

At this zoo, the food is as cute as the animals.

PyArrow strings in #Python #Pandas are smaller than Python strings. But they're also far faster:

s_pyarr = Series(alice)
s_py = Series(alice, dtype='object')

%timeit s_pyarr.str.len() # 106 µs
%timeit s_py.str.len() # 1.6 ms

In many examples, PyArrow was far faster.

Man sings to zoo animals, cat inserted into artwork, dog rats on owner, and panda massage.

Every Wednesday, Bamboo Weekly offers new #Python #Pandas data-analysis challenges. The next day, you get my solutions.

This week? I want to see *your* solutions, in the first-ever Bamboo Weekly Community Contest!

There are three ways to enter. And it's 100% free. So why not?

More info is at https://www.bambooweekly.com/contest/ .

This week's challenge is at https://www.bambooweekly.com/bamboo-weekly-184-parmesan-cheese/

Want PyArrow dtypes in your #Python #Pandas data frame?

df = pd.read_csv(filename, dtype_backend='pyarrow')

The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones.

Note: This is still experimental... but it's also the future.

If the dtype of your #Python #Pandas series is "object", use memory_usage(deep=True):

s_arr.memory_usage() # 161,148 -- PyArrow doesn't need "deep"
s_py.memory_usage() # 102,236 -- pointer size, not string size!
s_py.memory_usage(deep=True) # 684,477 -- actual string size

Using #Python #Pandas 3? Strings use PyArrow, not Pandas 2's Python strings (dtype "object"):

filename = 'alice'
alice = open(filename).read().split()
s_arr = Series(alice)
s_py = Series(alice, dtype='object')

s_arr.memory_usage() # 161148
s_py.memory_usage(deep=True) #684477

In #Python #Pandas 3, you can use PyArrow dtypes — which are nullable:

s = Series([10, 20, 30, 40], dtype='int64[pyarrow]')
s.loc[2] = pd.NA

What is s?

0 10
1 20
2 <NA> # pd.NA, not np.nan
3 40
dtype: int64[pyarrow] # see? Not float!

Assign either np.nan or pd.NA to a #Python #Pandas series with a NumPy dtype, and it'll be np.nan, a float.

Which means the entire series has a dtype of float:

s = Series([10, 20, 30, 40])
s.loc[2] = pd.NA

Result:

0 10.0
1 20.0
2 NaN
3 40.0
dtype: float64

If your dtype is too small, operations on your #Python #Pandas series will fail:

s = Series([10, 20, 30], dtype='int8')
s + 100

Returns:

0 110
1 120
2 -126 # 🤯
dtype: int8

This does give an error:

s + 500
OverflowError: Python integer 500 out of bounds for int8

Reading a CSV into a #Python #Pandas data frame? Use the "dtype" keyword arg and a dict to specify dtypes, and avoid the int64/float64/str defaults:

df = pd.read_csv(filename, dtype={'VendorID':'int8',
'passenger_count':'int8', 'RateCodeID':'int8',
'payment_type':'int8'})

Want to change the dtype of a #Python #Pandas series? You can't — at least, not by assigning:

df['passenger_count'].dtype = 'int32' # Error!

Instead, use "astype" to get a new series, and replace the old one:

df['passenger_count'] = df['passenger_count'].astype('int32')

Get the dtype of a #Python #Pandas series with dtype:

df['trip_distance'].dtype

Get the dtypes of all columns in a data frame with "dtypes", which returns a series:

df.dtypes

How many of each dtype? Use value_counts:

df.dtypes.value_counts()

Narwhals provides a lightweight and extensible compatibility layer between various DataFrame libraries: https://www.python4data.science/en/latest/performance/parallelise-pandas.html#narwhals
#Python #pandas #DataFrame

Want to get a quick look at a #Python #Pandas data frame you just created?

Option 1, use df.head(n) to look at the first n rows:

df.head(5) # first 5 rows

Option 2, use df.sample(n) to look at n randomly selected rows:

df.sample(5) # 5 random rows

Want to read a zipped CSV file into a #Python #Pandas data frame: Just pass the filename to read_csv:

df = pd.read_csv('data.zip')

The file can contain a single CSV file, with an extension of .gz, .bz2, .zip, .xz, .zst, .tar, .tar.gz, .tar.xz, or .tar.bz2.

Most #Pandas exercises use toy data sets. Bamboo Weekly uses data from coal plants, earthquakes, and Netflix views -- real (messy) data, with badly named columns.

500+ exercises with full answers, now free. Every issue is open after 2 years.

More info: https://BambooWeekly.com

What sheets are in an Excel document you're about to read into #Python #Pandas?

You can check with:

pd.ExcelFile(filename).sheet_names

You'll get a list of Python strings back.

Which is faster in #Python #Pandas, | or isin?

I prefer isin; it's clearer to write and read. Plus, fewer worries about parentheses.

But is it faster? Depends on the column type! isin edges out | on strings, but | wins on ints. Which surprised me.

Bottom line: Use %%timeit to check. Don't just guess!

Hej, jeśli ktoś z Was ma możliwość, pomożecie młodej osobie z wieloma chorobami przewlekłymi?
Znam ją z Instagrama, jest cudowna, ale ma wyjątkowo ciężko ❤️‍🩹
Napisała dzisiaj, że nie chce być statystyką. Nie będziesz, Ash. Nie będziesz szła sama, nigdy.

https://www.siepomaga.pl/natalka-mancewicz?utm_source=ig&utm_medium=social&utm_content=link_in_bio&fbclid=PAb21jcARxedZleHRuA2FlbQIxMQBzcnRjBmFwcF9pZA81NjcwNjczNDMzNTI0MjcAAafbIyJMeExLjbCtdHr8naL5qQwZde0Xa4PVSPZN7fPLrcWJTZjvLjSfLo2xIg_aem_5NyNKtwwDOsT8M7KU0qbww

#mutualaid #autisticsolidarity #pomocwzajemna #opór #spoonie #chorobaprzewlekla #niepełnosprawność #pots #eds #pandas #mecfs #zapaleniemózgu #stomia #pozajelitowe #spektrumautyzmu #solidarność #zrzutka #zbiórka #ozn

Hej, jeśli ktoś z Was ma możliwość, pomożecie młodej osobie z wieloma chorobami przewlekłymi?
Znam ją z Instagrama, jest cudowna, ale ma wyjątkowo ciężko ❤️‍🩹
Napisała dzisiaj, że nie chce być statystyką. Nie będziesz, Ash. Nie będziesz szła sama, nigdy.

https://www.siepomaga.pl/natalka-mancewicz?utm_source=ig&utm_medium=social&utm_content=link_in_bio&fbclid=PAb21jcARxedZleHRuA2FlbQIxMQBzcnRjBmFwcF9pZA81NjcwNjczNDMzNTI0MjcAAafbIyJMeExLjbCtdHr8naL5qQwZde0Xa4PVSPZN7fPLrcWJTZjvLjSfLo2xIg_aem_5NyNKtwwDOsT8M7KU0qbww

#mutualaid #autisticsolidarity #pomocwzajemna #opór #spoonie #chorobaprzewlekla #niepełnosprawność #pots #eds #pandas #mecfs #zapaleniemózgu #stomia #pozajelitowe #spektrumautyzmu #solidarność #zrzutka #zbiórka #ozn

Hej, jeśli ktoś z Was ma możliwość, pomożecie młodej osobie z wieloma chorobami przewlekłymi?
Znam ją z Instagrama, jest cudowna, ale ma wyjątkowo ciężko ❤️‍🩹
Napisała dzisiaj, że nie chce być statystyką. Nie będziesz, Ash. Nie będziesz szła sama, nigdy.

https://www.siepomaga.pl/natalka-mancewicz?utm_source=ig&utm_medium=social&utm_content=link_in_bio&fbclid=PAb21jcARxedZleHRuA2FlbQIxMQBzcnRjBmFwcF9pZA81NjcwNjczNDMzNTI0MjcAAafbIyJMeExLjbCtdHr8naL5qQwZde0Xa4PVSPZN7fPLrcWJTZjvLjSfLo2xIg_aem_5NyNKtwwDOsT8M7KU0qbww

#mutualaid #autisticsolidarity #pomocwzajemna #opór #spoonie #chorobaprzewlekla #niepełnosprawność #pots #eds #pandas #mecfs #zapaleniemózgu #stomia #pozajelitowe #spektrumautyzmu #solidarność #zrzutka #zbiórka #ozn

Hej, jeśli ktoś z Was ma możliwość, pomożecie młodej osobie z wieloma chorobami przewlekłymi?
Znam ją z Instagrama, jest cudowna, ale ma wyjątkowo ciężko ❤️‍🩹
Napisała dzisiaj, że nie chce być statystyką. Nie będziesz, Ash. Nie będziesz szła sama, nigdy.

https://www.siepomaga.pl/natalka-mancewicz?utm_source=ig&utm_medium=social&utm_content=link_in_bio&fbclid=PAb21jcARxedZleHRuA2FlbQIxMQBzcnRjBmFwcF9pZA81NjcwNjczNDMzNTI0MjcAAafbIyJMeExLjbCtdHr8naL5qQwZde0Xa4PVSPZN7fPLrcWJTZjvLjSfLo2xIg_aem_5NyNKtwwDOsT8M7KU0qbww

#mutualaid #autisticsolidarity #pomocwzajemna #opór #spoonie #chorobaprzewlekla #niepełnosprawność #pots #eds #pandas #mecfs #zapaleniemózgu #stomia #pozajelitowe #spektrumautyzmu #solidarność #zrzutka #zbiórka #ozn

Hej, jeśli ktoś z Was ma możliwość, pomożecie młodej osobie z wieloma chorobami przewlekłymi?
Znam ją z Instagrama, jest cudowna, ale ma wyjątkowo ciężko ❤️‍🩹
Napisała dzisiaj, że nie chce być statystyką. Nie będziesz, Ash. Nie będziesz szła sama, nigdy.

https://www.siepomaga.pl/natalka-mancewicz?utm_source=ig&utm_medium=social&utm_content=link_in_bio&fbclid=PAb21jcARxedZleHRuA2FlbQIxMQBzcnRjBmFwcF9pZA81NjcwNjczNDMzNTI0MjcAAafbIyJMeExLjbCtdHr8naL5qQwZde0Xa4PVSPZN7fPLrcWJTZjvLjSfLo2xIg_aem_5NyNKtwwDOsT8M7KU0qbww

#mutualaid #autisticsolidarity #pomocwzajemna #opór #spoonie #chorobaprzewlekla #niepełnosprawność #pots #eds #pandas #mecfs #zapaleniemózgu #stomia #pozajelitowe #spektrumautyzmu #solidarność #zrzutka #zbiórka #ozn

Парсинг тарифов интернета и ТВ: Анализируем тарифы провайдеров с инструментами Pandas, Seaborn, Matplotlib

Даже на относительно небольшой выборке данных анализ получился весьма интересным. Я провел исследование тарифных сеток двух крупных провайдеров России - Ростелекому и Дом.ру, в шести городах-миллионниках: Москва, Санкт-Петербург, Екатеринбург, Казань, Новосибирск, Красноярск. И вот какой результат получил, расскажу и покажу всё на графиках:

https://habr.com/ru/articles/1025008/

#аналитика #pandas #seaborn #провайдеры_интернет #провайдеры_связи #цены_на_тарифы #сравни #сравнительный_обзор #сравнительный_анализ

В эмиграции Цветаеву окружала серость и сырость. Установлено NLTK анализом с помощью Python

На примере стихотворения "Рассвет на рельсах" можно увидеть эмоции и настроения марины Цветаевой после отъезда в эмиграцию. В нём преобладают серые унылые тона. Но в то же время есть вера восстановить Россию.

https://habr.com/ru/articles/997036/

#python #nltk #nltk_python #nlpмодели #pandas #seaborn #matplotlib #чтение #поэзия #поэзия_серебряного_века

С помощью Python реабилитировал алкогольную романтику у Довлатова

Я проанализировал эпизоды с упоминанием алкоголя в полном корпусе произведений Довлатова и посмотрел, как и для чего он использует алкоголь в рассказах.

https://habr.com/ru/articles/985126/

#python #slovnet #razdel #natasha #pandas #pymorphy #seaborn #counter

So, when is geopolars coming?
maybe I want geonarwhals?

:blob_clown: #pandas #polars #narwhals #python #geopandas #geodata

New to data science? Dive into 5 hands‑on projects that walk you through practical Pandas EDA—cleaning missing values, spotting outliers, visualizing with seaborn, and more. Perfect for absolute beginners who want solid, reproducible workflows. Start exploring your own data today! #DataScience #Pandas #EDA #seaborn

🔗 https://aidailypost.com/news/5-data-science-projects-practical-pandas-eda-absolute-beginners

Making Math Less Stressful with a Python Super-Calculator https://hackaday.com/2025/10/26/making-math-less-stressful-with-a-python-super-calculator/ #WolframMathematica #datavisualization #SoftwareHacks #matplotlib #Seaborn #Pandas #python #maple #numpy #scipy #SymPy

Making Math Less Stressful with a Python Super-Calculator - In a recent write-up, [David Delony] explains how he built a Wolfram Mathematica-l... - https://hackaday.com/2025/10/26/making-math-less-stressful-with-a-python-super-calculator/ #wolframmathematica #datavisualization #softwarehacks #matplotlib #seaborn #pandas #python #maple #numpy #scipy #sympy

🐊 How do skills acquired through DataCamp translate to a better understanding of biodiversity data?

For Ibrahim Abdel Basir Gomaa, who completed courses in #Pandas, #Seaborn, machine learning, #SQL and statistics, these skills enabled him to confidently query large biodiversity datasets from sources like GBIF. 🔎

“One of my proudest achievements was developing a predictive model to assess species distribution patterns based on environmental factors...."