Has anyone tried to write something in Python with disabled GIL already? Which difference did you see?

I would imagine threading is somewhat more interesting.

Nah, not really interested in that. It requires to be compiled with a custom compile flag right? That's no issue actually, python has barely dependencies and compiling is even doable on my decade old laptop.

I have no idea if removing the GIL automatically means that the python threading became REAL threads. I do not think so tbh. I see some issues there.

If you decide to try it, please share results.

@retoor I think they said they would need like a year or more to test issues with libraries and turning off the GIL.

Probably stuff relies on that behavior.

all this low-level-stuff felt too hardcore for me back then. Now I feel like giving it a go. Building a proper DI framework is itching on my fingertips right now.

Honestly who gives a fuck about performance and Python. The two things are complete opposites.

@AlgoRythm the whole of machine learning community probably

@Hazarth it’s not the python that does that stuff. You hand it off to something performant. Python is just the orchestrator

@AlgoRythm

"Python is just the orchestrator"

I think Guido prefers: "AI Pimp"

@AlgoRythm Fair point, however, not all ML stuff is pandas, numpy and pytorch. Datasciene still includes stuff like custom python filters that are not vectorized, in which case maybe there's performance to be seen if done via threading while avoiding the overhead of manually managing shared memory and multiprocessing.

Though I didn't play with GIL off yet, so that's just speculation on my part, but I've seen plenty of code written by data science teams and sometimes they try to use threading so there's that :D

@Hazarth honestly my experience with Python’s performance has made me avoid the language at nearly all costs. I used to be a Python Stan in college. I know that R competes with Python in the data science field. I’ve never tried it, but hopefully it’s better

@Hazarth it really isn't that hard to make a module in Python. But maybe I am overestimating skills of Data Scientists.

@Demolishun They are not the best coders, but that's to be expected imo, their specialty is numbers, not hardware. Most basic threading will work auto-magically in python, but when dealing with huge datasets with tens of millions of rows you have to use better tools like memory sharing and mmaping to actually get any speed benefits from threads and processes. And I honestly don't expect them to pull those off, and that's fine. That's usually my job then to optimize their code and architecture, it's good teamwork, though some teams are harder to work with than others for sure :D