Six Star Source Classroom: Python Language Summit 2022: Python without GIL!

The topic of python GIL has been discussed countless times by the community, and recently at the 2022 language summit Sam Gross gave attendees an update on nogil (the project name is interesting, literal translation: no gil, really direct), The project took the Python community by storm when it was first announced in October 2021.

The GIL or "Global Interpreter Lock" is a key feature of Python that prevents true concurrency between threads, i.e. it is difficult to execute multiple tasks simultaneously when only one Python process is running. The main leader in previous GIL removal was Larry Hastings, with his famous "Gilectomy" project. The Gilectomy project was eventually abandoned because it made single-threaded Python code slower.

At the meeting, Sam Gross introduced the progress of the nogil project, and 3.9 has been selected as the target, because reaching the early adoption level is very important to judge whether the whole project is feasible. If third-party packages don't work when using nogil, early adopters won't be able to use the project effectively. Third-party packages still have broader support for Python 3.9 than Python 3.10, so Python 3.9 still makes more sense as a base branch than 3.10 or the master branch for now.

Another update introduced by Sam Gross is that he made changes in thread safety. In order for Python to work efficiently without the GIL, a lot of code needs to add new locks to ensure that it is still thread-safe. However, adding new locks to existing code can be very difficult, as there can be significant slowdowns in some areas. Gross' solution was to invent a new type of lock, a "more auspicious" lock.

Sam Gross also brought a proposal this time to introduce a new compiler flag in Python 3.12 that would disable the GIL, Gross' proposal was excited and strongly questioned by the gathered core developers, Carol Willing questioned Wouldn't it make more sense for nogil to continue as a separate branch of CPython, rather than having Gross aim to merge his work into the main branch of CPython itself. However, Gross responded that it was "not the path to success."

Samuel Colvin, maintainer of the pydantic library, expressed disappointment that the new proposal is for compiler flags rather than runtime flags. Colvin commented: "If the settings could be changed from within Python, I can't help but think the level of adoption would be greatly improved".

Both Barry Warsaw and Itamar Ostricher asked Gross about the impact nogil might have on third-party libraries if they wanted to support the new mode. Gross responded that the impact on many libraries is minimal - no impact at all on a library like scikit-learn, where numpy is probably only 15 lines of code. Gross said there has been considerable interest in Gross from scientific libraries, so he believes the pressure to build a separate C extension to support nogil mode will not be too overwhelming. Carol Willing encouraged Sam Gross to attend scientific computing conferences to gather more feedback.

Attendees were also very concerned about the impact that the introduction of nogil might have on CPython development. Some people worry that the introduction of nogil mode may mean that the number of tests run in CI has to be doubled. Others worry about the increased maintenance burden if two different versions of CPython are supported at the same time: one with the GIL and one without.

In general, the people present were still very excited and curious about the Nogier model. However, significant questions about the next steps of the project remain unresolved.

The above is all the content shared this time. If you want to know more Python information, please continue to pay attention to Six Star Source Classroom!