The idea is that the extra subpackage only has requirements specific to that extra.
The logic however only excluded requirements without markers,
but requirements with *a* marker that was correct leaked to all extras subpackages.
E.g. with the following requirements:
Requires-Dist: base-dependency
Requires-Dist: base-dependency-with-matching-marker ; python_version < "3.15"
Requires-Dist: base-dependency-with-unmatching-marker ; python_version < "3.8"
Provides-Extra: an-extra
Requires-Dist: extra-only-dependency-with-matching-marker ; extra == 'an-extra' and python_version < "3.15"
Requires-Dist: extra-only-dependency-with-unmatching-marker ; extra == 'an-extra' and python_version < "3.8"
On Python 3.10, the base package generated the following requirements:
python3.10dist(base-dependency)
python3.10dist(base-dependency-with-matching-marker)
And for the [an-extra] extra:
python3.10dist(base-dependency-with-matching-marker) <--- REDUNDANT, WRONG
python3.10dist(extra-only-dependency-with-matching-marker)
Now we no longer just check if the marker evaluates to True,
but we also check that the same marker evaluates to False when the extra is not given.
A real package with this issue is build[virtualenv] 0.8.0, which we use for tests. The package has:
Requires-Dist: tomli (>=1.0.0) ; python_version < "3.11"
And on Python 3.10, it generated the following dependency for python3-build+virtualenv-0.8.0-2.fc37.noarch.rpm:
python3.10dist(tomli) >= 1
Now it no longer does. This is asserted in tests.
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2090186
Upstream PR: https://github.com/rpm-software-management/python-rpm-packaging/pull/16
Improve handling of > operator, preventing post-release from satisfying most rpm requirements.
Improve handling of < operator, preventing pre-release from satisfying rpm requirement.
Improve handling of != operator with prefix matching, preventing pre-release from satisfying rpm requirements.
There were three problems:
- sys.version was not imported
- sys.version[:3] is not reliable on Python 3.10+
- distutils is deprecated on Python 3.10+
We were not hit by the missing import in Fedora because we only run the script
on .dist-info/.egg-info/.egg and not on .py files, so this if-branch never runs.
But when the script was fed with a .py path, it errored:
Traceback (most recent call last):
File "/usr/lib/rpm/pythondistdeps.py", line 344, in <module>
purelib = get_python_lib(standard_lib=0, plat_specific=0).split(version[:3])[0]
NameError: name 'version' is not defined
The sys.version[:3] thing kinda works for Python 3.10+ because *in this
particular case* splitting on '3.1' and taking the prefix yields the same
results as splitting on '3.10', but I consider that mere coincidence.
Finally, since the distutils import happened at module-level,
we got the Deprecation warning in all Fedora's Python packages:
/usr/lib/rpm/pythondistdeps.py:16: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12
Backported from https://github.com/rpm-software-management/python-rpm-packaging/commit/d12e039037
self.name in PathDistribution is a property in Python 3.10+ and thus we
can't redefine it as an instance variable. Instead we explicitly define
it as a property, which works on all supported Python versions.
Upstream change to importlib.metadata: https://github.com/rpm-software-management/rpm/pull/1317
Due to extras packages being hadled slightly differently by importlib,
one test case for this was added. And due to changes in handling
requires.txt files, comments were removed from the pyreq2rpm.tests
testing package.
Also because of the switch, we removed the dependency on setuptools and
added a dependency on packaging.
Note: Some packages with egg-info files might provide a different name
due to this change if there is a conflict between the filename and the
name in the metadata. Previously, the filename was sometimes used to
parse the name, now it is always the content of that file, which is what
packaging does, and thus also pip and other Python tooling. Currently,
this is known to affect only 1 package in Fedora (ntpsec).
The resulting script is different from upstream because of not yet upstreamed changes in Fedora:
- scripts/pythondistdeps: Rework error messages
- scripts/pythondistdeps: Add parameter --package-name
- scripts/pythondistdeps: Implement provides/requires for extras packages
- pythondistdeps.py: When parsing extras name, take the rightmost +
These changes are proposed in this upstream PR: https://github.com/rpm-software-management/rpm/pull/1546
See https://bugzilla.redhat.com/show_bug.cgi?id=1853597#c11
pkg_resources from setuptools 42+ no longer only use platform.python_version(),
but also platform.python_version_tuple() -- this was updated in packaging 19.1+.
This fix makes it work again with both new and old setuptools,
hopefully for some while.
bf069fe9dd86a443f318
Note that the code is completely unchanged except for the indentation
under the new if __name__ == "__main__":
Note that this change is necessary, but not sufficient to use the
RpmVersion class.
The init of the RpmVersion class will fail when called from an outside
script, because the `parse_version()` function is lazily imported from
the code outside the class. However, adding the import of
parse_version() to RpmVersion class is not done right now, because while
we would import it from `pkg_resources`, other scripts might want to
rely instead of the lightweight `packaging` module for the import. Thus
I'm leaving this conondrum to be addressed in the future.
--normalized-names-format FORMAT
FORMAT of normalized names can be `pep503` [default] or `legacy-dots` (dots allowed)
--normalized-names-provide-both
Provede both `pep503` and `legacy-dots` format of normalized names (useful for a transition period)
Notes from an attempted rewrite from pkg_resources to importlib.metadata in 2020:
1. While pkg_resources can open a metadata on a specified path
(Distribution.from_location()), importlib provides access only to
"installed package metadata", i.e. the the dist-info or egg-info directory
must be "discoverable", i.e. on the sys.path.
- Thankfully only the dist/egg-info directory must exist, the
corresponding Python module does not have to be present.
- The problems this causes:
(a) You have to manipulate the sys.path to add the specific location of
the site-packages directory inside the buildroot
(b) If you have package "foo" in this newly added directory on sys.path
and there is some problem and its dist/egg-info metadata are not found,
importlib.metadata continues searching the sys.path and may discover a
package with the same name (possibly same version) outside the
buildroot.
To get around this, you can manipulate the sys.path to remove all
other "site-packages" directories. But you have to leave the
standard library there, because importlib may import other modules
(in my testing: base64, quopri, random, socket, calendar, uu)
(c) I have not tested how well it works if you're ispecting metadata of
different Python versions than the one you run the script with
(especially Python 2 vs Python 3). This might also cause problems with
dependency specifiers (i.e. python_version != "3.4")
2. Handling of dependencies (requires) is problematic in importlib.metadata
- pkg_resources provides a way to separately list standard requires and a
requires for each "extras" category. importlib does not provide this, it
only spits out a list of strings, each string in the format:
- 'packaging>=14',
- 'towncrier>=18.5.0; extra == "docs"', or
- 'psutil<6,>=5.6.1; (python_version != "3.4") and extra == "testing"
you can either parse these with a regex (fragile) or use the external
`packaging` Python module. `packaging`, however, also doesn't have a great
support for figuring out extra dependencies, it provides the marker api:
- <Marker(\'python_version != "3.4" and extra == "testing"\')>
you can use Marker api to evaluate the condition, but not to parse.
For parsing you can access the private api Marker._markers:
- marker._markers=[[(<Variable('python_version')>, <Op('!=')>, \
<Value('3.4')>)], 'and', (<Variable('extra')>, <Op('==')>, \
<Value('testing')>)]
which beyond the problem of being private is also not very useful for
parsing due to its structure.
- pkg_resources also provides version parsing, which importlib does not
and `packaging` needs to be used
- importlib is part of the standard library, but packaging and its
2 runtime dependencies (pyparsing and six) are not, and therefore we
would go from 1 dependency to 3
3. A few minor issues, more in the next section about equivalents.
importlib.metadata.distribution equivalents of pkg_resources.Distribution attributes:
- pkg_resources: dist.py_version
importlib: # not implemented (but can be guessed from the /usr/lib/pythonXX.YY/ path)
- pkg_resources: dist.project_name
importlib: dist.metadata['name']
- pkg_resources: dist.key
importlib: # not implemented
- pkg_resources: dist.version
importlib: dist.version
- pkg_resources: dist.requires()
importlib: dist.requires # but returns strings with almost no parsing done, and also lists extras
- pkg_resources: dist.requires(extras=dist.extras)
importlib: # not implemented, has to be parsed from dist.requires
- pkg_resources: dist.get_entry_map('console_scripts')
importlib: [ep for ep in importlib.metadata.entry_points()['console_scripts'] if ep.name == pkg][0]
# I have not found a better way to get the console_scripts
- pkg_resources: dist.get_entry_map('gui_scripts')
importlib: # Presumably same as console_scripts, but untested
That is, we add new provides that replace dots with a dash.
Package that used to provide python3dist(zope.component) and python3.8dist(zope.component)
now also provides python3dist(zope-component) and python3.8dist(zope-component).
Package that used to provide python3dist(a.-.-.-.a) now provides python3dist(a-a) as well.
This is consistent with pip behavior, `pip install zope-component` installs zope.component.
Historically, we have always used dist.key (safe_name) from setuptools,
but that is a non-standardized convention -- whether or not it replaces dots
with dashes is not even documented.
We say we use "canonical name" or "normalized name" everywhere, yet we didn't.
We really need to follow the standard (PEP 503):
https://www.python.org/dev/peps/pep-0503/#normalized-names
The proper function here would be packaging.utils.canonicalize_name
https://packaging.pypa.io/en/latest/utils/#packaging.utils.canonicalize_name
-- we reimplement it here to avoid an external dependency.
This is the first required step needed if we want to change our requirements later.
If we decide we don't, for whatever reason, this doesn't break anything.
Puts bounded requirements into parenthesis
Fixes: https://github.com/rpm-software-management/rpm/issues/995
Upstream: https://github.com/rpm-software-management/rpm/pull/996
For this input: pyparsing>=2.0.1,!=2.0.4,!=2.1.2,!=2.1.6
Instead of (invalid):
(python3.8dist(pyparsing) >= 2.0.1 with
python3.8dist(pyparsing) < 2.1.2 or python3.8dist(pyparsing) >= 2.1.2.0 with
python3.8dist(pyparsing) < 2.1.6 or python3.8dist(pyparsing) >= 2.1.6.0 with
python3.8dist(pyparsing) < 2.0.4 or python3.8dist(pyparsing) >= 2.0.4.0)
Produces (valid):
(python3.8dist(pyparsing) >= 2.0.1 with
(python3.8dist(pyparsing) < 2.1.2 or python3.8dist(pyparsing) >= 2.1.2.0) with
(python3.8dist(pyparsing) < 2.0.4 or python3.8dist(pyparsing) >= 2.0.4.0) with
(python3.8dist(pyparsing) < 2.1.6 or python3.8dist(pyparsing) >= 2.1.6.0))
For this input: babel>=1.3,!=2.0
Instead of (invalid):
(python3.8dist(babel) >= 1.3 with
python3.8dist(babel) < 2 or python3.8dist(babel) >= 2.0)
Produces (valid):
(python3.8dist(babel) >= 1.3 with
(python3.8dist(babel) < 2 or python3.8dist(babel) >= 2.0))
For this input: pbr!=2.1.0,>=2.0.0
Instead of (invalid):
(python3.8dist(pbr) >= 2 with
python3.8dist(pbr) < 2.1 or python3.8dist(pbr) >= 2.1.0)
Produces (valid):
(python3.8dist(pbr) >= 2 with
(python3.8dist(pbr) < 2.1 or python3.8dist(pbr) >= 2.1.0))
The purelib and platlib were both defined to /usr/lib64/python on
64bits systems. This is because:
>>> get_python_lib(standard_lib=1, plat_specific=0)
'/usr/lib64/python3.7'
>>> get_python_lib(standard_lib=1, plat_specific=1)
'/usr/lib64/python3.7'
>>> get_python_lib(standard_lib=0, plat_specific=0)
'/usr/lib/python3.7/site-packages'
>>> get_python_lib(standard_lib=0, plat_specific=1)
'/usr/lib64/python3.7/site-packages'
So now we use standard_lib=0 to get the site-packages base path
from /usr/lib and not /usr/lib64.
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1609492
This package is not being kept up to date, it's hard to maintain and we
will need to tune it from time to time which is painful.
Also removes whole layer of bootstrapping.
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
Miro Hrončok
Sandro Mani
Gordon Messmer
Tomas Orsava
Igor Raits