Commit Graph

54 Commits

Author SHA1 Message Date
Miro Hrončok
c3f90ed2e8 Fix reversed grep exit codes in integration tests
grep -v only fails if there are no unmatched lines, but that's not what we want to test.
2020-05-04 13:45:57 +02:00
Miro Hrončok
6beec97e9e Add integartion test for the dist generator 2020-05-04 13:45:57 +02:00
Tomas Orsava
54e4aa751b Bump version, enable new features, add test suite to Fedora CI 2020-05-04 13:45:57 +02:00
Tomas Orsava
79790d12af scripts/pythondistdeps: Modify handling of dev versions 2020-04-30 22:24:44 +02:00
Tomas Orsava
972beac29a scripts/pythondistdeps: Version handling exception with better information 2020-04-30 22:24:44 +02:00
Tomas Orsava
d48f3500d8 scripts/pythondistdeps: Do anything only when called as a main script
Note that the code is completely unchanged except for the indentation
under the new if __name__ == "__main__":

Note that this change is necessary, but not sufficient to use the
RpmVersion class.
The init of the RpmVersion class will fail when called from an outside
script, because the `parse_version()` function is lazily imported from
the code outside the class.  However, adding the import of
parse_version() to RpmVersion class is not done right now, because while
we would import it from `pkg_resources`, other scripts might want to
rely instead of the lightweight `packaging` module for the import. Thus
I'm leaving this conondrum to be addressed in the future.
2020-04-30 22:24:44 +02:00
Tomas Orsava
1523def34e scripts/pythondistdeps: Implement --normalized-name-* options
--normalized-names-format FORMAT
    FORMAT of normalized names can be `pep503` [default] or `legacy-dots` (dots allowed)

--normalized-names-provide-both
    Provede both `pep503` and `legacy-dots` format of normalized names (useful for a transition period)
2020-04-30 22:24:44 +02:00
Tomas Orsava
e33d4e94c8 scripts/pythondistdeps: Add option to generate major-version provides only for specified Python versions 2020-04-30 22:24:44 +02:00
Tomas Orsava
89e1676cee scripts/pythondistdeps: Add tests
The test data download themselves using pip if not present
2020-04-30 22:24:44 +02:00
Tomas Orsava
1634914c2e scripts/pythondistdeps: Notes from an attempted rewrite to importlib.metadata
Notes from an attempted rewrite from pkg_resources to importlib.metadata in 2020:
1. While pkg_resources can open a metadata on a specified path
   (Distribution.from_location()), importlib provides access only to
   "installed package metadata", i.e. the the dist-info or egg-info directory
   must be "discoverable", i.e. on the sys.path.
   - Thankfully only the dist/egg-info directory must exist, the
     corresponding Python module does not have to be present.
   - The problems this causes:
     (a) You have to manipulate the sys.path to add the specific location of
         the site-packages directory inside the buildroot
     (b) If you have package "foo" in this newly added directory on sys.path
         and there is some problem and its dist/egg-info metadata are not found,
         importlib.metadata continues searching the sys.path and may discover a
         package with the same name (possibly same version) outside the
         buildroot.
         To get around this, you can manipulate the sys.path to remove all
         other "site-packages" directories. But you have to leave the
         standard library there, because importlib may import other modules
         (in my testing: base64, quopri, random, socket, calendar, uu)
     (c) I have not tested how well it works if you're ispecting metadata of
         different Python versions than the one you run the script with
         (especially Python 2 vs Python 3). This might also cause problems with
         dependency specifiers (i.e. python_version != "3.4")
2. Handling of dependencies (requires) is problematic in importlib.metadata
   - pkg_resources provides a way to separately list standard requires and a
     requires for each "extras" category. importlib does not provide this, it
     only spits out a list of strings, each string in the format:
     - 'packaging>=14',
     - 'towncrier>=18.5.0; extra == "docs"', or
     - 'psutil<6,>=5.6.1; (python_version != "3.4") and extra == "testing"
     you can either parse these with a regex (fragile) or use the external
     `packaging` Python module. `packaging`, however, also doesn't have a great
     support for figuring out extra dependencies, it provides the marker api:
     - <Marker(\'python_version != "3.4" and extra == "testing"\')>
     you can use Marker api to evaluate the condition, but not to parse.
     For parsing you can access the private api Marker._markers:
     - marker._markers=[[(<Variable('python_version')>, <Op('!=')>, \
           <Value('3.4')>)], 'and', (<Variable('extra')>, <Op('==')>, \
           <Value('testing')>)]
     which beyond the problem of being private is also not very useful for
     parsing due to its structure.
   - pkg_resources also provides version parsing, which importlib does not
     and `packaging` needs to be used
   - importlib is part of the standard library, but packaging and its
     2 runtime dependencies (pyparsing and six) are not, and therefore we
     would go from 1 dependency to 3
3. A few minor issues, more in the next section about equivalents.

importlib.metadata.distribution equivalents of pkg_resources.Distribution attributes:
- pkg_resources: dist.py_version
  importlib: # not implemented (but can be guessed from the /usr/lib/pythonXX.YY/ path)
- pkg_resources: dist.project_name
  importlib: dist.metadata['name']
- pkg_resources: dist.key
  importlib: # not implemented
- pkg_resources: dist.version
  importlib: dist.version
- pkg_resources: dist.requires()
  importlib: dist.requires  # but returns strings with almost no parsing done, and also lists extras
- pkg_resources: dist.requires(extras=dist.extras)
  importlib: # not implemented, has to be parsed from dist.requires
- pkg_resources: dist.get_entry_map('console_scripts')
  importlib: [ep for ep in importlib.metadata.entry_points()['console_scripts'] if ep.name == pkg][0]
             # I have not found a better way to get the console_scripts
- pkg_resources: dist.get_entry_map('gui_scripts')
  importlib: # Presumably same as console_scripts, but untested
2020-04-30 22:24:44 +02:00
Tomas Orsava
1639424a51 Sync with upstream RPM dist generator 2020-04-30 22:24:42 +02:00
Miro Hrončok
c8249102ec Don't define global Lua variables from Python generator 2020-04-28 14:48:06 +02:00
Gordon Messmer
0ec8581037 Handle all-zero versions without crashing
From https://github.com/rpm-software-management/rpm/pull/1184
2020-04-20 13:55:53 +02:00
Igor Raits
783dcc7147 Sync with upstream RPM dist generator 2020-04-10 12:31:30 +02:00
Miro Hrončok
8eef42cbaa Use dynamic %_prefix value when matching files for python(abi) provides
See https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject.org/thread/UFKUM5UKCTNGIT3KJVYEI5VXPI23QMBN/

Flatpak builds redefine %_prefix and the dependencies should remain present.

Also get rid of one useless ^ and prep the pattern for two digit Python major versions.

Add a test that tests that we match our default %_prefix (was the case even before this commit).
2020-04-07 16:25:11 +02:00
Miro Hrončok
bbfe4930d9 Automatically call %python_provide
This allows us to drop the %python_provide macro from most spec files,
except where we want to use it for virtual provides or empty packages.
2020-04-03 16:06:46 +02:00
Miro Hrončok
eae8dd0f57 Add CI tests for python(abi) provides 2020-04-03 14:35:16 +02:00
Miro Hrončok
486ca7e540 Drop tabs from python.attr 2020-04-01 15:57:00 +02:00
Miro Hrončok
ff7b9b1ae0 Reimplement pythondeps.sh as parametric macro generators
pythondeps.sh was written in shell and unlike the Python dist generators,
it uses no Python, it plainly determines the provide / requires from the path.
As the script was run for every Python file, we were potentially doing hundreds
of shelling outs to execute a script that calls grep and sed.

In Lua, this is much more efficient.

Some timings:
    https://github.com/rpm-software-management/rpm/pull/1153#issuecomment-607146356

Parametric macro generators require RPM 4.16+:
    https://fedoraproject.org/wiki/Changes/RPM-4.16

Fixes https://github.com/rpm-software-management/rpm/issues/1152
Upstream PR: https://github.com/rpm-software-management/rpm/pull/1153

Since this is intended for Fedora 33+ only, clean some old cruft.
2020-04-01 15:53:15 +02:00
Fedora Release Engineering
caccd3e498 - Rebuilt for https://fedoraproject.org/wiki/Fedora_32_Mass_Rebuild
Signed-off-by: Fedora Release Engineering <releng@fedoraproject.org>
2020-01-30 15:05:48 +00:00
Miro Hrončok
7d819e0000 Also provide pythonXdist() with PEP 503 normalized names (#1791530)
That is, we add new provides that replace dots with a dash.

Package that used to provide python3dist(zope.component) and python3.8dist(zope.component)
now also provides python3dist(zope-component) and python3.8dist(zope-component).

Package that used to provide python3dist(a.-.-.-.a) now provides python3dist(a-a) as well.

This is consistent with pip behavior, `pip install zope-component` installs zope.component.

Historically, we have always used dist.key (safe_name) from setuptools,
but that is a non-standardized convention -- whether or not it replaces dots
with dashes is not even documented.
We say we use "canonical name" or "normalized name" everywhere, yet we didn't.

We really need to follow the standard (PEP 503):

https://www.python.org/dev/peps/pep-0503/#normalized-names

The proper function here would be packaging.utils.canonicalize_name
https://packaging.pypa.io/en/latest/utils/#packaging.utils.canonicalize_name
-- we reimplement it here to avoid an external dependency.

This is the first required step needed if we want to change our requirements later.
If we decide we don't, for whatever reason, this doesn't break anything.
2020-01-17 17:27:46 +01:00
Miro Hrončok
724a52a5f2 Fix more complicated requirement expressions by adding parenthesis
Puts bounded requirements into parenthesis

Fixes: https://github.com/rpm-software-management/rpm/issues/995
Upstream: https://github.com/rpm-software-management/rpm/pull/996

For this input: pyparsing>=2.0.1,!=2.0.4,!=2.1.2,!=2.1.6

Instead of (invalid):
(python3.8dist(pyparsing) >= 2.0.1 with
 python3.8dist(pyparsing) < 2.1.2 or python3.8dist(pyparsing) >= 2.1.2.0 with
 python3.8dist(pyparsing) < 2.1.6 or python3.8dist(pyparsing) >= 2.1.6.0 with
 python3.8dist(pyparsing) < 2.0.4 or python3.8dist(pyparsing) >= 2.0.4.0)

Produces (valid):
(python3.8dist(pyparsing) >= 2.0.1 with
 (python3.8dist(pyparsing) < 2.1.2 or python3.8dist(pyparsing) >= 2.1.2.0) with
 (python3.8dist(pyparsing) < 2.0.4 or python3.8dist(pyparsing) >= 2.0.4.0) with
 (python3.8dist(pyparsing) < 2.1.6 or python3.8dist(pyparsing) >= 2.1.6.0))

For this input: babel>=1.3,!=2.0

Instead of (invalid):
(python3.8dist(babel) >= 1.3 with
 python3.8dist(babel) < 2 or python3.8dist(babel) >= 2.0)

Produces (valid):
(python3.8dist(babel) >= 1.3 with
 (python3.8dist(babel) < 2 or python3.8dist(babel) >= 2.0))

For this input: pbr!=2.1.0,>=2.0.0

Instead of (invalid):
(python3.8dist(pbr) >= 2 with
 python3.8dist(pbr) < 2.1 or python3.8dist(pbr) >= 2.1.0)

Produces (valid):
(python3.8dist(pbr) >= 2 with
 (python3.8dist(pbr) < 2.1 or python3.8dist(pbr) >= 2.1.0))
2020-01-03 11:00:19 +01:00
Miro Hrončok
ca811dbf35 Sync with upstream RPM
- Handle version ending with ".*"
 - Handle compatible-release operator "~="
 - Use rich deps for semantically versioned dependencies
 - Match Python version if minor has multiple digits (e.g. 3.10)
 - Only add setuptools requirement for egg-info packages

https://github.com/rpm-software-management/rpm/pull/951
https://github.com/rpm-software-management/rpm/pull/973
https://github.com/rpm-software-management/rpm/pull/982

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1758141
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1777382
2020-01-01 23:21:46 +01:00
Fedora Release Engineering
b9fe0e7182 - Rebuilt for https://fedoraproject.org/wiki/Fedora_31_Mass_Rebuild
Signed-off-by: Fedora Release Engineering <releng@fedoraproject.org>
2019-07-26 16:06:05 +00:00
Miro Hrončok
ff085a044d Canonicalize Python versions and properly handle != spec
Fixes https://github.com/rpm-software-management/rpm/issues/639

From upstream PR:  https://github.com/rpm-software-management/rpm/pull/757
2019-06-24 14:44:19 +02:00
Miro Hrončok
70b3ebc993 console_scripts entry points to require setuptools
https://github.com/rpm-software-management/rpm/pull/666
2019-04-17 14:35:46 +02:00
Fedora Release Engineering
67cc59dd22 - Rebuilt for https://fedoraproject.org/wiki/Fedora_30_Mass_Rebuild
Signed-off-by: Fedora Release Engineering <releng@fedoraproject.org>
2019-02-02 09:09:35 +00:00
Igor Gnatenko
e745e149a6
Enable requires generator
References: https://fedoraproject.org/wiki/Changes/EnablingPythonGeneratorsByDefault
References: https://pagure.io/fesco/issue/2026
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-12-20 14:21:14 +01:00
Igor Gnatenko
9f6f709036
Tighten regex for depgen
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-10-03 14:21:26 +02:00
Miro Hrončok
1879d8a0e2 Use nonstandardlib for purelib definition (#1609492)
The purelib and platlib were both defined to /usr/lib64/python on
64bits systems. This is because:

    >>> get_python_lib(standard_lib=1, plat_specific=0)
    '/usr/lib64/python3.7'

    >>> get_python_lib(standard_lib=1, plat_specific=1)
    '/usr/lib64/python3.7'

    >>> get_python_lib(standard_lib=0, plat_specific=0)
    '/usr/lib/python3.7/site-packages'

    >>> get_python_lib(standard_lib=0, plat_specific=1)
    '/usr/lib64/python3.7/site-packages'

So now we use standard_lib=0 to get the site-packages base path
from /usr/lib and not /usr/lib64.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1609492
2018-07-28 22:36:22 +02:00
Igor Gnatenko
fdad4ede04
fix wrong regex in dist attr
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-07-28 14:39:01 +02:00
Igor Gnatenko
40740c1747
fix the conflicting version
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-07-28 14:32:50 +02:00
Igor Gnatenko
a51b52a5e4
remove gitingore/sources
We don't use that anymore.

Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-07-28 14:31:43 +02:00
Igor Gnatenko
1d1b5f8e22
reference new attr file in %files
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-07-28 14:30:57 +02:00
Igor Gnatenko
dc64d7b436
Split python to pythondist generator
Running this python script on all possible files is way too expensive.
Some of the packages timeout due to that.

Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-07-28 14:26:48 +02:00
Fedora Release Engineering
0fb362073a - Rebuilt for https://fedoraproject.org/wiki/Fedora_29_Mass_Rebuild
Signed-off-by: Fedora Release Engineering <releng@fedoraproject.org>
2018-07-14 01:58:03 +00:00
Igor Gnatenko
5aa670bb39
"Fix" support of environment markers
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-02-11 00:50:57 +01:00
Igor Gnatenko
b3ca5d8622
pythondistdeps.py: change shebang to python3
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-02-11 00:50:56 +01:00
Igor Gnatenko
6d7c3b291d
attr: match everything in python directories
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-02-11 00:50:56 +01:00
Igor Gnatenko
257dcc8865
attr: print provides in modern format
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-02-11 00:50:55 +01:00
Igor Gnatenko
ad70cabb97
Fork upstream generators
This package is not being kept up to date, it's hard to maintain and we
will need to tune it from time to time which is painful.

Also removes whole layer of bootstrapping.

Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2018-02-11 00:50:54 +01:00
Fedora Release Engineering
f3aa226019 - Rebuilt for https://fedoraproject.org/wiki/Fedora_28_Mass_Rebuild
Signed-off-by: Fedora Release Engineering <releng@fedoraproject.org>
2018-02-09 10:39:37 +00:00
Tomas Orsava
17d4b252d8 Bump the release, add a changelog entry 2017-11-28 15:19:41 +01:00
Tomas Orsava
d2ec6ae95b Switch bootsrapping macro to a bcond for modularity 2017-11-28 11:29:21 +01:00
Tomas Orsava
eee8ac4de7 Rebase to rpm 4.14.0 final 2017-10-20 11:54:21 +02:00
Tomas Orsava
9af0545c30 Always remember to upload new-sources, kids! 2017-09-18 16:59:41 +02:00
Tomas Orsava
97d7b2f546 Update to a new upstream version of RPM
- Drop upstreamed patches
- Renumber remaining patches
2017-09-18 16:54:24 +02:00
Miro Hrončok
d64bb3efc4 Do not provide pythonXdist for platform-python packages 2017-08-24 13:50:23 +02:00
Iryna Shcherbina
ea8d0d41e8 Generate requires and provides for platform-python(abi) 2017-08-14 15:41:30 +02:00
Fedora Release Engineering
16e420392f - Rebuilt for https://fedoraproject.org/wiki/Fedora_27_Mass_Rebuild 2017-07-27 11:44:11 +00:00