nightly: include shortname in fedmsgs, clean up version vars

The main point here is to include the compose 'shortname' in
compose fedmsgs, as #484 does for post-release compose types.
While doing this I noticed there's rather a lot of mess and
cruft in the various version-related variables here (and a few
others).

`GIT_BRANCH` variable is never used anywhere any more, so it's
removed.

`BRANCH` was never used anywhere, so it is gone. (It appears
to have been added in a25b08c7 but never used). There seemed
to be no useful distinction between `DIST` and `BRANCHED`, and
both actually seem to be used mainly to specify a release
version. The exception is that `BRANCHED` was set to
"Modular-Bikeshed" in nightly-modular.sh , but there is no
justification for this that I can find, and its only practical
use was as the value of the `branch` key in fedmsgs. So far as
I can figure out, the original purpose of the `branch` key was
for when we sent out `compose.branched.(foo)` fedmsgs - it
was e.g. 'f18' when Branched was Fedora 18. As we now don't
send out `compose.branched` messages any more, but include the
release number in the message topic for Branched messages
(e.g. `compose.27`), this purpose no longer applies. So the
key just doesn't seem to have any particular use at all any
more, and I don't think changing its value will have any real
consequences.

Thus, `BRANCH`, `BRANCHED` and `DIST` are replaced by `RELEASE`
and `RELEASE_TITLE`, which specify the release version. Their
values will be identical for numerical release versions, but
for Rawhide and Bikeshed we need to know both the lower-case
and title-case versions of the name; RELEASE_TITLE is the
title-case version.

We use `$SHORT-$RELEASE_TITLE` to replace several instances
of hardcoding in the scripts (e.g. Fedora-Modular-Bikeshed).

The `ATOMIC`, `ATOMICREPO` and `ATOMICDEST` variables that were
blindly copied from nightly.sh to nightly-modular.sh, but never
actually used in it, are also removed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
This commit is contained in:
Adam Williamson 2017-12-04 17:24:43 -08:00
parent e0fb303122
commit ded7d70929
2 changed files with 70 additions and 75 deletions

View File

@ -11,18 +11,14 @@ NIGHTLY="--nightly"
SKIP_PHASES="--skip-phase=productimg"
DEST=$(pwd)
DATE=$(date "+%Y%m%d")
DIST="bikeshed"
BRANCHED="Modular-Bikeshed"
BRANCH="rawhide"
GIT_BRANCH="master"
TMPDIR=`mktemp -d /tmp/$DIST.$DATE.XXXX`
SHORT="Fedora-Modular"
RELEASE="bikeshed"
RELEASE_TITLE="Bikeshed"
TMPDIR=`mktemp -d /tmp/$RELEASE.$DATE.XXXX`
TOMAIL="devel@lists.fedoraproject.org test@lists.fedoraproject.org"
FROM="Fedora Rawhide Report <rawhide@fedoraproject.org>"
RSYNCPREFIX="sudo -u ftpsync"
ATOMIC=$(mktemp -d /tmp/atomic.${DIST}.$DATE.XXXX)
ATOMICREPO="/mnt/koji/compose/atomic/$BRANCHED/"
ATOMICDEST="/mnt/koji/atomic/$BRANCHED/"
OLDCOMPOSE_ID=$(cat $TARGET_DIR/latest-Fedora-Modular-Bikeshed/COMPOSE_ID)
OLDCOMPOSE_ID=$(cat $TARGET_DIR/latest-$SHORT-$RELEASE_TITLE/COMPOSE_ID)
# assume a releng dir is a git checkout of the releng repo
# if it does not exist clone it
@ -40,8 +36,8 @@ FEDMSG_CERTPREFIX="releng"
. ./releng/scripts/fedmsg-functions.sh
# Announce that we are starting, even though we don't know the compose_id yet..
fedmsg_json_start=$(printf '{"log": "start", "branch": "%s", "arch": "%s"}' "$BRANCHED" "$ARCH")
send_fedmsg "${fedmsg_json_start}" ${DIST} start
fedmsg_json_start=$(printf '{"log": "start", "branch": "%s", "arch": "%s", "short": "%s"}' "$RELEASE" "$ARCH" "$SHORT")
send_fedmsg "${fedmsg_json_start}" ${RELEASE} start
CMD="pungi-koji --notification-script=/usr/bin/pungi-fedmsg-notification --config=$CONFIG --old-composes=$TARGET_DIR $OLD_COMPOSES_DIR $NIGHTLY $SKIP_PHASES"
@ -56,54 +52,54 @@ if [ "$?" != "0" ]; then
exit 1
fi
NEWCOMPOSE_ID=$(cat $TARGET_DIR/latest-Fedora-Modular-Bikeshed/COMPOSE_ID)
NEWCOMPOSE_ID=$(cat $TARGET_DIR/latest-$SHORT-$RELEASE_TITLE/COMPOSE_ID)
SHORTCOMPOSE_ID=$(echo $NEWCOMPOSE_ID|sed -e 's|Fedora-Modular.*-||g')
fedmsg_json_start=$(printf '{"log": "start", "branch": "%s", "arch": "%s", "compose_id": "%s"}' "$BRANCHED" "$ARCH" "$NEWCOMPOSE_ID")
fedmsg_json_done=$(printf '{"log": "done", "branch": "%s", "arch": "%s", "compose_id": "%s"}' "$BRANCHED" "$ARCH" "$NEWCOMPOSE_ID")
fedmsg_json_start=$(printf '{"log": "start", "branch": "%s", "arch": "%s", "short": "%s", "compose_id": "%s"}' "$RELEASE" "$ARCH" "$SHORT" "$NEWCOMPOSE_ID")
fedmsg_json_done=$(printf '{"log": "done", "branch": "%s", "arch": "%s", "short": "%s", "compose_id": "%s"}' "$RELEASE" "$ARCH" "$SHORT" "$NEWCOMPOSE_ID")
# Set this to use later for a few items include depcheck
DESTDIR=$TARGET_DIR/$NEWCOMPOSE_ID
compose-changelog -p $TARGET_DIR/$NEWCOMPOSE_ID/logs/ $TARGET_DIR/$OLDCOMPOSE_ID/ $TARGET_DIR/$NEWCOMPOSE_ID/
# Figure out a version for broken deps e-mail that goes to package maintainers.
# In Rawhide it's just rawhide, for branched versions we prepend F- to the number.
if [ "$BRANCHED" = "Modular-Bikeshed" ]; then
TREENAME="$BRANCHED"
# For Bikeshed it's just bikeshed, for branched versions we prepend F- to the number.
if [ "$RELEASE" = "bikeshed" ]; then
TREENAME="$RELEASE"
else
TREENAME="F-$BRANCHED"
TREENAME="F-$RELEASE"
fi
/usr/share/mash/spam-o-matic --nomail --treename="$TREENAME" "$DESTDIR/compose/Server/" >"$DESTDIR/logs/depcheck"
# Tell interested persons that the rsync is starting (zomg!)
send_fedmsg "${fedmsg_json_start}" ${DIST} rsync.start
send_fedmsg "${fedmsg_json_start}" ${RELEASE} rsync.start
# Sync the content to /pub/fedora
if [ ! -d /pub/fedora/linux/modular/development/$DIST ]; then
mkdir /pub/fedora/linux/modular/development/$DIST
if [ ! -d /pub/fedora/linux/modular/development/$RELEASE ]; then
mkdir /pub/fedora/linux/modular/development/$RELEASE
fi
$RSYNCPREFIX compose-partial-copy --arch=armhfp --arch=x86_64 --arch=aarch64 --arch=ppc64 --arch=ppc64le --arch=s390x --arch src \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora/linux/modular/development/$DIST/ \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora/linux/modular/development/$RELEASE/ \
--variant Server \
--exclude=repodata
$RSYNCPREFIX compose-partial-copy --arch=armhfp --arch=x86_64 --arch=aarch64 --arch=ppc64 --arch=ppc64le --arch=s390x --arch src \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora/linux/modular/development/$DIST/ \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora/linux/modular/development/$RELEASE/ \
--variant Server \
--delete-after
$RSYNCPREFIX rm /pub/fedora/linux/modular/development/$DIST/.composeinfo
$RSYNCPREFIX ./releng/scripts/build_composeinfo /pub/fedora/linux/modular/development/$DIST/ --name "$NEWCOMPOSE_ID"
$RSYNCPREFIX rm /pub/fedora/linux/modular/development/$RELEASE/.composeinfo
$RSYNCPREFIX ./releng/scripts/build_composeinfo /pub/fedora/linux/modular/development/$RELEASE/ --name "$NEWCOMPOSE_ID"
# Tell interested persons that the rsync is done.
send_fedmsg "${fedmsg_json_done}" ${DIST} rsync.complete
send_fedmsg "${fedmsg_json_done}" ${RELEASE} rsync.complete
# Tell everyone by fedmsg about the compose
send_fedmsg "${fedmsg_json_done}" ${DIST} complete
send_fedmsg "${fedmsg_json_done}" ${RELEASE} complete
# Tell everyone by email about the compose
SUBJECT='Fedora Modular '$DIST' compose report: '$SHORTCOMPOSE_ID' changes'
SUBJECT='Fedora Modular '$RELEASE' compose report: '$SHORTCOMPOSE_ID' changes'
for tomail in $TOMAIL ; do
cat $TARGET_DIR/$NEWCOMPOSE_ID/logs/*verbose | \
mutt -e "set from=\"$FROM\"" -e 'set envelope_from=yes' -s "$SUBJECT" $tomail
done
# Remove old composes older than 14 days
find $TARGET_DIR/Fedora-Modular-Bikeshed* -xdev -depth -maxdepth 2 -mtime +14 -exec rm -rf {} \;
find $TARGET_DIR/$SHORT-$RELEASE_TITLE* -xdev -depth -maxdepth 2 -mtime +14 -exec rm -rf {} \;

View File

@ -10,20 +10,19 @@ NIGHTLY="--nightly"
SKIP_PHASES="--skip-phase=productimg"
DEST=$(pwd)
DATE=$(date "+%Y%m%d")
DIST="rawhide"
BRANCHED="rawhide"
BRANCH="rawhide"
SHORT="Fedora"
RELEASE="rawhide"
RELEASE_TITLE="Rawhide"
COMPSFILE="comps-rawhide.xml"
GIT_BRANCH="master"
TMPDIR=`mktemp -d /tmp/$DIST.$DATE.XXXX`
TMPDIR=`mktemp -d /tmp/$RELEASE.$DATE.XXXX`
TOMAIL="devel@lists.fedoraproject.org test@lists.fedoraproject.org"
FROM="Fedora Rawhide Report <rawhide@fedoraproject.org>"
RSYNCPREFIX="sudo -u ftpsync"
ATOMIC=$(mktemp -d /tmp/atomic.${DIST}.$DATE.XXXX)
ATOMICHOSTREPO="/mnt/koji/compose/atomic/$BRANCHED/"
ATOMICWSREPO="/mnt/koji/compose/ostree/$BRANCHED/"
ATOMICHOSTDEST="/mnt/koji/atomic/$BRANCHED/"
OLDCOMPOSE_ID=$(cat $TARGET_DIR/latest-Fedora-Rawhide/COMPOSE_ID)
ATOMIC=$(mktemp -d /tmp/atomic.${RELEASE}.$DATE.XXXX)
ATOMICHOSTREPO="/mnt/koji/compose/atomic/$RELEASE/"
ATOMICWSREPO="/mnt/koji/compose/ostree/$RELEASE/"
ATOMICHOSTDEST="/mnt/koji/atomic/$RELEASE/"
OLDCOMPOSE_ID=$(cat $TARGET_DIR/latest-$SHORT-$RELEASE_TITLE/COMPOSE_ID)
# uncomment and edit for resuming a failed compose
#COMPOSE_ID="Fedora-23-20150530.n.0"
@ -43,8 +42,8 @@ FEDMSG_CERTPREFIX="releng"
. ./releng/scripts/fedmsg-functions.sh
# Announce that we are starting, even though we don't know the compose_id yet..
fedmsg_json_start=$(printf '{"log": "start", "branch": "%s", "arch": "%s"}' "$BRANCHED" "$ARCH")
send_fedmsg "${fedmsg_json_start}" ${DIST} start
fedmsg_json_start=$(printf '{"log": "start", "branch": "%s", "arch": "%s", "short": "%s"}' "$RELEASE" "$ARCH" "$SHORT")
send_fedmsg "${fedmsg_json_start}" ${RELEASE} start
#pushd $TMPDIR
#git clone https://pagure.io/fedora-comps.git && {
@ -72,10 +71,10 @@ if [ "$?" != "0" ]; then
exit 1
fi
NEWCOMPOSE_ID=$(cat $TARGET_DIR/latest-Fedora-Rawhide/COMPOSE_ID)
NEWCOMPOSE_ID=$(cat $TARGET_DIR/latest-$SHORT-$RELEASE_TITLE/COMPOSE_ID)
SHORTCOMPOSE_ID=$(echo $NEWCOMPOSE_ID|sed -e 's|Fedora-.*-||g')
fedmsg_json_start=$(printf '{"log": "start", "branch": "%s", "arch": "%s", "compose_id": "%s"}' "$BRANCHED" "$ARCH" "$NEWCOMPOSE_ID")
fedmsg_json_done=$(printf '{"log": "done", "branch": "%s", "arch": "%s", "compose_id": "%s"}' "$BRANCHED" "$ARCH" "$NEWCOMPOSE_ID")
fedmsg_json_start=$(printf '{"log": "start", "branch": "%s", "arch": "%s", "short": "%s", "compose_id": "%s"}' "$RELEASE" "$ARCH" "$SHORT" "$NEWCOMPOSE_ID")
fedmsg_json_done=$(printf '{"log": "done", "branch": "%s", "arch": "%s", "short": "%s", "compose_id": "%s"}' "$RELEASE" "$ARCH" "$SHORT" "$NEWCOMPOSE_ID")
# Set this to use later for a few items include depcheck
DESTDIR=$TARGET_DIR/$NEWCOMPOSE_ID
@ -83,10 +82,10 @@ DESTDIR=$TARGET_DIR/$NEWCOMPOSE_ID
compose-changelog -p $TARGET_DIR/$NEWCOMPOSE_ID/logs/ $TARGET_DIR/$OLDCOMPOSE_ID/ $TARGET_DIR/$NEWCOMPOSE_ID/
# Figure out a version for broken deps e-mail that goes to package maintainers.
# In Rawhide it's just rawhide, for branched versions we prepend F- to the number.
if [ "$BRANCHED" = "rawhide" ]; then
TREENAME="$BRANCHED"
if [ "$RELEASE" = "rawhide" ]; then
TREENAME="$RELEASE"
else
TREENAME="F-$BRANCHED"
TREENAME="F-$RELEASE"
fi
/usr/share/mash/spam-o-matic --treename="$TREENAME" "$DESTDIR/compose/Everything/" >"$DESTDIR/logs/depcheck"
@ -99,61 +98,61 @@ ostree prune --repo=$ATOMICHOSTREPO --keep-younger-than="14 days ago" --refs-onl
ostree prune --repo=$ATOMICWSREPO --keep-younger-than="14 days ago" --refs-only
# Tell interested persons that the rsync is starting (zomg!)
send_fedmsg "${fedmsg_json_start}" ${DIST} rsync.start
send_fedmsg "${fedmsg_json_start}" ${RELEASE} rsync.start
# Sync the content to /pub/fedora
if [ ! -d /pub/fedora/linux/development/$DIST ]; then
mkdir /pub/fedora/linux/development/$DIST
if [ ! -d /pub/fedora/linux/development/$RELEASE ]; then
mkdir /pub/fedora/linux/development/$RELEASE
fi
$RSYNCPREFIX compose-partial-copy --arch=armhfp --arch=x86_64 --arch src \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora/linux/development/$DIST/ \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora/linux/development/$RELEASE/ \
--variant Everything --variant Atomic --variant CloudImages --variant Docker --variant Server --variant Spins --variant Workstation \
--link-dest=/pub/fedora/linux/development/$DIST/Everything --exclude=repodata
--link-dest=/pub/fedora/linux/development/$RELEASE/Everything --exclude=repodata
$RSYNCPREFIX compose-partial-copy --arch=armhfp --arch=x86_64 --arch src \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora/linux/development/$DIST/ \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora/linux/development/$RELEASE/ \
--variant Everything --variant Atomic --variant CloudImages --variant Docker --variant Server --variant Spins --variant Workstation \
--link-dest=/pub/fedora/linux/development/$DIST/Everything --delete-after
$RSYNCPREFIX rm /pub/fedora/linux/development/$DIST/.composeinfo
$RSYNCPREFIX ./releng/scripts/build_composeinfo /pub/fedora/linux/development/$DIST/ --name "$NEWCOMPOSE_ID"
--link-dest=/pub/fedora/linux/development/$RELEASE/Everything --delete-after
$RSYNCPREFIX rm /pub/fedora/linux/development/$RELEASE/.composeinfo
$RSYNCPREFIX ./releng/scripts/build_composeinfo /pub/fedora/linux/development/$RELEASE/ --name "$NEWCOMPOSE_ID"
# Sync the content to /pub/alt
if [ ! -d /pub/alt/development/$DIST ]; then
mkdir /pub/alt/development/$DIST
if [ ! -d /pub/alt/development/$RELEASE ]; then
mkdir /pub/alt/development/$RELEASE
fi
$RSYNCPREFIX compose-partial-copy --arch=armhfp --arch=x86_64 \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/alt/development/$DIST/ \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/alt/development/$RELEASE/ \
--variant Cloud --variant Labs \
--link-dest=/pub/fedora/linux/development/$DIST/Everything/ --exclude=repodata
--link-dest=/pub/fedora/linux/development/$RELEASE/Everything/ --exclude=repodata
$RSYNCPREFIX compose-partial-copy --arch=armhfp --arch=x86_64 \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/alt/development/$DIST/ \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/alt/development/$RELEASE/ \
--variant Cloud --variant Labs \
--link-dest=/pub/fedora/linux/development/$DIST/Everything/ --delete-after
$RSYNCPREFIX rm /pub/alt/development/$DIST/.composeinfo
$RSYNCPREFIX ./releng/scripts/build_composeinfo /pub/alt/development/$DIST/ --name "$NEWCOMPOSE_ID"
--link-dest=/pub/fedora/linux/development/$RELEASE/Everything/ --delete-after
$RSYNCPREFIX rm /pub/alt/development/$RELEASE/.composeinfo
$RSYNCPREFIX ./releng/scripts/build_composeinfo /pub/alt/development/$RELEASE/ --name "$NEWCOMPOSE_ID"
# Sync the content to /pub/fedora-secondary
if [ ! -d /pub/fedora-secondary/development/$DIST ]; then
mkdir /pub/fedora-secondary/development/$DIST
if [ ! -d /pub/fedora-secondary/development/$RELEASE ]; then
mkdir /pub/fedora-secondary/development/$RELEASE
fi
$RSYNCPREFIX compose-partial-copy --arch=aarch64 --arch=i386 --arch=ppc64 --arch=ppc64le --arch=s390x \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora-secondary/development/$DIST/ \
--link-dest=/pub/fedora/linux/development/$DIST/Everything/ --exclude=repodata
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora-secondary/development/$RELEASE/ \
--link-dest=/pub/fedora/linux/development/$RELEASE/Everything/ --exclude=repodata
$RSYNCPREFIX compose-partial-copy --arch=aarch64 --arch=i386 --arch=ppc64 --arch=ppc64le --arch=s390x \
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora-secondary/development/$DIST/ \
--link-dest=/pub/fedora/linux/development/$DIST/Everything/ --delete-after
$RSYNCPREFIX rm /pub/fedora-secondary/development/$DIST/.composeinfo
$RSYNCPREFIX ./releng/scripts/build_composeinfo /pub/fedora-secondary/development/$DIST/ --name "$NEWCOMPOSE_ID"
"$TARGET_DIR/$NEWCOMPOSE_ID" /pub/fedora-secondary/development/$RELEASE/ \
--link-dest=/pub/fedora/linux/development/$RELEASE/Everything/ --delete-after
$RSYNCPREFIX rm /pub/fedora-secondary/development/$RELEASE/.composeinfo
$RSYNCPREFIX ./releng/scripts/build_composeinfo /pub/fedora-secondary/development/$RELEASE/ --name "$NEWCOMPOSE_ID"
$RSYNCPREFIX rsync -avhH --delete-after $ATOMICHOSTREPO $ATOMICHOSTDEST
# Tell interested persons that the rsync is done.
send_fedmsg "${fedmsg_json_done}" ${DIST} rsync.complete
send_fedmsg "${fedmsg_json_done}" ${RELEASE} rsync.complete
# Tell everyone by fedmsg about the compose
send_fedmsg "${fedmsg_json_done}" ${DIST} complete
send_fedmsg "${fedmsg_json_done}" ${RELEASE} complete
# Tell everyone by email about the compose
SUBJECT='Fedora '$DIST' compose report: '$SHORTCOMPOSE_ID' changes'
SUBJECT='Fedora '$RELEASE' compose report: '$SHORTCOMPOSE_ID' changes'
for tomail in $TOMAIL ; do
cat $TARGET_DIR/$NEWCOMPOSE_ID/logs/*verbose $DESTDIR/logs/depcheck | \
mutt -e "set from=\"$FROM\"" -e 'set envelope_from=yes' -s "$SUBJECT" $tomail