rpms
/
qemu
2
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Update to qemu 4.2.0 rc1 

Signed-off-by: Cole Robinson <crobinso@redhat.com>
This commit is contained in:
Cole Robinson 2019-11-12 15:47:45 -05:00
parent 8e6758e973
commit 0038f84388
8 changed files with 22 additions and 925 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -35,3 +35,4 @@
/qemu-4.1.0-rc1.tar.xz
/qemu-4.1.0-rc2.tar.xz
/qemu-4.1.0.tar.xz
/qemu-4.2.0-rc1.tar.xz

151
0001-file-posix-Handle-undetectable-alignment.patch
View File

@ -1,151 +0,0 @@
From: Nir Soffer <nirsof@gmail.com>
Date: Tue, 13 Aug 2019 21:21:03 +0300
Subject: [PATCH] file-posix: Handle undetectable alignment
In some cases buf_align or request_alignment cannot be detected:
1. With Gluster, buf_align cannot be detected since the actual I/O is
done on Gluster server, and qemu buffer alignment does not matter.
Since we don't have alignment requirement, buf_align=1 is the best
value.
2. With local XFS filesystem, buf_align cannot be detected if reading
from unallocated area. In this we must align the buffer, but we don't
know what is the correct size. Using the wrong alignment results in
I/O error.
3. With Gluster backed by XFS, request_alignment cannot be detected if
reading from unallocated area. In this case we need to use the
correct alignment, and failing to do so results in I/O errors.
4. With NFS, the server does not use direct I/O, so both buf_align cannot
be detected. In this case we don't need any alignment so we can use
buf_align=1 and request_alignment=1.
These cases seems to work when storage sector size is 512 bytes, because
the current code starts checking align=512. If the check succeeds
because alignment cannot be detected we use 512. But this does not work
for storage with 4k sector size.
To determine if we can detect the alignment, we probe first with
align=1. If probing succeeds, maybe there are no alignment requirement
(cases 1, 4) or we are probing unallocated area (cases 2, 3). Since we
don't have any way to tell, we treat this as undetectable alignment. If
probing with align=1 fails with EINVAL, but probing with one of the
expected alignments succeeds, we know that we found a working alignment.
Practically the alignment requirements are the same for buffer
alignment, buffer length, and offset in file. So in case we cannot
detect buf_align, we can use request alignment. If we cannot detect
request alignment, we can fallback to a safe value. To use this logic,
we probe first request alignment instead of buf_align.
Here is a table showing the behaviour with current code (the value in
parenthesis is the optimal value).
Case Sector buf_align (opt) request_alignment (opt) result
======================================================================
1 512 512 (1) 512 (512) OK
1 4096 512 (1) 4096 (4096) FAIL
----------------------------------------------------------------------
2 512 512 (512) 512 (512) OK
2 4096 512 (4096) 4096 (4096) FAIL
----------------------------------------------------------------------
3 512 512 (1) 512 (512) OK
3 4096 512 (1) 512 (4096) FAIL
----------------------------------------------------------------------
4 512 512 (1) 512 (1) OK
4 4096 512 (1) 512 (1) OK
Same cases with this change:
Case Sector buf_align (opt) request_alignment (opt) result
======================================================================
1 512 512 (1) 512 (512) OK
1 4096 4096 (1) 4096 (4096) OK
----------------------------------------------------------------------
2 512 512 (512) 512 (512) OK
2 4096 4096 (4096) 4096 (4096) OK
----------------------------------------------------------------------
3 512 4096 (1) 4096 (512) OK
3 4096 4096 (1) 4096 (4096) OK
----------------------------------------------------------------------
4 512 4096 (1) 4096 (1) OK
4 4096 4096 (1) 4096 (1) OK
I tested that provisioning VMs and copying disks on local XFS and
Gluster with 4k bytes sector size work now, resolving bugs [1],[2].
I tested also on XFS, NFS, Gluster with 512 bytes sector size.
[1] https://bugzilla.redhat.com/1737256
[2] https://bugzilla.redhat.com/1738657
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit a6b257a08e3d72219f03e461a52152672fec0612)
---
block/file-posix.c | 36 +++++++++++++++++++++++++-----------
1 file changed, 25 insertions(+), 11 deletions(-)
diff --git a/block/file-posix.c b/block/file-posix.c
index 4479cc7ab4..b8b4dad553 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -323,6 +323,7 @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
BDRVRawState *s = bs->opaque;
char *buf;
size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
+ size_t alignments[] = {1, 512, 1024, 2048, 4096};
/* For SCSI generic devices the alignment is not really used.
With buffered I/O, we don't have any restrictions. */
@@ -349,25 +350,38 @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
}
#endif
- /* If we could not get the sizes so far, we can only guess them */
- if (!s->buf_align) {
+ /*
+ * If we could not get the sizes so far, we can only guess them. First try
+ * to detect request alignment, since it is more likely to succeed. Then
+ * try to detect buf_align, which cannot be detected in some cases (e.g.
+ * Gluster). If buf_align cannot be detected, we fallback to the value of
+ * request_alignment.
+ */
+
+ if (!bs->bl.request_alignment) {
+ int i;
size_t align;
- buf = qemu_memalign(max_align, 2 * max_align);
- for (align = 512; align <= max_align; align <<= 1) {
- if (raw_is_io_aligned(fd, buf + align, max_align)) {
- s->buf_align = align;
+ buf = qemu_memalign(max_align, max_align);
+ for (i = 0; i < ARRAY_SIZE(alignments); i++) {
+ align = alignments[i];
+ if (raw_is_io_aligned(fd, buf, align)) {
+ /* Fallback to safe value. */
+ bs->bl.request_alignment = (align != 1) ? align : max_align;
break;
}
}
qemu_vfree(buf);
}
- if (!bs->bl.request_alignment) {
+ if (!s->buf_align) {
+ int i;
size_t align;
- buf = qemu_memalign(s->buf_align, max_align);
- for (align = 512; align <= max_align; align <<= 1) {
- if (raw_is_io_aligned(fd, buf, align)) {
- bs->bl.request_alignment = align;
+ buf = qemu_memalign(max_align, 2 * max_align);
+ for (i = 0; i < ARRAY_SIZE(alignments); i++) {
+ align = alignments[i];
+ if (raw_is_io_aligned(fd, buf + align, max_align)) {
+ /* Fallback to request_aligment. */
+ s->buf_align = (align != 1) ? align : bs->bl.request_alignment;
break;
}
}

369
0002-block-posix-Always-allocate-the-first-block.patch
View File

@ -1,369 +0,0 @@
From: Nir Soffer <nirsof@gmail.com>
Date: Tue, 27 Aug 2019 04:05:27 +0300
Subject: [PATCH] block: posix: Always allocate the first block
When creating an image with preallocation "off" or "falloc", the first
block of the image is typically not allocated. When using Gluster
storage backed by XFS filesystem, reading this block using direct I/O
succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the optimal
value (512), which may lead to unneeded data copying when aligning
requests. Allocating the first block avoids the fallback.
Since we allocate the first block even with preallocation=off, we no
longer create images with zero disk size:
$ ./qemu-img create -f raw test.raw 1g
Formatting 'test.raw', fmt=raw size=1073741824
$ ls -lhs test.raw
4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw
And converting the image requires additional cluster:
$ ./qemu-img measure -f raw -O qcow2 test.raw
required size: 458752
fully allocated size: 1074135040
When using format like vmdk with multiple files per image, we allocate
one block per file:
$ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat
$ ls -lhs test*.vmdk
4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
4.0K -rw-r--r--. 1 nsoffer nsoffer 353 Aug 27 03:23 test.vmdk
I did quick performance test for copying disks with qemu-img convert to
new raw target image to Gluster storage with sector size of 512 bytes:
for i in $(seq 10); do
rm -f dst.raw
sleep 10
time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
done
Here is a table comparing the total time spent:
Type Before(s) After(s) Diff(%)
---------------------------------------
real 530.028 469.123 -11.4
user 17.204 10.768 -37.4
sys 17.881 7.011 -60.7
We can see very clear improvement in CPU usage.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190827010528.8818-2-nsoffer@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 3a20013fbb26d2a1bd11ef148eefdb1508783787)
---
block/file-posix.c | 51 ++++++++++++++++++++++++++++++++
tests/qemu-iotests/059.out | 2 +-
tests/qemu-iotests/150.out | 11 -------
tests/qemu-iotests/150.out.qcow2 | 11 +++++++
tests/qemu-iotests/150.out.raw | 12 ++++++++
tests/qemu-iotests/175 | 19 ++++++++----
tests/qemu-iotests/175.out | 8 ++---
tests/qemu-iotests/178.out.qcow2 | 4 +--
tests/qemu-iotests/221.out | 12 +++++---
tests/qemu-iotests/253.out | 12 +++++---
10 files changed, 110 insertions(+), 32 deletions(-)
delete mode 100644 tests/qemu-iotests/150.out
create mode 100644 tests/qemu-iotests/150.out.qcow2
create mode 100644 tests/qemu-iotests/150.out.raw
diff --git a/block/file-posix.c b/block/file-posix.c
index b8b4dad553..8ea98896ce 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -1749,6 +1749,43 @@ static int handle_aiocb_discard(void *opaque)
return ret;
}
+/*
+ * Help alignment probing by allocating the first block.
+ *
+ * When reading with direct I/O from unallocated area on Gluster backed by XFS,
+ * reading succeeds regardless of request length. In this case we fallback to
+ * safe alignment which is not optimal. Allocating the first block avoids this
+ * fallback.
+ *
+ * fd may be opened with O_DIRECT, but we don't know the buffer alignment or
+ * request alignment, so we use safe values.
+ *
+ * Returns: 0 on success, -errno on failure. Since this is an optimization,
+ * caller may ignore failures.
+ */
+static int allocate_first_block(int fd, size_t max_size)
+{
+ size_t write_size = (max_size < MAX_BLOCKSIZE)
+ ? BDRV_SECTOR_SIZE
+ : MAX_BLOCKSIZE;
+ size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
+ void *buf;
+ ssize_t n;
+ int ret;
+
+ buf = qemu_memalign(max_align, write_size);
+ memset(buf, 0, write_size);
+
+ do {
+ n = pwrite(fd, buf, write_size, 0);
+ } while (n == -1 && errno == EINTR);
+
+ ret = (n == -1) ? -errno : 0;
+
+ qemu_vfree(buf);
+ return ret;
+}
+
static int handle_aiocb_truncate(void *opaque)
{
RawPosixAIOData *aiocb = opaque;
@@ -1788,6 +1825,17 @@ static int handle_aiocb_truncate(void *opaque)
/* posix_fallocate() doesn't set errno. */
error_setg_errno(errp, -result,
"Could not preallocate new data");
+ } else if (current_length == 0) {
+ /*
+ * posix_fallocate() uses fallocate() if the filesystem
+ * supports it, or fallback to manually writing zeroes. If
+ * fallocate() was used, unaligned reads from the fallocated
+ * area in raw_probe_alignment() will succeed, hence we need to
+ * allocate the first block.
+ *
+ * Optimize future alignment probing; ignore failures.
+ */
+ allocate_first_block(fd, offset);
}
} else {
result = 0;
@@ -1849,6 +1897,9 @@ static int handle_aiocb_truncate(void *opaque)
if (ftruncate(fd, offset) != 0) {
result = -errno;
error_setg_errno(errp, -result, "Could not resize file");
+ } else if (current_length == 0 && offset > current_length) {
+ /* Optimize future alignment probing; ignore failures. */
+ allocate_first_block(fd, offset);
}
return result;
default:
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index 4fab42a28c..fe3f861f3c 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -27,7 +27,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMax
image: TEST_DIR/t.vmdk
file format: vmdk
virtual size: 0.977 TiB (1073741824000 bytes)
-disk size: 16 KiB
+disk size: 1.97 MiB
Format specific information:
cid: XXXXXXXX
parent cid: XXXXXXXX
diff --git a/tests/qemu-iotests/150.out b/tests/qemu-iotests/150.out
deleted file mode 100644
index 2a54e8dcfa..0000000000
--- a/tests/qemu-iotests/150.out
+++ /dev/null
@@ -1,11 +0,0 @@
-QA output created by 150
-
-=== Mapping sparse conversion ===
-
-Offset Length File
-
-=== Mapping non-sparse conversion ===
-
-Offset Length File
-0 0x100000 TEST_DIR/t.IMGFMT
-*** done
diff --git a/tests/qemu-iotests/150.out.qcow2 b/tests/qemu-iotests/150.out.qcow2
new file mode 100644
index 0000000000..2a54e8dcfa
--- /dev/null
+++ b/tests/qemu-iotests/150.out.qcow2
@@ -0,0 +1,11 @@
+QA output created by 150
+
+=== Mapping sparse conversion ===
+
+Offset Length File
+
+=== Mapping non-sparse conversion ===
+
+Offset Length File
+0 0x100000 TEST_DIR/t.IMGFMT
+*** done
diff --git a/tests/qemu-iotests/150.out.raw b/tests/qemu-iotests/150.out.raw
new file mode 100644
index 0000000000..3cdc7727a5
--- /dev/null
+++ b/tests/qemu-iotests/150.out.raw
@@ -0,0 +1,12 @@
+QA output created by 150
+
+=== Mapping sparse conversion ===
+
+Offset Length File
+0 0x1000 TEST_DIR/t.IMGFMT
+
+=== Mapping non-sparse conversion ===
+
+Offset Length File
+0 0x100000 TEST_DIR/t.IMGFMT
+*** done
diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index 51e62c8276..7ba28b3c1b 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -37,14 +37,16 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
# the file size. This function hides the resulting difference in the
# stat -c '%b' output.
# Parameter 1: Number of blocks an empty file occupies
-# Parameter 2: Image size in bytes
+# Parameter 2: Minimal number of blocks in an image
+# Parameter 3: Image size in bytes
_filter_blocks()
{
extra_blocks=$1
- img_size=$2
+ min_blocks=$2
+ img_size=$3
- sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \
- -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/"
+ sed -e "s/blocks=$min_blocks\\(\$\\|[^0-9]\\)/min allocation/" \
+ -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/"
}
# get standard environment, filters and checks
@@ -60,16 +62,21 @@ size=$((1 * 1024 * 1024))
touch "$TEST_DIR/empty"
extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
+# We always write the first byte; check how many blocks this filesystem
+# allocates to match empty image alloation.
+printf "\0" > "$TEST_DIR/empty"
+min_blocks=$(stat -c '%b' "$TEST_DIR/empty")
+
echo
echo "== creating image with default preallocation =="
_make_test_img $size | _filter_imgfmt
-stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
for mode in off full falloc; do
echo
echo "== creating image with preallocation $mode =="
IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
- stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+ stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
done
# success, all done
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index 6d9a5ed84e..263e521262 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -2,17 +2,17 @@ QA output created by 175
== creating image with default preallocation ==
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
-size=1048576, nothing allocated
+size=1048576, min allocation
== creating image with preallocation off ==
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
-size=1048576, nothing allocated
+size=1048576, min allocation
== creating image with preallocation full ==
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
-size=1048576, everything allocated
+size=1048576, max allocation
== creating image with preallocation falloc ==
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
-size=1048576, everything allocated
+size=1048576, max allocation
*** done
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
index 55a8dc926f..9e7d8c44df 100644
--- a/tests/qemu-iotests/178.out.qcow2
+++ b/tests/qemu-iotests/178.out.qcow2
@@ -101,7 +101,7 @@ converted image file size in bytes: 196608
== raw input image with data (human) ==
Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
-required size: 393216
+required size: 458752
fully allocated size: 1074135040
wrote 512/512 bytes at offset 512
512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -257,7 +257,7 @@ converted image file size in bytes: 196608
Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
{
- "required": 393216,
+ "required": 458752,
"fully-allocated": 1074135040
}
wrote 512/512 bytes at offset 512
diff --git a/tests/qemu-iotests/221.out b/tests/qemu-iotests/221.out
index 9f9dd52bb0..dca024a0c3 100644
--- a/tests/qemu-iotests/221.out
+++ b/tests/qemu-iotests/221.out
@@ -3,14 +3,18 @@ QA output created by 221
=== Check mapping of unaligned raw image ===
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
wrote 1/1 bytes at offset 65536
1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
{ "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
{ "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
{ "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
{ "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
*** done
diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
index 607c0baa0b..3d08b305d7 100644
--- a/tests/qemu-iotests/253.out
+++ b/tests/qemu-iotests/253.out
@@ -3,12 +3,16 @@ QA output created by 253
=== Check mapping of unaligned raw image ===
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
wrote 65535/65535 bytes at offset 983040
63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
*** done

134
0003-tests-make-filemonitor-test-more-robust-to-event-ord.patch
View File

@ -1,134 +0,0 @@
From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= <berrange@redhat.com>
Date: Wed, 21 Aug 2019 16:14:27 +0100
Subject: [PATCH] tests: make filemonitor test more robust to event ordering
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The ordering of events that are emitted during the rmdir
test have changed with kernel >= 5.3. Semantically both
new & old orderings are correct, so we must be able to
cope with either.
To cope with this, when we see an unexpected event, we
push it back onto the queue and look and the subsequent
event to see if that matches instead.
Tested-by: Peter Xu <peterx@redhat.com>
Tested-by: Wei Yang <richardw.yang@linux.intel.com>
Tested-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
(cherry picked from commit bf9e0313c27d8e6ecd7f7de3d63e1cb25d8f6311)
---
tests/test-util-filemonitor.c | 43 +++++++++++++++++++++++++++--------
1 file changed, 34 insertions(+), 9 deletions(-)
diff --git a/tests/test-util-filemonitor.c b/tests/test-util-filemonitor.c
index 46e781c022..301cd2db61 100644
--- a/tests/test-util-filemonitor.c
+++ b/tests/test-util-filemonitor.c
@@ -45,6 +45,11 @@ typedef struct {
const char *filedst;
int64_t *watchid;
int eventid;
+ /*
+ * Only valid with OP_EVENT - this event might be
+ * swapped with the next OP_EVENT
+ */
+ bool swapnext;
} QFileMonitorTestOp;
typedef struct {
@@ -98,6 +103,10 @@ qemu_file_monitor_test_handler(int64_t id,
QFileMonitorTestData *data = opaque;
QFileMonitorTestRecord *rec = g_new0(QFileMonitorTestRecord, 1);
+ if (debug) {
+ g_printerr("Queue event id %" PRIx64 " event %d file %s\n",
+ id, event, filename);
+ }
rec->id = id;
rec->event = event;
rec->filename = g_strdup(filename);
@@ -125,7 +134,8 @@ qemu_file_monitor_test_record_free(QFileMonitorTestRecord *rec)
* to wait for the event to be queued for us.
*/
static QFileMonitorTestRecord *
-qemu_file_monitor_test_next_record(QFileMonitorTestData *data)
+qemu_file_monitor_test_next_record(QFileMonitorTestData *data,
+ QFileMonitorTestRecord *pushback)
{
GTimer *timer = g_timer_new();
QFileMonitorTestRecord *record = NULL;
@@ -139,9 +149,15 @@ qemu_file_monitor_test_next_record(QFileMonitorTestData *data)
}
if (data->records) {
record = data->records->data;
- tmp = data->records;
- data->records = g_list_remove_link(data->records, tmp);
- g_list_free(tmp);
+ if (pushback) {
+ data->records->data = pushback;
+ } else {
+ tmp = data->records;
+ data->records = g_list_remove_link(data->records, tmp);
+ g_list_free(tmp);
+ }
+ } else if (pushback) {
+ qemu_file_monitor_test_record_free(pushback);
}
qemu_mutex_unlock(&data->lock);
@@ -158,13 +174,15 @@ static bool
qemu_file_monitor_test_expect(QFileMonitorTestData *data,
int64_t id,
QFileMonitorEvent event,
- const char *filename)
+ const char *filename,
+ bool swapnext)
{
QFileMonitorTestRecord *rec;
bool ret = false;
- rec = qemu_file_monitor_test_next_record(data);
+ rec = qemu_file_monitor_test_next_record(data, NULL);
+ retry:
if (!rec) {
g_printerr("Missing event watch id %" PRIx64 " event %d file %s\n",
id, event, filename);
@@ -172,6 +190,11 @@ qemu_file_monitor_test_expect(QFileMonitorTestData *data,
}
if (id != rec->id) {
+ if (swapnext) {
+ rec = qemu_file_monitor_test_next_record(data, rec);
+ swapnext = false;
+ goto retry;
+ }
g_printerr("Expected watch id %" PRIx64 " but got %" PRIx64 "\n",
id, rec->id);
goto cleanup;
@@ -347,7 +370,8 @@ test_file_monitor_events(void)
.filesrc = "fish", },
{ .type = QFILE_MONITOR_TEST_OP_EVENT,
.filesrc = "", .watchid = &watch4,
- .eventid = QFILE_MONITOR_EVENT_IGNORED },
+ .eventid = QFILE_MONITOR_EVENT_IGNORED,
+ .swapnext = true },
{ .type = QFILE_MONITOR_TEST_OP_EVENT,
.filesrc = "fish", .watchid = &watch0,
.eventid = QFILE_MONITOR_EVENT_DELETED },
@@ -493,8 +517,9 @@ test_file_monitor_events(void)
g_printerr("Event id=%" PRIx64 " event=%d file=%s\n",
*op->watchid, op->eventid, op->filesrc);
}
- if (!qemu_file_monitor_test_expect(
- &data, *op->watchid, op->eventid, op->filesrc))
+ if (!qemu_file_monitor_test_expect(&data, *op->watchid,
+ op->eventid, op->filesrc,
+ op->swapnext))
goto cleanup;
break;
case QFILE_MONITOR_TEST_OP_CREATE:

224
0004-Revert-block-avoid-recursive-block_status-call-if-po.patch
View File

@ -1,224 +0,0 @@
From: Cole Robinson <crobinso@redhat.com>
Date: Thu, 24 Oct 2019 11:32:46 -0400
Subject: [PATCH] Revert "block: avoid recursive block_status call if possible"
This reverts commit 69f47505ee66afaa513305de0c1895a224e52c45.
Workaround for qcow2 triggered XFS corruption, bug 1763519
---
block/io.c | 9 +--------
block/qcow2-refcount.c | 32 --------------------------------
block/qcow2.c | 11 -----------
block/qcow2.h | 4 ----
include/block/block.h | 8 +-------
tests/qemu-iotests/102 | 2 +-
tests/qemu-iotests/102.out | 3 +--
tests/qemu-iotests/141.out | 2 +-
tests/qemu-iotests/144.out | 2 +-
9 files changed, 6 insertions(+), 67 deletions(-)
diff --git a/block/io.c b/block/io.c
index 06305c6ea6..ef057ba5db 100644
--- a/block/io.c
+++ b/block/io.c
@@ -2156,12 +2156,6 @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs,
*/
assert(*pnum && QEMU_IS_ALIGNED(*pnum, align) &&
align > offset - aligned_offset);
- if (ret & BDRV_BLOCK_RECURSE) {
- assert(ret & BDRV_BLOCK_DATA);
- assert(ret & BDRV_BLOCK_OFFSET_VALID);
- assert(!(ret & BDRV_BLOCK_ZERO));
- }
-
*pnum -= offset - aligned_offset;
if (*pnum > bytes) {
*pnum = bytes;
@@ -2192,8 +2186,7 @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs,
}
}
- if (want_zero && ret & BDRV_BLOCK_RECURSE &&
- local_file && local_file != bs &&
+ if (want_zero && local_file && local_file != bs &&
(ret & BDRV_BLOCK_DATA) && !(ret & BDRV_BLOCK_ZERO) &&
(ret & BDRV_BLOCK_OFFSET_VALID)) {
int64_t file_pnum;
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index ef965d7895..81ecb88f9c 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -3448,35 +3448,3 @@ int64_t qcow2_get_last_cluster(BlockDriverState *bs, int64_t size)
"There are no references in the refcount table.");
return -EIO;
}
-
-int qcow2_detect_metadata_preallocation(BlockDriverState *bs)
-{
- BDRVQcow2State *s = bs->opaque;
- int64_t i, end_cluster, cluster_count = 0, threshold;
- int64_t file_length, real_allocation, real_clusters;
-
- file_length = bdrv_getlength(bs->file->bs);
- if (file_length < 0) {
- return file_length;
- }
-
- real_allocation = bdrv_get_allocated_file_size(bs->file->bs);
- if (real_allocation < 0) {
- return real_allocation;
- }
-
- real_clusters = real_allocation / s->cluster_size;
- threshold = MAX(real_clusters * 10 / 9, real_clusters + 2);
-
- end_cluster = size_to_clusters(s, file_length);
- for (i = 0; i < end_cluster && cluster_count < threshold; i++) {
- uint64_t refcount;
- int ret = qcow2_get_refcount(bs, i, &refcount);
- if (ret < 0) {
- return ret;
- }
- cluster_count += !!refcount;
- }
-
- return cluster_count >= threshold;
-}
diff --git a/block/qcow2.c b/block/qcow2.c
index 039bdc2f7e..86e88f6af4 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1895,12 +1895,6 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs,
unsigned int bytes;
int status = 0;
- if (!s->metadata_preallocation_checked) {
- ret = qcow2_detect_metadata_preallocation(bs);
- s->metadata_preallocation = (ret == 1);
- s->metadata_preallocation_checked = true;
- }
-
bytes = MIN(INT_MAX, count);
qemu_co_mutex_lock(&s->lock);
ret = qcow2_get_cluster_offset(bs, offset, &bytes, &cluster_offset);
@@ -1923,11 +1917,6 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs,
} else if (ret != QCOW2_CLUSTER_UNALLOCATED) {
status |= BDRV_BLOCK_DATA;
}
- if (s->metadata_preallocation && (status & BDRV_BLOCK_DATA) &&
- (status & BDRV_BLOCK_OFFSET_VALID))
- {
- status |= BDRV_BLOCK_RECURSE;
- }
return status;
}
diff --git a/block/qcow2.h b/block/qcow2.h
index fc1b0d3c1e..567375e56c 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -356,9 +356,6 @@ typedef struct BDRVQcow2State {
int nb_threads;
BdrvChild *data_file;
-
- bool metadata_preallocation_checked;
- bool metadata_preallocation;
} BDRVQcow2State;
typedef struct Qcow2COWRegion {
@@ -658,7 +655,6 @@ int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order,
void *cb_opaque, Error **errp);
int qcow2_shrink_reftable(BlockDriverState *bs);
int64_t qcow2_get_last_cluster(BlockDriverState *bs, int64_t size);
-int qcow2_detect_metadata_preallocation(BlockDriverState *bs);
/* qcow2-cluster.c functions */
int qcow2_grow_l1_table(BlockDriverState *bs, uint64_t min_size,
diff --git a/include/block/block.h b/include/block/block.h
index 50a07c1c33..269c0931e7 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -156,15 +156,10 @@ typedef struct HDGeometry {
* BDRV_BLOCK_EOF: the returned pnum covers through end of file for this
* layer, set by block layer
*
- * Internal flags:
+ * Internal flag:
* BDRV_BLOCK_RAW: for use by passthrough drivers, such as raw, to request
* that the block layer recompute the answer from the returned
* BDS; must be accompanied by just BDRV_BLOCK_OFFSET_VALID.
- * BDRV_BLOCK_RECURSE: request that the block layer will recursively search for
- * zeroes in file child of current block node inside
- * returned region. Only valid together with both
- * BDRV_BLOCK_DATA and BDRV_BLOCK_OFFSET_VALID. Should not
- * appear with BDRV_BLOCK_ZERO.
*
* If BDRV_BLOCK_OFFSET_VALID is set, the map parameter represents the
* host offset within the returned BDS that is allocated for the
@@ -189,7 +184,6 @@ typedef struct HDGeometry {
#define BDRV_BLOCK_RAW 0x08
#define BDRV_BLOCK_ALLOCATED 0x10
#define BDRV_BLOCK_EOF 0x20
-#define BDRV_BLOCK_RECURSE 0x40
#define BDRV_BLOCK_OFFSET_MASK BDRV_SECTOR_MASK
typedef QSIMPLEQ_HEAD(BlockReopenQueue, BlockReopenQueueEntry) BlockReopenQueue;
diff --git a/tests/qemu-iotests/102 b/tests/qemu-iotests/102
index b898df436f..749ff66b8a 100755
--- a/tests/qemu-iotests/102
+++ b/tests/qemu-iotests/102
@@ -55,7 +55,7 @@ $QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io
$QEMU_IMG resize -f raw --shrink "$TEST_IMG" $((5 * 64 * 1024))
$QEMU_IO -c map "$TEST_IMG"
-$QEMU_IMG map "$TEST_IMG" | _filter_qemu_img_map
+$QEMU_IMG map "$TEST_IMG"
echo
echo '=== Testing map on an image file truncated outside of qemu ==='
diff --git a/tests/qemu-iotests/102.out b/tests/qemu-iotests/102.out
index cd2fdc7f96..4401b08fee 100644
--- a/tests/qemu-iotests/102.out
+++ b/tests/qemu-iotests/102.out
@@ -7,8 +7,7 @@ wrote 65536/65536 bytes at offset 0
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
Image resized.
64 KiB (0x10000) bytes allocated at offset 0 bytes (0x0)
-Offset Length File
-0 0x10000 TEST_DIR/t.IMGFMT
+Offset Length Mapped to File
=== Testing map on an image file truncated outside of qemu ===
diff --git a/tests/qemu-iotests/141.out b/tests/qemu-iotests/141.out
index 4d71d9dcae..41c7291258 100644
--- a/tests/qemu-iotests/141.out
+++ b/tests/qemu-iotests/141.out
@@ -42,9 +42,9 @@ Formatting 'TEST_DIR/o.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.
{"return": {}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}}
-{"return": {}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}
+{"return": {}}
{"error": {"class": "GenericError", "desc": "Node 'drv0' is busy: block device is in use by block job: commit"}}
{"return": {}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "job0"}}
diff --git a/tests/qemu-iotests/144.out b/tests/qemu-iotests/144.out
index a9a8216bea..55299201e4 100644
--- a/tests/qemu-iotests/144.out
+++ b/tests/qemu-iotests/144.out
@@ -14,10 +14,10 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "virtio0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "virtio0"}}
-{"return": {}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "virtio0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "virtio0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}
{"return": {}}
+{"return": {}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "virtio0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "virtio0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "virtio0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}

27
0005-qcow2-Fix-QCOW2_COMPRESSED_SECTOR_MASK.patch
View File

@ -1,27 +0,0 @@
From: Max Reitz <mreitz@redhat.com>
Date: Mon, 28 Oct 2019 17:18:40 +0100
Subject: [PATCH] qcow2: Fix QCOW2_COMPRESSED_SECTOR_MASK
Masks for L2 table entries should have 64 bit.
Fixes: b6c246942b14d3e0dec46a6c5868ed84e7dbea19
Buglink: https://bugs.launchpad.net/qemu/+bug/1850000
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
block/qcow2.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block/qcow2.h b/block/qcow2.h
index 567375e56c..45f9585c08 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -77,7 +77,7 @@
/* Defined in the qcow2 spec (compressed cluster descriptor) */
#define QCOW2_COMPRESSED_SECTOR_SIZE 512U
-#define QCOW2_COMPRESSED_SECTOR_MASK (~(QCOW2_COMPRESSED_SECTOR_SIZE - 1))
+#define QCOW2_COMPRESSED_SECTOR_MASK (~(QCOW2_COMPRESSED_SECTOR_SIZE - 1ULL))
/* Must be at least 2 to cover COW */
#define MIN_L2_CACHE_SIZE 2 /* cache entries */

39
qemu.spec
View File

@ -115,6 +115,7 @@
%define requires_ui_curses Requires: %{name}-ui-curses = %{evr}
%define requires_ui_gtk Requires: %{name}-ui-gtk = %{evr}
%define requires_ui_sdl Requires: %{name}-ui-sdl = %{evr}
%define requires_ui_spice_add Requires: %{name}-ui-spice-app = %{evr}
%global requires_all_modules \
%{requires_block_curl} \
@ -138,7 +139,7 @@
%{obsoletes_block_rbd}
# Release candidate version tracking
#global rcver rc2
%global rcver rc1
%if 0%{?rcver:1}
%global rcrel .%{rcver}
%global rcstr -%{rcver}
@ -147,8 +148,8 @@
Summary: QEMU is a FAST! processor emulator
Name: qemu
Version: 4.1.0
Release: 6%{?rcrel}%{?dist}
Version: 4.2.0
Release: 0.1%{?rcrel}%{?dist}
Epoch: 2
License: GPLv2 and BSD and MIT and CC-BY
URL: http://www.qemu.org/
@ -172,16 +173,6 @@ Source20: kvm-x86.modprobe.conf
# /etc/security/limits.d/95-kvm-ppc64-memlock.conf
Source21: 95-kvm-ppc64-memlock.conf
# gluster 4K block size fixes (bz #1737256)
Patch0001: 0001-file-posix-Handle-undetectable-alignment.patch
Patch0002: 0002-block-posix-Always-allocate-the-first-block.patch
# Fix tests on kernel 5.3+
Patch0003: 0003-tests-make-filemonitor-test-more-robust-to-event-ord.patch
# Workaround for qcow2 triggered XFS corruption (bz #1763519)
Patch0004: 0004-Revert-block-avoid-recursive-block_status-call-if-po.patch
# Fix compressed qcow2 'qemu-img check' errors (bz #1768541)
Patch0005: 0005-qcow2-Fix-QCOW2_COMPRESSED_SECTOR_MASK.patch
# documentation deps
BuildRequires: texinfo
@ -526,6 +517,12 @@ Requires: %{name}-common%{?_isa} = %{epoch}:%{version}-%{release}
%description ui-sdl
This package provides the additional SDL UI for QEMU.
%package ui-spice-app
Summary: QEMU spice-app UI driver
Requires: %{name}-common%{?_isa} = %{epoch}:%{version}-%{release}
%description ui-spice-app
This package provides the additional spice-app UI for QEMU.
%if %{have_kvm}
%package kvm
@ -1189,7 +1186,7 @@ popd
# Copy some static data into place
install -D -p -m 0644 -t %{buildroot}%{qemudocdir} Changelog README COPYING COPYING.LIB LICENSE
install -D -p -m 0644 -t %{buildroot}%{qemudocdir} Changelog README.rst COPYING COPYING.LIB LICENSE
install -D -p -m 0644 qemu.sasl %{buildroot}%{_sysconfdir}/sasl2/qemu.conf
@ -1374,18 +1371,18 @@ getent passwd qemu >/dev/null || \
%files common -f %{name}.lang
%dir %{qemudocdir}
%doc %{qemudocdir}/Changelog
%doc %{qemudocdir}/COPYING
%doc %{qemudocdir}/COPYING.LIB
%doc %{qemudocdir}/LICENSE
%doc %{qemudocdir}/qemu-doc.html
%doc %{qemudocdir}/qemu-doc.txt
%doc %{qemudocdir}/qemu-ga-ref.html
%doc %{qemudocdir}/qemu-ga-ref.txt
%doc %{qemudocdir}/qemu-qmp-ref.html
%doc %{qemudocdir}/qemu-qmp-ref.txt
%doc %{qemudocdir}/README
%doc %{qemudocdir}/README.rst
%doc %{qemudocdir}/interop
%doc %{qemudocdir}/specs
%license %{qemudocdir}/COPYING
%license %{qemudocdir}/COPYING.LIB
%license %{qemudocdir}/LICENSE
%dir %{_datadir}/%{name}/
%{_datadir}/applications/qemu.desktop
%{_datadir}/icons/hicolor/*/apps/*
@ -1495,6 +1492,8 @@ getent passwd qemu >/dev/null || \
%{_libdir}/qemu/ui-gtk.so
%files ui-sdl
%{_libdir}/qemu/ui-sdl.so
%files ui-spice-app
%{_libdir}/qemu/ui-spice-app.so
%files -n ivshmem-tools
@ -1774,7 +1773,6 @@ getent passwd qemu >/dev/null || \
%{_datadir}/%{name}/ppc_rom.bin
%{_datadir}/%{name}/qemu_vga.ndrv
%{_datadir}/%{name}/skiboot.lid
%{_datadir}/%{name}/spapr-rtas.bin
%{_datadir}/%{name}/u-boot.e500
%{_datadir}/%{name}/u-boot-sam460-20100605.bin
%ifarch %{power64}
@ -1867,6 +1865,9 @@ getent passwd qemu >/dev/null || \
%changelog
* Tue Nov 12 2019 Cole Robinson <aintdiscole@gmail.com> - 2:4.2.0-0.1.rc1
- Update to qemu-4.2.0 rc1
* Mon Nov 11 2019 Cole Robinson <crobinso@redhat.com> - 2:4.1.0-6
- Fix compressed qcow2 'qemu-img check' errors (bz #1768541)

2
sources
View File

@ -1 +1 @@
SHA512 (qemu-4.1.0.tar.xz) = 82fd51702a7b9b1b00b2f1bd3b4a832b80249018dbba1add0b0a73e7d4bee452afd45574b4d8df7ce4477d8711f3bda4ca072a1a6de25895c93eb21cf78fc4b2
SHA512 (qemu-4.2.0-rc1.tar.xz) = 8ad5e0472fd384a9ba03b2e8fbb1e887169abb47a50a3f130b1943b39f45677a9e65ca5d1deb96338a5b3c3953db67f50e194a6763e9121c0eb5f620896162a9